Convert Simple apps to Kubernetes

While doing research on Kubernetes, I was getting tired of running the same nginx server with no pizzazz, so I set up a couple of simple apps to make things more interesting.

I decided to modify nginx to display different colors for each instance: Red and Blue. For each instance I did similar steps

mkdir red && cd red
echo Red > index.html
echo 'FROM nginx
COPY index.html /usr/share/nginx/html' > Dockerfile
sudo docker build .
...

Successfully built 7bb6d362f021

Now that we have an image, let’s tag it and send it to our local docker registry.

sudo docker tag 7bb6d362f021 localhost:5000/red

This results in a new image on our registry

sudo docker images |grep red
localhost:5000/red latest 7bb6d362f021 2 minutes ago 109MB

And run it with docker

$ docker run -d -p 8085:80 localhost:5000/red
$ curl localhost:8085
Red

In general, when converting an application to using docker, we:

  1. Get a (simple) working version of the application
  2. Create a new docker image that starts with a known-good base and adds application specifics.
  3. Test
  4. Repeat steps 2 and 3
  5. Publish to a docker registry.

What does working with Kubernetes look like?

We’ve been working on spreading Kubernetes (k8s) knowledge among our team, and the question keeps coming up “How is this different?” or “How do you get things done?”

There are hundreds of articles that talk about WHAT k8s is, but this talks about HOW to use it.

In general, interacting with Kubernetes means running commands from the command line. Kubernetes comes with a command line tool called kubectl, which interacts with the k8s back end via an API endpoint.

Prerequisites

Before we can set up kubectl, we need to get the config file that sets up

  1. The user
  2. What clusters it has access to.

If we’re setting up a local environment, it should be in the instructions near the end of the process. If you’re accessing a remote cluster, you can get the kubeconfig file from the administrator. If we’re using one cluster, we can just copy the file to ~/.kube/config. If we need to access more than one cluster, we can see how to manage that here.

Once the config file is in place, we can start using kubectl with a command like this:

kubectl cluster-info

If that returns a couple of lines, we’re set. If not, see if the config files and kubectl executable is in place.

Structure of Commands

In general, the kubectl commands use the format

kubectl <action> <resource type> <name> <options>

The action is something like get, create, delete. The resource type can be a pod, deployment, or service. The options are usually prefixed by two dashes, and can specify specific attributes for this command.

Getting Started

Now were starting to get started. First, let’s see what’s been deployed

kubectl get pods

If it’s a new cluster, you probably won’t have anything displayed. Let’s fix that by deploying the simplest kubernetes app: nginx.

kubectl run nginx –image=nginx

Now if you run the get pods, you’ll see something like this

nginx-56f766d96f-l8z8n 1/1 Running 1 332d

Actions

The most common actions are

Get – Get a brief description of a resource

Describe – Get detailed information about a resource

Create – Create a new resource object in the cluster. Most of the time, you’ll want to create objects from files, so you can re-use them later. This is called declarative management. If you want something quick, and won’t need to check in a copy, you can create objects directly from the command line using imperative management.

Delete – Similar to create, in that it can be used as declarative or imperative. Be aware that when we delete a pod that has been created for a deployment, Kubernetes will see that the pod isn’t running and try to start it back up. This may or may not be what you want. If pods keep getting created, look for deployments that you can “describe” to see what pods they are starting.

There are many others, but these are the most common ones.

Troubleshooting

Most of the time, our actions will look something like this.

When we’re troubleshooting, it’s good to start with pods. Since they act as a unit, it’s handy to see which ones have been created or have failed to start.

kubectl get pods

NAME                          READY     STATUS    RESTARTS   AGE
hello-node-7f5b6bd6b8-tqqjn 1/1 Running 1 327d
nginx-56f766d96f-l8z8n 1/1 Running 1 332d

The important parts here are the name and the status. Regular pods start with the name of the container and add a random string to it. We need that string to reference a specific pod. Let’s see what’s going on with our nginx node.

kubectl describe pod nginx-56f766d96f-l8z8n

Name:           nginx-56f766d96f-l8z8n
Namespace: default
Node: minikube/10.0.2.15
Start Time: Sat, 16 Feb 2019 14:36:40 -0700
Labels: app=nginx
pod-template-hash=1293228529
Annotations:
Status: Running
IP: 172.17.0.3
Controlled By: ReplicaSet/nginx-56f766d96f
Containers:
nginx:
Container ID: docker://cf767ded2f705d2fd2560750b030a5b1723a45bb536001271090ec226d041144
Image: nginx
Image ID: docker-pullable://nginx@sha256:8aa7f6a9585d908a63e5e418dc5d14ae7467d2e36e1ab4f0d8f9d059a3d071ce
Port:
Host Port:
State: Running
Started: Wed, 15 Jan 2020 09:45:12 -0700
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Sat, 16 Feb 2019 14:37:12 -0700
Finished: Wed, 15 Jan 2020 09:42:10 -0700
Ready: True
Restart Count: 1
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bdd9p (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-bdd9p:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bdd9p
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulMountVolume 25m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-bdd9p"
Normal SandboxChanged 25m kubelet, minikube Pod sandbox changed, it will be killed and re-created.
Normal Pulling 25m kubelet, minikube pulling image "nginx"
Normal Pulled 24m kubelet, minikube Successfully pulled image "nginx"
Normal Created 24m kubelet, minikube Created container
Normal Started 24m kubelet, minikube Started container

That is a lot of information! The most interesting bits for troubleshooting are:

Status – is it running, or is there a problem?

Containers – a description of what containers this pod will try to start. This is a list, so be aware that while there’s USUALLY one container per pod, there CAN be more than one.

Conditions – Is the pod scheduled to run on a specific node, and if so, is it ready?

Events – This is a list of recent events that pertain to the node. If there is a problem scheduling, downloading, or starting the node, it should be listed here.

Fixing Things

When it comes to changing things in kubernetes, there are a couple of options.

  1. If you need to change or create an object, get the source yaml file for the object in question, modify the yaml as needed, and delete/recreate the object as needed.
  2. If you need to kill a pod that is part of a deployment (stuck, going crazy with memory, etc.) you can use kubectl delete <pod> and it will be terminated. If it’s part of a deployment, k8s will bring up a new one. If not, you’ll need to recreate it.

Finally

Kubernetes is complicated. There are thousands of details, dozens of big concepts, and more than one way to do almost everything.

However, when you’re just getting started it’s easy to get stuck on the details. This gives you an idea of how most of the work managing a kubernetes cluster looks like.

Learning Kubernetes the Easier Way

I started learning Kubernetes in earnest last Thanksgiving. I got the Black Friday deal for LinuxAcademey.com, and went through their course. In theory, it covered the requirements of the Certified Kubernetes Administrator, and maybe it did, but I didn’t learn all that much.

Next, I tried out the CNFC’s official online training. This was OK, but pretty dry.  I couldn’t really get into it, so I got frustrated and stopped about halfway through.

Finally, I went to Udemy.com and got Mumshad Mashewari’s excellent classes for the Certified Kubernetes Developer and the Certified Kubernetes Administrator. Both of them were very well done. What was different?
1. The videos were short, instructive, and professional.

2. After each video or two, there would be a link to an exercise on KodeKloud.com.

The videos were really good, but the link to the exercise allowed me to dig right in to the scenario we learned about.  There are guided questions during the exercise, which asks you to perform tasks, and if you get stuck there are hints.

No setup, script running, installing minikube, checking for running processes, or other hassle required. Answer the first question right (with kubectl commands) and you’ll advance. Pick the wrong answer and it’ll tell you. If it needs to set something up before asking the question, it’ll do that before it asks you, so you have things to debug/research/tease apart, just like in a real environment. Best of all, if you mess everything up, just reload the browser and start over.

If you haven’t checked out KodeKloud, you’re misssing out.

Fair_Flow: A Simple Workflow Library

This summer I started learning python in earnest.  I learn best by doing, so I wanted a project that I could put together to see how all the python parts worked.

I eventually settled on a workflow library.  If you’re not familiar, a workflow is made up of steps that need to be accomplished, like a programming language.  Each step is a stand-alone piece of functionality and you can string steps together to do different things.  It usually has a graphical component so you can _see_ what’s happening in your workflow.

The advantage of this is that it’s a simple programming language that non-programmers can use.  If you have functionality in a traditional programming language, users have to wait for the entire software development life cycle to change the smallest part.  The cost in time and effort is too high, so users just put up with what works instead of trying out new things.  But with a workflow system in place, the users just configure the existing steps, so it’s easy to tweak the functionality to add or skip a step.  They can play with new ideas to see if they are worthwhile, and throw away the ones that don’t work.

With that in mind, I set out to make a library to allow just that.  It’s a good learning problem, because the challenges range from string manipulation and go all the way to dynamic class loading.  Plus, there’s all the little things that you don’t use every day, like structuring the packages and uploading to PyPi.

The library itself came together very quickly.  I’m only doing this in my free time (ha) but I’d say the library code came together in about 20 hours.  It was a big mess of functionality that consisted of only two files: the code and the tests to run it.  I always _try_ to write tests for code because it makes the code better and the time to get things working shorter. Finding bugs is just icing on the cake.  Repackaging the code took less than a day, and uploading to PyPi was pretty easy.  You can check it out at https://github.com/fairanswers/fair_flow

But that was just the tip of the iceberg.  The problem was that I had a library, but no easy way to show off how cool it was. So I set up a separate project that lets you use a web page to create and run workflows.  This required wrapping the library in a REST API and creating a front end to make it easy to use.  This turned out to be a lot more work than the original library!  I got a head start by modifying the code over at http://viz-js.com/ for the web page editor and rendering process, came up with some examples, and polished it until I was satisfied.  You can create your own workflows or use the built-in examples to run the whole thing or just one step at a time.  You can try it yourself at https://github.com/fairanswers/fair_flow_example

I still have some tweaks that I’d like to do. Python is such a fast language for getting things done it would be easy to add things for database access, API calls and asynchronous events.  If you have a problem where you need to give the users some control, think about using workflows.

Quick Guide to Packaging Python2 libraries

I’ve been learning python this summer (more on that later), and one thing that is a challenge for me was packaging my first library and shipping it to PyPi, the online repository for python modules.

First, set up your directories:

mkdir fair_flow/

mkdir fair_flow/fair_flow

cd fair_flow

touch __init__.py

sudo apt-get install python-pip python-setuptools

vi setup.py and put this in it

import setuptools

with open(“README.md”, “r”) as fh:
long_description = fh.read()

setuptools.setup(
name=”fair_flow”,
version=”0.0.1″,
author=”Joe Fair”,
author_email=”joe@fairanswers.com”,
description=”Simple Workflow Library”,
url=”https://fairanswers.com”,
packages=setuptools.find_packages(),
classifiers=[
“Programming Language :: Python :: 2”,
“License :: OSI Approved :: MIT License”,
“Operating System :: OS Independent”,
],
)

In the fair_flow/fair_flow/__init__.py put something simple, for testing ou tthe functionality.

name = "fair_flow"

def doit():
  return "Did It"

Install it locally

pip install .

Test it from the command line

python -c 'import fair_flow; print fair_flow.doit()'

Once you're ready, you can upload it to Pypi.  First, create your account on PyPi and install twine with 

pip install twine

Build the distribution

python setup.py sdist

And upload it

twine upload 
python -c 'import fair_flow; print fair_flow.doit()'

Then you can see it on Pypi

https://pypi.org/project/fair-flow/

Thanks for hints and tips from:

  • https://python-packaging.readthedocs.io/en/latest/minimal.html
  • https://packaging.python.org/tutorials/packaging-projects/

Fixing Citrix alt-tab issue on windows 10 (remoting to windows 7)

Last week we upgraded Citrix, which for some reason broke the ability to alt-tab in fullscreen mode to send that command to the remote computer.

There are instructions in the connection window to set it so it goes to the remote, but those instructions didn’t work for me on windows 10 host going to a windows 7 remote box.

After going without it for a week, I got time this morning to google the fix, which is updating a registry key.

Computer\HKEY_CURRENT_USER\Software\Citrix\ICA Client\Engine\Lockdown Profiles\All Regions\Lockdown\Virtual Channels\Keyboard

“TransparentKeyPassthrough”=”Remote”

Now when I log in to work, the windows, alt- keys and everything else works much better!

Cloud Training

I’ve been looking at cloud training this summer, and I went with Linux Academy for a couple reasons:

  1. It was pretty inexpensive.
  2. The videos are really helpful.
  3. One subscription covers a LOT of technologies (AWS, Open Stack, Linux, Security, Azure).
  4. There is a new feature for the Associate Architect called The Orion Papers.

This last point is really helpful. During the videos you get a lecture about the latest version of AWS, which is what you’d expect.

But on the right side of the screen there is a very detailed multi-level diagram of what’s going on. Each video starts with the top-level diagram and zooms in to the area that the video is going to talk about.

For training, this gives you and idea of where you are in the landscape before digging down in the details. It gives you more hooks to relate the content to each other rather than stand-alone, 7 minute videos.  It’s the ‘scaffolding’ that you can hang new knowledge on.

For visual learners I think it’s great. I’ve been trying to figure out how to steal this idea for the docs for my current project.  If I find out how, I’ll let you know.

If you’re interested check it out here. If you sign up with this link, I’ll get a free month.

Lone-Ranger Productivity Tips

Next week I will celebrate my 5 year anniversary of working from home full-time. Here are some tips I’ve picked up.

Work Space

  • A good chair makes all the difference.  A good desk doesn’t make any difference.
  • Good headphones are handy when you need to block noise.  I use over-the ear Sony noise canceling headphones I’ve had for a decade, but if I had to do it over again I’d go cheaper and keep the comfort.
  • For my working telephone I use a battery-powered phone with a speaker option on my home line for work.  It’s Ok, but it ties up the home line.  Recently I splurged on a $20 set of earbuds and started using my cell phone instead.  The calls are clear, and I filter out external noise.
  • I have a great big laptop.  It’s criminally overpowered for what I use it for.  But, if I want to do something ELSE while I’m working, I have power to spare.
  • Walking desks are terrible for typing.  They are worse for detailed mouse-ing.
  • Some people have a problem stopping work.  They feel compelled to check email and see what’s going on on the evenings and week ends.  I avoid that by setting up my work space out of sight. If I saw it all evening I’d be tempted to log in, but since I don’t think about it, I can go all weekend without thinking about work.

Productivity

  • I keep track of tasks with my own system.  Starting with the Pomodoro technique, I simplified it to a spiral bound notebook for tasks.  Instead of a pomodoro timer, I use Shoutcast.com to remind me not to get distracted.  There are fewer commercials and more choice than regular radio.  Using all other task-keeping systems (TidlyWiki, Rally, Getting Things Done) takes too much effort.
  • Communication is key.  You can get about 3 days worth of work done in a day if you have everything you need.  You can waste days and weeks waiting for information.  I do a short daily meeting and I check in with my co-workers at least once or twice a day.
  • It can get lonely.  I get antsy and have to run errands once or twice a week.  Shopping, lunch with others, and driving in the country all help.  Working on housework and working on my own computer-based projects does not.  I don’t go out to coffee shops as much as I thought I would.  Part of that is the comfy chair, part of it is the 10 pound laptop with 90 minutes of battery life.
  • You never, ever, get good gossip.  I’ve been through two layoffs, and I didn’t know about either of them until the day of.

These work for me.  Your mileage may vary.

Log4j2 Note

If you really only need to get log4j2 working in your maven build, do this.

1. Start with deps

<dependencies>

  <dependency>

    <groupId>org.apache.logging.log4j</groupId>

    <artifactId>log4j-api</artifactId>

    <version>2.6.1</version>

  </dependency>

  <dependency>

    <groupId>org.apache.logging.log4j</groupId>

    <artifactId>log4j-core</artifactId>

    <version>2.6.1</version>

  </dependency>

</dependencies>

 

2.  Import these

import org.apache.logging.log4j.LogManager;

import org.apache.logging.log4j.Logger;

 

3. Use this

 private static Logger log = LogManager.getLogger();

Make your life easier: Briefing Books

One of the most difficult parts of working with other people is getting the right information.  Sometimes it pays to have all the information in one place.  One form of this idea is the Briefing Book.

Briefing Books are usually used to give a speaker an overview of a topic before they begin an interview or debate.  It’s not an in-depth tutorial, but more of a refresher before the meeting.  It will include an overview of what the topic is, what questions are likely to come up and the appropriate answers, an agenda for what should happen, and background info for all the people involved.

For example, in 1963, President Kennedy’s advisors prepared briefing books for a meeting in Honolulu with representatives of the South Vietnam government.  It’s what the advisors would review before the meeting to make sure they were prepared with the latest information.  It’s very practical instead of academic.

  • It starts out with travel plans and a short summary of what to expect: the meeting should be about 8 hours, in a specific room room, with about 100 people.
  • It goes on to outline the agenda for the meeting, where each topic has a 3-4 line summary.  You can see that if you walked in to the middle of this meeting, you could find your place in the agenda and get up to speed quickly.
  • The next section is a political review of the situation, recent events, topics for discussion, and a list of “problem areas”, which look like questions that need to be answered. There are also lists of people expected to be there, with their title and a 1 line summary of their experience, or a description of their position.  Some of the leaders also have a separate, longer description of their background. There is also a military summary and a list of maps and charts.

Looking at this briefing book, you can see it was a LOT of work.  However, given the importance of running this meeting smoothly, you can see how important it would be to give everyone the same background.

Back to making our lives easier:  When would we ever need this?

This much preparation is really only necessary for big meetings where you want to have all the facts at hand.  For job negotiations or contract discussions, for example, having an org chart of the other company, pics and bios of the leaders, a list of recent competitors and acquisitions, and salary surveys for your area would be pretty handy.  Having answers to common questions pre-planned and written down that are consistent with your message would be invaluable.

It’s all work that we know we should be doing, but a briefing book is a handy, time-honored way to put it together.  Using a structure like this can help you spot holes in your research, and once it’s put together it’s a concise record of the picture you had at the time.

Here are some other links: