Fair_Flow: A Simple Workflow Library

This summer I started learning python in earnest.  I learn best by doing, so I wanted a project that I could put together to see how all the python parts worked.

I eventually settled on a workflow library.  If you’re not familiar, a workflow is made up of steps that need to be accomplished, like a programming language.  Each step is a stand-alone piece of functionality and you can string steps together to do different things.  It usually has a graphical component so you can _see_ what’s happening in your workflow.

The advantage of this is that it’s a simple programming language that non-programmers can use.  If you have functionality in a traditional programming language, users have to wait for the entire software development life cycle to change the smallest part.  The cost in time and effort is too high, so users just put up with what works instead of trying out new things.  But with a workflow system in place, the users just configure the existing steps, so it’s easy to tweak the functionality to add or skip a step.  They can play with new ideas to see if they are worthwhile, and throw away the ones that don’t work.

With that in mind, I set out to make a library to allow just that.  It’s a good learning problem, because the challenges range from string manipulation and go all the way to dynamic class loading.  Plus, there’s all the little things that you don’t use every day, like structuring the packages and uploading to PyPi.

The library itself came together very quickly.  I’m only doing this in my free time (ha) but I’d say the library code came together in about 20 hours.  It was a big mess of functionality that consisted of only two files: the code and the tests to run it.  I always _try_ to write tests for code because it makes the code better and the time to get things working shorter. Finding bugs is just icing on the cake.  Repackaging the code took less than a day, and uploading to PyPi was pretty easy.  You can check it out at https://github.com/fairanswers/fair_flow

But that was just the tip of the iceberg.  The problem was that I had a library, but no easy way to show off how cool it was. So I set up a separate project that lets you use a web page to create and run workflows.  This required wrapping the library in a REST API and creating a front end to make it easy to use.  This turned out to be a lot more work than the original library!  I got a head start by modifying the code over at http://viz-js.com/ for the web page editor and rendering process, came up with some examples, and polished it until I was satisfied.  You can create your own workflows or use the built-in examples to run the whole thing or just one step at a time.  You can try it yourself at https://github.com/fairanswers/fair_flow_example

I still have some tweaks that I’d like to do. Python is such a fast language for getting things done it would be easy to add things for database access, API calls and asynchronous events.  If you have a problem where you need to give the users some control, think about using workflows.

Quick Guide to Packaging Python2 libraries

I’ve been learning python this summer (more on that later), and one thing that is a challenge for me was packaging my first library and shipping it to PyPi, the online repository for python modules.

First, set up your directories:

mkdir fair_flow/

mkdir fair_flow/fair_flow

cd fair_flow

touch __init__.py

sudo apt-get install python-pip python-setuptools

vi setup.py and put this in it

import setuptools

with open(“README.md”, “r”) as fh:
long_description = fh.read()

setuptools.setup(
name=”fair_flow”,
version=”0.0.1″,
author=”Joe Fair”,
author_email=”joe@fairanswers.com”,
description=”Simple Workflow Library”,
url=”https://fairanswers.com”,
packages=setuptools.find_packages(),
classifiers=[
“Programming Language :: Python :: 2”,
“License :: OSI Approved :: MIT License”,
“Operating System :: OS Independent”,
],
)

In the fair_flow/fair_flow/__init__.py put something simple, for testing ou tthe functionality.

name = "fair_flow"

def doit():
  return "Did It"

Install it locally

pip install .

Test it from the command line

python -c 'import fair_flow; print fair_flow.doit()'

Once you're ready, you can upload it to Pypi.  First, create your account on PyPi and install twine with 

pip install twine

Build the distribution

python setup.py sdist

And upload it

twine upload 
python -c 'import fair_flow; print fair_flow.doit()'

Then you can see it on Pypi

https://pypi.org/project/fair-flow/

Thanks for hints and tips from:

  • https://python-packaging.readthedocs.io/en/latest/minimal.html
  • https://packaging.python.org/tutorials/packaging-projects/

Fixing Citrix alt-tab issue on windows 10 (remoting to windows 7)

Last week we upgraded Citrix, which for some reason broke the ability to alt-tab in fullscreen mode to send that command to the remote computer.

There are instructions in the connection window to set it so it goes to the remote, but those instructions didn’t work for me on windows 10 host going to a windows 7 remote box.

After going without it for a week, I got time this morning to google the fix, which is updating a registry key.

Computer\HKEY_CURRENT_USER\Software\Citrix\ICA Client\Engine\Lockdown Profiles\All Regions\Lockdown\Virtual Channels\Keyboard

“TransparentKeyPassthrough”=”Remote”

Now when I log in to work, the windows, alt- keys and everything else works much better!

Cloud Training

I’ve been looking at cloud training this summer, and I went with Linux Academy for a couple reasons:

  1. It was pretty inexpensive.
  2. The videos are really helpful.
  3. One subscription covers a LOT of technologies (AWS, Open Stack, Linux, Security, Azure).
  4. There is a new feature for the Associate Architect called The Orion Papers.

This last point is really helpful. During the videos you get a lecture about the latest version of AWS, which is what you’d expect.

But on the right side of the screen there is a very detailed multi-level diagram of what’s going on. Each video starts with the top-level diagram and zooms in to the area that the video is going to talk about.

For training, this gives you and idea of where you are in the landscape before digging down in the details. It gives you more hooks to relate the content to each other rather than stand-alone, 7 minute videos.  It’s the ‘scaffolding’ that you can hang new knowledge on.

For visual learners I think it’s great. I’ve been trying to figure out how to steal this idea for the docs for my current project.  If I find out how, I’ll let you know.

If you’re interested check it out here. If you sign up with this link, I’ll get a free month.

Lone-Ranger Productivity Tips

Next week I will celebrate my 5 year anniversary of working from home full-time. Here are some tips I’ve picked up.

Work Space

  • A good chair makes all the difference.  A good desk doesn’t make any difference.
  • Good headphones are handy when you need to block noise.  I use over-the ear Sony noise canceling headphones I’ve had for a decade, but if I had to do it over again I’d go cheaper and keep the comfort.
  • For my working telephone I use a battery-powered phone with a speaker option on my home line for work.  It’s Ok, but it ties up the home line.  Recently I splurged on a $20 set of earbuds and started using my cell phone instead.  The calls are clear, and I filter out external noise.
  • I have a great big laptop.  It’s criminally overpowered for what I use it for.  But, if I want to do something ELSE while I’m working, I have power to spare.
  • Walking desks are terrible for typing.  They are worse for detailed mouse-ing.
  • Some people have a problem stopping work.  They feel compelled to check email and see what’s going on on the evenings and week ends.  I avoid that by setting up my work space out of sight. If I saw it all evening I’d be tempted to log in, but since I don’t think about it, I can go all weekend without thinking about work.

Productivity

  • I keep track of tasks with my own system.  Starting with the Pomodoro technique, I simplified it to a spiral bound notebook for tasks.  Instead of a pomodoro timer, I use Shoutcast.com to remind me not to get distracted.  There are fewer commercials and more choice than regular radio.  Using all other task-keeping systems (TidlyWiki, Rally, Getting Things Done) takes too much effort.
  • Communication is key.  You can get about 3 days worth of work done in a day if you have everything you need.  You can waste days and weeks waiting for information.  I do a short daily meeting and I check in with my co-workers at least once or twice a day.
  • It can get lonely.  I get antsy and have to run errands once or twice a week.  Shopping, lunch with others, and driving in the country all help.  Working on housework and working on my own computer-based projects does not.  I don’t go out to coffee shops as much as I thought I would.  Part of that is the comfy chair, part of it is the 10 pound laptop with 90 minutes of battery life.
  • You never, ever, get good gossip.  I’ve been through two layoffs, and I didn’t know about either of them until the day of.

These work for me.  Your mileage may vary.

Log4j2 Note

If you really only need to get log4j2 working in your maven build, do this.

1. Start with deps

<dependencies>

  <dependency>

    <groupId>org.apache.logging.log4j</groupId>

    <artifactId>log4j-api</artifactId>

    <version>2.6.1</version>

  </dependency>

  <dependency>

    <groupId>org.apache.logging.log4j</groupId>

    <artifactId>log4j-core</artifactId>

    <version>2.6.1</version>

  </dependency>

</dependencies>

 

2.  Import these

import org.apache.logging.log4j.LogManager;

import org.apache.logging.log4j.Logger;

 

3. Use this

 private static Logger log = LogManager.getLogger();

Make your life easier: Briefing Books

One of the most difficult parts of working with other people is getting the right information.  Sometimes it pays to have all the information in one place.  One form of this idea is the Briefing Book.

Briefing Books are usually used to give a speaker an overview of a topic before they begin an interview or debate.  It’s not an in-depth tutorial, but more of a refresher before the meeting.  It will include an overview of what the topic is, what questions are likely to come up and the appropriate answers, an agenda for what should happen, and background info for all the people involved.

For example, in 1963, President Kennedy’s advisors prepared briefing books for a meeting in Honolulu with representatives of the South Vietnam government.  It’s what the advisors would review before the meeting to make sure they were prepared with the latest information.  It’s very practical instead of academic.

  • It starts out with travel plans and a short summary of what to expect: the meeting should be about 8 hours, in a specific room room, with about 100 people.
  • It goes on to outline the agenda for the meeting, where each topic has a 3-4 line summary.  You can see that if you walked in to the middle of this meeting, you could find your place in the agenda and get up to speed quickly.
  • The next section is a political review of the situation, recent events, topics for discussion, and a list of “problem areas”, which look like questions that need to be answered. There are also lists of people expected to be there, with their title and a 1 line summary of their experience, or a description of their position.  Some of the leaders also have a separate, longer description of their background. There is also a military summary and a list of maps and charts.

Looking at this briefing book, you can see it was a LOT of work.  However, given the importance of running this meeting smoothly, you can see how important it would be to give everyone the same background.

Back to making our lives easier:  When would we ever need this?

This much preparation is really only necessary for big meetings where you want to have all the facts at hand.  For job negotiations or contract discussions, for example, having an org chart of the other company, pics and bios of the leaders, a list of recent competitors and acquisitions, and salary surveys for your area would be pretty handy.  Having answers to common questions pre-planned and written down that are consistent with your message would be invaluable.

It’s all work that we know we should be doing, but a briefing book is a handy, time-honored way to put it together.  Using a structure like this can help you spot holes in your research, and once it’s put together it’s a concise record of the picture you had at the time.

Here are some other links:

 

How to make your life easier: Custom batch file

I constantly look for ways to make my work more productive.  One of the things I like to do is make batch files to string together commands that have to be done in order.

The Problem:

On one project, I have to do this 5-30 times a day:

Compile the code

Stop the server

Deploy the project to the server

Start the server

However, about once a week the temporary directory gets corrupted.  Sometimes that takes HOURS to debug.  Also, checking out a new branch sometimes points me to the wrong environment and eats up time, too.  To avoid it I just overwrite these files each time I deploy.  But now I do this 5-30 times a day:

Compile the code

Stop the server

Clean out the temp files

Copy the configuration files

Deploy the server

Start the server

How can we make this easier?

The Solution:

I have a directory that is on my PATH (I work in Windows, but Linux would be the similar) that holds all my batch files.  They are short and all follow the same format:

  1. Set up the environment
  2. Do the work
  3. Show that you are done

1.  Setup

Setting up the environment doesn’t seem like a big deal at first.  You know what version of Java you’re using, where tomcat is installed, and what ant version to use, so you can just hard code those paths.  However, it only takes a few extra minutes to set up a file that sets environment variable that you can use (and re-use) in other batch files.  I set tomcat install directories, useful code directories, and add directories to other useful tools.

apsetup.bat:

set JAVA_HOME=C:\bea9\jdk150_04
set ANT_HOME=C:\apps\ant\apache-ant-1.6.5
set ANT_OPTS=”-Xmx512M”
set PATH=%ANT_HOME%\bin;%JAVA_HOME%\bin;%PATH%
set PATH=C:\work\bin;%PATH%
set SVN_EDITOR=c:\windows\system32\notepad.exe
set TOMCAT_HOME=c:\apps\Tomcat7.0.27
set AP=\work\awesomeProject\version_3\repo

now if I run \work\apsetup.bat I can type

cd %ap%

instead of

cd \work\awesomeProject\version_3\repo

However, if I want to change %ap% to a different version, I can change the last line “set AP=…” to that directory.

2.  Do the work

Now that I have the environment set up, I can call %ap%\compile.bat ore %ap%\runTests.bat from the command line window.  Or add this like to skip %AP%\ each time:

set PATH=%AP%\bin;%PATH%

Do the work

Once your environment is set, it’s handy to use absolute paths to show which programs.

%AP%\bin\compileAP.bat

Where compileAP can

Kill the server

Compile cleanly (and remember all the options)

Remove temp files

Deploy

Or %AP%\bin\bounce.bat can

Kill the server

Clean the temp files (but not the deployed ones)

Start the server

3.  Show you’re done

Sometimes it takes a while to complete all these steps.  If distractions occur it make be a while before I get back to my tasks.  It’s frustrating to lose track of where you are and have to start over.

To solve this, I try to do two things:  Give a status update with a timestamp, and give an audible cue when I’m done.

For timestamps I like to use the tools at http://unxutils.sourceforge.net/ , which give you unix-style commands from the windows command line.  Most of the useful commands are there, and I use them all day. It’s easy to put a “date” command at the end of a long process, so I know when it got done.  Also, using the tee command (split output to the stdout and a file) is super handy if we need to see what happened when.

The other timesaver is to send a beep command to the standard out.  For long running commands this reminds me to stop writing email and get back to work.  In windows this can be tricky, but the easiest way I’ve found is to type this in a directory on your path

copy con beep.bat

Then type control-G control-Z.   This gives you two beeps, no muss, no fuss.

 

Fun with Arduino and AtTiny!

Lately I’ve been playing with the Arduino microcontroller and I have to say I really like it.  You can find it at Radio Shack for less than $40, and it comes with the board and a USB cable.  There is a free IDE that makes it fast and easy to get started blinking LEDs, no soldering required.

My first (interesting) project was to create laser tag guns.  We went to a laser tag arena about two weeks ago, and I thought this would be an interesting project.  There are several examples of arduino laser tag, from a commercial site to and IBM project walkthrough, but I decided to go cheap and build it myself.

And when I say cheap, I mean CHEAP! I found some paired-down chips that can be programmed with the arduino at the AniBit.com site for a total of 2 bucks each.  Radio Shack has emitter/receiver pairs for $4, and my kids made lego gun cases for free (and they let me help).

I found some small breadboards on Ebay for $1.25 including shipping, and I had some resistors and LEDs laying around for misc. parts.

I’m still learning about programming the Attinys but I’ve got the process down and the parts, so I think I just need some time to put it together.

One of the interesting problems I found is that the Arduino can communicate back and forth to the IDE via the Serial Monitor.  Values on the chip can be sent to the screen, so you can tell what’s going on.

With the AtTiny is that it doesn’t have a connection to the PC, except through the Arduino, so it’s hard to debug what’s going on.  I used a couple of work around to get the serial interface to go from the Tiny to the Arduino to the computer.

1.  Connect the arduino.  Disconnect the capacitor to the reset (if connected).

2.  Upload ArduinoISP

3.  Connect the capacitor to reset, and pins 10-13.

4.  Load the code you want to test on the tiny.  I cobbled together this
#include <SoftwareSerial.h>

SoftwareSerial mySerial(1,0); // RX, TX
const int led = 4;

void setup()
{
mySerial.begin(9600);
pinMode(led, OUTPUT);
}

void loop()
{
while(true)
{
for (int i = 0; i <= 5; i++)
{
mySerial.println (i); //Send i to Rx Arduino
digitalWrite(led, HIGH);
delay(100);
digitalWrite(led, LOW);
delay(100);
}
delay(1000);
}
}

5.  Disconnect the capacitor and pins 10-13.

6.  Change the board to the Arduino Uno

7.  Upload the following code.

/*
Software serial multple serial test

Receives from the hardware serial, sends to software serial.
Receives from software serial, sends to hardware serial.

The circuit:
* RX is digital pin 10 (connect to TX of other device)
* TX is digital pin 11 (connect to RX of other device)

Note:
Not all pins on the Mega and Mega 2560 support change interrupts,
so only the following can be used for RX:
10, 11, 12, 13, 50, 51, 52, 53, 62, 63, 64, 65, 66, 67, 68, 69

Not all pins on the Leonardo support change interrupts,
so only the following can be used for RX:
8, 9, 10, 11, 14 (MISO), 15 (SCK), 16 (MOSI).

created back in the mists of time
modified 25 May 2012
by Tom Igoe
based on Mikal Hart’s example

This example code is in the public domain.

*/
#include <SoftwareSerial.h>

SoftwareSerial mySerial(11, 12); // RX, TX
int led = 13;

void setup()
{
delay(5000);
// Open serial communications and wait for port to open:
digitalWrite(led, HIGH);
delay(500);
digitalWrite(led, LOW);
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for Leonardo only
}

Serial.println(“Goodnight moon!”);

// set the data rate for the SoftwareSerial port
mySerial.begin(9600);
//mySerial.println(“Hello, world?”);
}

void loop() // run over and over
{
if (mySerial.available())
Serial.write(mySerial.read());
if (Serial.available())
mySerial.write(Serial.read());
}

 

8.  Open the serial window (ctrl-shift-M) and watch the communication work!

 

AutoHotKey – My go-to automation tool

I spent a lot of time programming, and sometimes that means retyping the same thing.  For instance, when I’m in the database, I type ‘select * from ‘ dozens of times a day.  Instead, I can type s*f and autohotkey erases that and puts in ‘select * from’.

::s*f::
SendInput select * from
return

This works in any window, not just my sql editor.  I have about a dozen auto replace commands that I use all the time, and some of them type in 20+ lines of sql for me.  This 6 character command

s*audv

Generates this sql

–Report how long queries take
select operation_name,(percentile_cont(0.90) within group (order by duration))/1000 Percetile_90,
(percentile_cont(0.99) within group (order by duration))/1000 Percetile_99,
(avg(duration))/1000average_duration,
(max(duration))/1000 max_duration,
(min(duration))/1000 min_duration,
count(*)
from audit_service aud
where source_application = ‘SLDB’
and aud.CORRELATION_ID in (
select aud.CORRELATION_ID from audit_service aud
where aud.SOURCE_APPLICATION=’SLDB’
and aud.OPERATION_NAME in (‘validateServiceAvailability’)
and aud.AUDIT_TIMESTAMP > sysdate – 10/1440 )
group by operation_name
order by max_duration desc

That’s a pretty good return on investment!

You can also set hot keys to perform actions, just like on fancy multimedia keyboards.  If you’re not using F8 through F12, use those for pause, play, fast-forward, and rewind.  Or Open Chrome, Open IE, Open Firefox, Open Gvim, if you’d rather.

There is even a language for setting up GUIs and programming your own applications, which I haven’t gotten into.  In my next post, I’ll show you how to navigate your web application so you don’t have to keep clicking the same things over and over.

So, if you’re interested in being more productive, making fewer mistakes, and generally being a happier developer, check out AutoHotKey.