A Little Python Makes Dad Happy

My daughter, Jessica was born the Thursday before Father’s Day in 1992, the 3rd of September. That means this year her birthday occurs on Father’s Day.

That begs the question, what other years will this occur? Here’s some Python code that answers the question.

# Find all the years when my daughter's
# birthday and Father's day coincide
# from the year of her birth till 2100

import time

for i in range(1992, 2100):
	dt = "3 09 " + str(i)
	tt = time.strptime(dt, "%d %m %Y")
    	if tt.tm_wday == 6:
    		print i

I love you, Jessica. My favourite Father’s Day gift ever, always.


Bash Completion For Pandoc Is Built In

This is more in the way of a note to myself. I was just starting to write a bash completion script for Pandoc when I came upon this  in the Pandoc documentation:


Generate a bash completion script. To enable bash completion with pandoc, add this to your .bashrc:

 eval "$(pandoc --bash-completion)"


So no need for me to write one. Neat trick, generate your own bash completion script. John McFarlane really is a god. Oh, and the completion is top quality, it knows when you’ve typed an option that takes an input or output format and completes on those and other little tricks. I may end up using some of his tricks for my completions.

A Little Shell Will Fix It

Last night I went to Lights For The Wild at Taronga Zoo. As usual, I took a lot of photos with my DSLR camera, over 200, though a lot of that number are quite similar as I often take two or three to increase the chances of getting the right shot, sometimes I vary the speed so that one is better exposed.

The camera saves both a RAW file​ and a JPEG so I end up with over 400 images. Looking through them in QuickLook in the Finder can be painful as the RAW images take quite a while to load, then you get the problem that when you have decided which of the three shots you want to keep you also have to delete the matching JPEG or RAW file.

The easiest solution to both of these is to only go through the JPEG files and then delete the matching NEF file (which is what the Mac calls the RAW file).

So I open the folder and sort by ‘Kind’ which puts the JPEGs at the top. I then open the first in QuickLook by hitting space and using the up and down arrow keys to move through the list commanddelete deletes a file and displays the next. Easy.

Now I have 80 JPEG files from the original 240. How to get rid of the NEF files that match the JPEG files I have deleted? A little bash programming to the rescue.

for i in *.NEF ; do
  if [ ! -e `basename $i NEF`JPG ]; then 
    rm $i; 

The secret to this is the basename utility. It’s a neat little tool. Pass it a full file path such as /Users/tonyw/Documents/UselessRamblings.txt and it will return just
the file name without the path, UselessRamblings.txt. It has a matching tool, dirname which returns just the path portion. As you can see from my code basename has another trick, it will happily strip the suffix from the filename if you tell it what to strip.

More Tools For Building Tools

I’m working on more bash completions. This time for some of the command line tools Apple provides for sysadmins.

I decided there had to be a way to get a list of the options from the man page for a tool. After all they are all in there.

So I built a command line piece by piece. As an example let’s get a list of the options (with some caveats) for the tool pkgbuild. We start with man pkgbuild | col -b , the col -b step takes out the special characters man uses to show bold on screen. Now find all lines containing -- with grep, I liked grep -e '--'. If you have a look at the output of that we are getting close.

Next I decided to use sed to do a find and replace for the option itself. After some playing around I ended up with sed -e 's#.*\(--[a-zA-Z-]*\).*$#\1#' An important note for young players, it did take some time and a few tries to get that substitution just right. Don’t be afraid and remember Google (and Stack Exchange) are your friends.

First, I should point out an old Unix hand’s trick. Most of the time you see sed substitution commands using / as the separator but sed can use anything but \ or newline – it uses the first character it sees after the ‘s’. I usually use # as it makes the \ used for special characters easier to spot and the string easier to read.

Continue reading

Now We Have bash Completion For Munki

I’m on a roll. I’ve written the bash completions for Munki.

(tl;dr The completions are on Github )

It’s getting easier to write them. There was one little trick I used that I didn’t
mention in my last post that I thought I’d share. How to use find and replace with
regular expressions to generate some of your code.

For this I use Find... in BBEdit. I started with a list of the commands, one on each
line. Continue reading

bash completion for autopkg

Over the weekend I was feeling a little bored so I decided to try my hand at writing a shell script to add custom completion for autopkg to bash.

(tl;dr – the script is on GitHub.)

I found an example for the zsh shell which lacked a couple of features and I spent some time examining the script for brew so I wasn’t totally in the dark.

There are a number of tutorials available for writing them but none are particularly detailed so that wasn’t much help.

Writing Shell Scripts

The first thing I should say is that I find writing shell scripts totally different to writing for any other language. I probably write shell scripts incredibly old school, shell and C were the two languages I was paid to write way back in the 1980’s. It feels like coming home.

Continue reading

Containers Rock! Why I’m A Docker Fan

Docker for the Macintosh has recently emerged from beta and I’m ecstatic.

Docker implements a way of walling off a piece of software from the underlying operating system using a tech they call “containers”.

This is an absolute godsend for deploying services. One of the problems in system administration is the cost and complexity of spinning up a new service and then removing it from a computer once it is no longer required.

Software when it is installed and run can spray pieces of itself all over the computer’s file system and getting it out again is difficult.

Previously we have used virtual machines to isolate this problem. That has it’s own costs, a virtual machine means you are running (at least) two complete operating systems on the hardware. It also has a cost in memory and hard disk space.

Containers lower the cost considerably. They have all the advantages of virtual machines but share the operating system kernel with each other and the underlying OS. This makes them smaller and consuming considerably less resources than virtual machines. This also makes them quicker to download and deploy.

Since Docker is open source it means that there is now a huge community around it. Docker containers are easily available for a huge range of applications, a quick visit to Dockerhub will show you how large.

Docker containers may well be the holy grail of app deployment. They certainly tick all the boxes system administrators require.

Using Docker

So how easy is it to use? Installing it is trivial, just download the install package and copy the Docker application to your Applications folder. You might also want to download
Kitematic which provides a GUI interface to Docker, it also just requires downloading and copying the app to your Applications folder. It is just as easily installed on a Linux box.

You can also install bash completion for docker with this

curl -XGET https://raw.githubusercontent.com/docker/docker/master/contrib/completion/bash/docker > brew --prefix/etc/bash_completion.d/docker

I wish I could tell you how easy it is to build a Docker container from scratch but every time I searched DockerHub for a container I wanted someone else had already built it, or built a large chunk of it.

As an example, I wanted a container running Python 3, Jupyter and the add-on for bash notebooks. Sure, I could have built it from scratch but Continuum, the Anaconda people, already have a Docker container with Python 3 and Jupyter (along with a bunch of other useful Python libraries) installed so :-

docker run -it continuumio/anaconda3 /bin/bash

which will download and run the Python 3 version of Anaconda in a container. Then when the container runs (the -it makes it an interactive container) :-

pip install bash_kernel
python -m bash_kernel.install

then exit the container and at the terminal prompt

docker ps -a
docker commit <container_name> tonyw/jupyter

The ps -a lists all the containers so I know which one to commit and the commit saves the changed container with (optionally) a new name. Now we can run the new container.

docker -d -p 8888:8888 -v /Users:/Users -rm tonyw/jupyter 
 jupyter notebook --ip='*' --port=8888 
 --notebook-dir /Users/tonyw/dev/Notebooks 

This runs the Docker container in ‘daemon’ mode and when the container starts runs the command at the end, in this case Jupyter in notebook mode.

Of course if I just want to run Python 3.5 instead of Jupyter I can always replace the -d with -it and the jupyter command with bash and I get a shell in the container.

Docker Magic

Now all the Docker gurus out there are screaming at me that I should use a Dockerfile to build my custom container and define all sorts of magical stuff like the default command to run when the container starts and the working directory and all the rest so I didn’t need them all in my long command line. Frankly, while that would probably be a good idea I haven’t quite managed to learn how to do all that automated magic and it almost seems like too much work.

Perhaps for my next blog post.

Further Reading

Macadmins Dockerhub
Pepijn Bruienne’s talk on Docker from PSU MacAdmins 2015