Once again this year I will be teaching a “bash Beginners” workshop at the AUC’s World conference.
I’ve started re-working and improving the workshop and I’ve decided to spend at least some time on regular expressions in
awk as they are so often the hard bits of shell programming. Sorry about that cough, must have been some terrible syntax caught in my throat.
Do you know how hard it is to find a good regular expression experiment and demonstration tool that is compatible with those venerable tools? Well nigh impossible. Perl, Ruby and cough PHP compatible you can find but not for those two.
I recently decided to set myself up with a Unix box in the cloud. I want it so I can use it for things I can’t do on my iPad, most notably using
pandoc for converting MarkDown documents to various other formats.
There are any number of places that offer an incredibly cheap box. I found a special offer at one and bought myself a year for less than $20. I then fired up an Ubuntu 16 server.
Here’s what I did to get it all working in a (fairly) secure manner.
This is more in the way of a note to myself. I was just starting to write a bash completion script for Pandoc when I came upon this in the Pandoc documentation:
Generate a bash completion script. To enable bash completion with pandoc, add this to your
eval "$(pandoc --bash-completion)"
So no need for me to write one. Neat trick, generate your own bash completion script. John McFarlane really is a god. Oh, and the completion is top quality, it knows when you’ve typed an option that takes an input or output format and completes on those and other little tricks. I may end up using some of his tricks for my completions.
Last night I went to Lights For The Wild at Taronga Zoo. As usual, I took a lot of photos with my DSLR camera, over 200, though a lot of that number are quite similar as I often take two or three to increase the chances of getting the right shot, sometimes I vary the speed so that one is better exposed.
The camera saves both a RAW file and a JPEG so I end up with over 400 images. Looking through them in QuickLook in the Finder can be painful as the RAW images take quite a while to load, then you get the problem that when you have decided which of the three shots you want to keep you also have to delete the matching JPEG or RAW file.
The easiest solution to both of these is to only go through the JPEG files and then delete the matching NEF file (which is what the Mac calls the RAW file).
So I open the folder and sort by ‘Kind’ which puts the JPEGs at the top. I then open the first in QuickLook by hitting space and using the up and down arrow keys to move through the list command–delete deletes a file and displays the next. Easy.
Now I have 80 JPEG files from the original 240. How to get rid of the NEF files that match the JPEG files I have deleted? A little
bash programming to the rescue.
for i in *.NEF ; do
if [ ! -e `basename $i NEF`JPG ]; then
The secret to this is the
basename utility. It’s a neat little tool. Pass it a full file path such as
/Users/tonyw/Documents/UselessRamblings.txt and it will return just
the file name without the path,
UselessRamblings.txt. It has a matching tool,
dirname which returns just the path portion. As you can see from my code basename has another trick, it will happily strip the suffix from the filename if you tell it what to strip.
I’m working on more bash completions. This time for some of the command line tools Apple provides for sysadmins.
I decided there had to be a way to get a list of the options from the
man page for a tool. After all they are all in there.
So I built a command line piece by piece. As an example let’s get a list of the options (with some caveats) for the tool
pkgbuild. We start with
man pkgbuild | col -b , the
col -b step takes out the special characters
man uses to show bold on screen. Now find all lines containing
grep, I liked
grep -e '--'. If you have a look at the output of that we are getting close.
Next I decided to use
sed to do a find and replace for the option itself. After some playing around I ended up with
sed -e 's#.*\(--[a-zA-Z-]*\).*$#\1#' An important note for young players, it did take some time and a few tries to get that substitution just right. Don’t be afraid and remember Google (and Stack Exchange) are your friends.
First, I should point out an old Unix hand’s trick. Most of the time you see
sed substitution commands using
/ as the separator but sed can use anything but
\ or newline – it uses the first character it sees after the ‘s’. I usually use
# as it makes the
\ used for special characters easier to spot and the string easier to read.
In my job I am constantly visiting client offices where you need to know a number of IP addresses that aren’t often kept in the local DNS, even if they have one.
Recently I had a revelation, my computer already holds a (very short) list of names to be translated into IP addresses. The ‘hosts’ file in
/etc/ – here’s what it looks like on your Mac:
# Host Database
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
Short, right? Not terribly useful, but we can easily fix that.
Great news for those that want to run IPython on any platform. Continuum Analytics offers a marvellous tool (for free) called Anaconda that will install Python and iPython in one swift step.
Not only can you install all the requirements in one swift step but you can even install them in your home folder if you don’t have administrator access to your Mac.
The icing on the cake is that Anaconda also takes over the task of virtualenv, allowing you to build specific environments with different sets of libraries or a different version of Python. They even offer a package install tool and package repository, all checked to see they work properly with the current version of Anaconda.
Anaconda installs a huge list of packages along with Python and IPython, if you want to install just a few packages they offer a tool, miniconda, that installs only the tools and Python so you can pick and choose what else gets installed.
This is a highly recommended set of tools for Python and IPython development. They even have installs for Windows and one for Linux that packs the installer in a bash script so it can be installed anywhere (once again you can easily install it in your home directory.)