Hunting Again

My 12 months at the Commonwealth Bank will soon be over. Therefore I need to find a new position. The 12 months at CBA has taught me quite a lot and I’m keen to find a new place to do some more learning.

If you have read my blog, visited my GitHub repositories, or seen some of my presentations you already know a bit about me. I’m a decent engineer with pretty good skills at Python and the shell and I get the job done. I’m a quick learner and a voracious reader of tech sites.

But let me quote from my CV.

“I design and build systems that allow organisations to deploy and secure Macintosh computers, iPhones and iPads.”

“My personal mission is to build a back end that supports well designed, well built, and fuss free devices for the end user. I’d like to work for an organisation that has the same goal and believes Apple devices are a great way of delivering that.”

My CV can be found at https://bit.ly/Tony-CV-2024. If you hear of anything drop me a note at honestpuck at gmail dot com

Another Solution for Uploading Scripts

Some time ago I wrote Scriptorium to keep the scripts on your Mac in sync with the same scripts on your Jamf Pro server.

Thanks to the excellent tool jctl from the people at the Marriot Library of the University of Utah I have a simple solution using their tool and git hooks so when you push to your repo the script is updated in Jamf Pro.

Install jctl from their repo at https://github.com/univ-of-utah-marriott-library-apple/jctl/wiki/Installing. Once you have done that you need the hook in .git/hooks in your repo. Call it pre_push

#!/bin/bash
if [[ $(git diff origin/main@{1} origin/main --name-status | grep -c ${name_of_script}) > 0 ]]; then
    var=$(cat "${path_of_script}")
    /opt/homebrew/bin/jctl scripts -i ${id_of_script} -u script_contents="${var}" --use-the-force-luke
fi

pre-push means that the hook runs after you have typed git push and before the actual push occurs. If you want to you can stop the push from happening by returning non-zero from your script. I don’t do that as I’m not doing any testing in the hook.

name_of_script is, of course, the name of the script. path_of_script is the name of, and path to, the script. id_of_script is the ID in Jamf Pro of the script, you can find it in the URL of the script.

I use direnv to set the variables. You could hard code them but that seems like actual work. Now every time you do a git push the script is sent to your server.

Let me know if you find this useful.

Solve The New York Times Spelling Bee

I am currently visiting my brother in Maryland and his wife, Michele, likes doing the NYT Spelling Bee. It’s a word puzzle where you get 7 letters and one of the 7 is required, with a minimum length of 4 letters. Letters can be repeated in a word. There is at least one word that uses every character.

This raises the question, given a list of words what is the shortest piece of code to give all possible answer words. I think I’m close with a single awk command.

bee() { ;  awk  "/^[$1]{4,}$/ && /$2/" /usr/share/dict/words ; }

You call the function bee <letters> <letter required>. For example the puzzle on September 12 is bee gviotnp t.

As I cannot get the list of words the NYT is using I am using the word list on my Mac which returns more words than the NYT. If I could find a shorter word list closer to the NYT list it would improve the output.

Designing Scriptorium

I just released Scriptorium, a small console program. Here are some notes on how I used argparse to do that.

We need a function to parse our arguments. Parsing is taking the line of words from the command line and processing them to extract the structure and meaning. The term ‘word’ can be fraught with complexity on the shell command line but a simple definition is any set of characters delimited by a space or matching quotes.

Scriptorium has a simple structure, scriptorium <command> [<argument>]. Some commands don’t have any arguments, for some arguments are entirely optional and for others arguments are required. But let’s start building that parser.

    """ build our command line parser """
    parser = argparse.ArgumentParser(
        epilog="for command help: `scriptorium <command> -h`"
    )
    subparsers = parser.add_subparsers(description="", required=True)

You can also see we are starting to build some help in ‘epilog’ – this is the final line printed when you run scriptorium --help. Then we need to have some code to parse each individual commands arguments. This is a subparser.

AutoPkg Repo List Fiddling Again

After my last post Graham Pugh mentioned that the AutoPkg repository list is stored in the AutoPkg preference file as RECIPE_REPOS with the search order in RECIPE_SEARCH_DIRS.

He suggested doing a while loop on the defaults read output but I thought it was just fiddly enough a task in the shell that I might resort to a few lines of Python, so here it is, a Python script to dump out your repository list in search order. Tiny but it does the job.

(Thanks to Graham for taking the time to comment on the previous post, it was just what I needed to get me to spend the few minutes doing this.)

#!/usr/bin/env python3

# repos.py
# print the list of AutoPkg repos in search order
# NOTE: Totally lacking in any error checking or handling

import plistlib
from os import path

plist = path.expanduser('~/Library/Preferences/com.github.autopkg.plist')
fp = open(plist, 'rb')
prefs = plistlib.load(fp)
search = prefs['RECIPE_SEARCH_DIRS']
repos = prefs['RECIPE_REPOS']

# start at 3 to skip the built in ones
for i in range(3, len(search)):
    print(repos[search[i]]['URL'])

Handling Xcode for non administrators

I work in a high security environment and we do not give anyone administrator privileges. Nobody can run sudo. This makes it difficult for developers installing and managing tools such as Xcode. Solutions had to be found for installing Xcode, installing the extras and managing multiple versions. You may have also discovered, as I did, that uploading an 8GB package to the Jamf Cloud is difficult, let alone the time it takes to package it.

So it turns out developers are perfectly capable of downloading the xip file. They also appreciate the better feedback than the total lack of it with the Jamf install process. We just need to install it for them.

Bash Completion For Pandoc Is Built In

This is more in the way of a note to myself. I was just starting to write a bash completion script for Pandoc when I came upon this  in the Pandoc documentation:

--bash-completion

Generate a bash completion script. To enable bash completion with pandoc, add this to your .bashrc:

 eval "$(pandoc --bash-completion)"

 

So no need for me to write one. Neat trick, generate your own bash completion script. John McFarlane really is a god. Oh, and the completion is top quality, it knows when you’ve typed an option that takes an input or output format and completes on those and other little tricks. I may end up using some of his tricks for my completions.

A Little Shell Will Fix It

Last night I went to Lights For The Wild at Taronga Zoo. As usual, I took a lot of photos with my DSLR camera, over 200, though a lot of that number are quite similar as I often take two or three to increase the chances of getting the right shot, sometimes I vary the speed so that one is better exposed.

The camera saves both a RAW file​ and a JPEG so I end up with over 400 images. Looking through them in QuickLook in the Finder can be painful as the RAW images take quite a while to load, then you get the problem that when you have decided which of the three shots you want to keep you also have to delete the matching JPEG or RAW file.

The easiest solution to both of these is to only go through the JPEG files and then delete the matching NEF file (which is what the Mac calls the RAW file).

So I open the folder and sort by ‘Kind’ which puts the JPEGs at the top. I then open the first in QuickLook by hitting space and using the up and down arrow keys to move through the list command–delete deletes a file and displays the next. Easy.

Now I have 80 JPEG files from the original 240. How to get rid of the NEF files that match the JPEG files I have deleted? A little bash programming to the rescue.

for i in *.NEF ; do
  if [ ! -e `basename $i NEF`JPG ]; then 
    rm $i; 
  fi 
done

The secret to this is the basename utility. It’s a neat little tool. Pass it a full file path such as /Users/tonyw/Documents/UselessRamblings.txt and it will return just
the file name without the path, UselessRamblings.txt. It has a matching tool, dirname which returns just the path portion. As you can see from my code basename has another trick, it will happily strip the suffix from the filename if you tell it what to strip.

BBEdit Really Doesn’t Suck

Recently, with version 11, BBEdit introduced a demo mode so I thought to take another look at the big brother of TextWrangler. I have to say BareBones Software’s tag line for BBEdit is true “BBEdit – It doesn’t suck!”.

There are two tasks that I use an editor for, writing Python and writing Markdown so those are the two that I looked at.

There are a number of things you can do to improve BBEdit as a Python IDE. The first is to install Dash. This is a brilliant tool for searching documentation sets and can be easily searched from BBEdit. Just select a library call and choose “Find In Reference…” under the Search menu and BBEdit will pass the search to Dash. Dash will search across all your documentation sets but it is easy to set the sort order so the Python entries are close to the top and in the Dash results window there is a little Python icon next to the Python results.

The other neat item under the Search menu is “Find Definition”, this will find where in your file a function is defined – useful if you have a long source file.

But how does that work if our project is in multiple source files? Well, Unix has long known of that problem and had a solution. It’s a tags file, first used in vi. This is a file that lists all the function definitions and variables used in all the files in a directory tree. Not only can BBEdit use a tags file but it can (using the open source utility ctags) generate them. At the top of your project directory tree, on the command line bbedit --maketags will generate a tags file and now “Find Definition” will work across all the Python files in the tree.

BBEdit can also run a syntax check across your source. You will find “Check Syntax under the ”#!“ menu which also allows you to run your Python code. The final entry in this menu ”Show Module Documentation” displays a new text window with the output from running pydoc across your file. I love this, it encourages me to properly document my code as I write with pydoc strings for each function. The output is extremely useful as a memory aide for large programs and modules.

Next up is running a lint across our Python source. BBEdit comes with another command line tool, bbresults which turns formatted error output from Unix command-line tools into a BBEdit results windows. This is an exceptionally neat trick. At the command line flake8 example.py | bbresults will give you a window in BBEdit with each of the errors and warnings listed and a click on one will take you to the exact spot in your source. If you don’t have flake8 installed then you can install it with conda or pip.

By the way, this works because the bbedit and bbresults command line tools understand the +n argument syntax for going to line n in a file. Sublime Text and other editors on the Mac could learn this.

A final tip for programmers, BBEdit recommends setting the $EDITOR shell variable to bbedit -w where the -w flag has the bbedit command line tool wait till you close the window before exiting. If you add the --resume flag as well then when you close the window in BBEdit it will return the Terminal to the front. Exceptionally handy.

Markdown

One complaint I would make, and I make it about a number of editors, is that the Markdown syntax highlighting is on the stupid side. This is generally due to the flaws in using nothing but regular expressions to do the highlighting. The most obvious flaw is that underlines in such things as a URL will trigger highlighting for italics.

If you want you can “lint” your prose using proselint and bbresults. Personally I find proselint rarely throws up something I actually want to change but your mileage might vary, it’s a good tool for looking at prose text.

BBEdit has no special facilities for writing Markdown such as inserting the codes for text styles or formatting but it does have the ability to use “Clippings”, a short piece of text, and clippings can be kept in sets and a clipping can have a keyboard shortcut. I don’t use it, I have a few Keyboard Maestro macros for such things as web links and otherwise just type the few extra keystrokes.

BBEdit also has “Text Filters”, which allow you to run the current selection through a script. For Markdown I have one that turns tab separated text into a Markdown table, incredibly useful for tables copied from a spreadsheet. Not sure where I got it but I suspect it was from Brett Terpstra’s blog.

BBEdit is a good editor, well worth the $50 purchase price and has a number of advantages over it’s free little brother TextWrangler. As both a general purpose editor and an editor for programming I’d have to say that it is the best editor available on the Mac at the moment though Sublime Text comes close.

Jupyter Releasing Some Nice Software

The Jupyter group have released an alpha version of a new Notebook environment called JupyterLab

JupyterLab is browser based, just like the old notebook system but adds a multiple pane environment. I’m not going to go into the details of the collaboration between the large number of organisations that have gone in to the development, go read the blog post announcing JupyterLab. Suffice to say that I’m glad such a high powered group are working on my favourite Python environment.

I installed the alpha (it’s quickly done with pip) and had a look. It’s an exciting looking development and will make a brilliant Python development environment.

At the moment it seems to be suffering from minor speed problems and minor layout problems in Safari (they are minor, don’t appear in Google Chrome and Safari is not currently listed as a supported browser so I’m not going to complain too loud.)

The built in editor can syntax colour Python. It even has colour themes for those, like me, who like a particular look in their editor. At the moment it is indenting only two characters with a tab (PEP 8 says it should be 4) and if you hit return with the cursor in column 1 then you get a first level indent on the next line.

These are the sort of problems you an expect in alpha software. I think I might install the current development version from Github and check there before filing a couple of bug reports. I’m a bit idiosyncratic, nothing I like more than spending an hour or two getting a bug down to it’s essentials and filing a report.

IPython 5

They have also released a new version of IPython they are calling IPython 5.0 LTS. It has some nice new features including syntax highlighting as you type and much better multi-line support. This is due to shifting from various command line interfaces to the purely Python readline replacement prompt_toolkit.

I think the move to prompt_toolkit is going to show major dividends as the library (currently at version 1.0.3) adds yet more functionality and that functionality moves into IPython. Jonathon Slenders, the author of the library, is also developing clones of Vim and tmux in pure Python using it and intends to fold features from those projects back in to prompt_toolkit.

They are designating this as “Long Term Support” as it will be the last IPython to run under Python 2. IPython 6 will require Python 3. Not is all lost though, they say they will continue to support Python 2 kernels with Jupyter Notebooks (and we assume the new Jupyter Lab). As they say in their announcement “For the 5.x series releases we are making an exception to that rule: until the end of 2017 the core team will do its best to provide fixes for critical bugs in the 5.x release series. Beyond that, we will deprioritise this work, but we will continue to accept pull requests from the community to fix bugs through 2018 and 2019, and make releases when necessary.” So it will be a while before us OS X users are forced to run Python 3 for IPython and break PyObjC and it’s brethren which are written in 2.7 (we can also hope that well before the 20202 deadline Apple moves to Python 3 and does the port of PyObjC.)

Easy Python Development

Taken together these two new releases improve Python development enormously for me. I have always been a fan of iterative development of my code in IPython and this just makes the explore and iterate method easier and easier.