Book Update – Moving to zsh v3

I have pushed an update for the “Moving to zsh” book.

Just a few changes and fixes that have accumulated over the past two weeks. Much of this has been from feedback of readers. Thanks to everyone who sent in their notes.

The update is free if you have already purchased the book. You should get a notification from the Books application to update. (On macOS, I have seen that it can help to delete the local download of the book to force the update.)

If you are enjoying the book, please rate it on the Books store, or (even better) leave a review. These really help, thank you!

Also, please recommend the book to friends, co-workers, and anyone else (not just MacAdmins) who might be facing the zsh transition as they upgrade to Catalina.

The changes in v3 are listed here. This list is also in the ‘Version History’ section in the book. There, you will get links to the relevant section of the book, so you can find the changes quickly.

  • Added a section explaining how to work with upper- or lower-case strings in zsh scripts
  • Added a section explaining the differences in the read built-in command
  • Clarified the section on Connected Variables
  • Fixed file names in the table for Configuration Files and added a note for how to use configuration files with python environments
  • As usual, several typos and clarifications (Thanks to many readers)

Random Terminal Background Colors

In an older post, I showed a trick to get random terminal backgrounds from a selection of colors. Others have used similar approaches.

While I have used this for a long time, the limited number of colors has always annoyed me. And with the introduction of Dark Mode in macOS it seemed just not useful anymore.

Mike’s approach to create the color files with a python script sent me down a rabbit hole to recreate this in Swift. I actually succeeded in creating such a Swift tool, but then, when I worked on connecting the tool with Terminal, I found an even simpler, and arguably better way to do this.

Surprisingly, it involved AppleScript.

Changing the Terminal Background color

Terminal has a powerful AppleScript library, which allows to read and change (among other things) the background color of a Terminal window or tab:

tell application "Terminal"
    get background color of selected tab of window 1
            --> {65535, 65533, 65534}
end tell

The background color is returned as a list of three RGB numbers ranging from 0 to 65535 (216 – 1). You can also set the background color:

tell application "Terminal"
    set background color of selected tab of window 1 to {60000, 45000, 45000}
end tell

This will set the background color of the current window to pastel pink (salmon?).

Armed with this knowledge it is fairly straight forward to write a script that will set the background color to a random color:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    
    set redValue to random number from 0 to maxValue
    set greenValue to random number from 0 to maxValue
    set blueValue to random number from 0 to maxValue
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

You can paste this code in Script Editor and hit the ‘play’ button and watch the Terminal window change. But since we want to use from within Terminal, we will take a different approach: paste the code into your favorite text editor and save it as a text file named randombackground (no extension).

Then open Terminal and change directory to where you save the file and set its executable bit:

> chmod +x randombackground

Now you can run this AppleScript file like any other script file from Terminal:

> ./randombackground

This is fun!

I am not the first to discover and use this, Daniel Jalkut and Erik Barzeski have documented this in 2006.

Enter Dark Mode

Fast forward back to 2018: Along with the rest of macOS, Terminal gained “Dark Mode” in macOS Mojave.

The default “Basic” window profile in Terminal has black text on a white background in light mode and white text on a black background in dark mode. There is some “magic” that happens when the system switches to Dark or Light mode.

However, once we customize the background color (or any other color) the magic does not work any more. When our random backgrounds are too dark in light mode (or vice versa), they don’t really look nice any more, and the text becomes hard to read or completely illegible.

So, we want to change the script to detect dark or light mode and limit the colors accordingly. You can detect dark mode in AppleScript with:

tell application "System Events"
  get dark mode of appearance preferences
end tell

This will return true for dark mode and false for light mode. We modify the script to use just a subrange of all available colors, depending on the mode:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    tell application "System Events"
        set mode to dark mode of appearance preferences
    end tell
    
    if mode then
        set rangestart to 0
        set rangeend to (maxValue * 0.4)
    else
        set rangestart to (maxValue * 0.6)
        set rangeend to maxValue
    end if
    
    set redValue to random number from rangestart to rangeend
    set greenValue to random number from rangestart to rangeend
    set blueValue to random number from rangestart to rangeend
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

When you run this from Terminal for the first time, it may prompt you to allow access to send events to “System Events.” Click ‘OK’ to confirm that:

Automatically setting the background color

Now we can randomize the color by running the command. For simplicity, you may want to put the script file somewhere in your PATH. I put mine in ~/bin/, a folder which a few useful tools and I also added to my PATH(in bash and in zsh).

It is still annoying that it doesn’t happen automatically when we create a new window or tab, but that is exactly what the shell configuration files are for. Add this code to your bash or zsh configuration file.

# random background color
if [[ $TERM_PROGRAM == "Apple_Terminal" ]]; then
    if [[ -x ~/bin/randombackground ]]; then
        ~/bin/randombackground
    fi
fi

Our script will likely fail when the shell is run in any other terminal application or context (such as over ssh). The first if clause checks if the shell is running in Terminal.app. Then the code check to see if the script exists and is executable, and then it executes the script.

This will result in random Terminal colors, matching your choice of dark or light mode.

Note: macOS Catalina added the option to automatically switch the theme depending on the time of day. This script will detect the mode correctly when creating a new window, but Terminal windows and tabs that are already open will retain their color. I am working on a solution…

Upper- or lower-casing strings in bash and zsh

String comparison in most programming languages is case-sensitive. That means that the string 'A' and 'a' are considered different. Humans usually don’t think that way, so there is bound to be trouble and confusion.

If you are looking at single letters, the bracket expansion can be quite useful:

case $input in
    [aA])
        # handle 'a'
        ;;   
    [bB])   
        # handle 'b'
        ;;
    [qQ])
        # handle 'q': quit
        exit 0
        ;;
    *)  
        echo "Option is not available. Please try again"
        ;;
esac

However, for longer strings the bracket expansion gets cumbersome. To cover all case combinations of the word cat you need [cC][aA][tT]. For longer, or unknown strings, it is easier to convert a string to uppercase (all capitals) or lower case before comparing.

sh and bash3

Bash3 and sh have no built-in means to convert case of a string, but you can use the tr tool:

name="John Doe"
# sh
echo $(echo "$name" |  tr '[:upper:]' '[:lower:]' )
john doe

# bash3
echo $(tr '[:upper:]' '[:lower:]' <<< "$name")
john doe

Switch the [:upper:] and [:lower:] arguments to convert to upper case.

There are many other tools available that can provide this functionality, such as awk or sed.

Bash 5

Bash 5 has a special parameter expansion for upper- and lowercasing strings:

name="John Doe"
echo ${name,,}
john doe
echo ${name^^}
JOHN DOE

Zsh

In zsh you can use expansion modifiers:

% name="John Doe"
% echo ${name:l}
john doe
% echo ${name:u}
JOHN DOE

You can also use expansion flags:

% name="John Doe"    
% echo ${(L)name}     
john doe
% echo ${(U)name}
JOHN DOE

In zsh you can even declare a variable as inherently lower case or upper case. This will not affect the contents of the variable, but it will automatically be lower- or uppercased on expansion:

% typeset -l name      
% name="John Doe"
% echo $name        
john doe
% typeset -u name      
% echo $name        
JOHN DOE

Comparing Version strings in zsh

Another excerpt from the book “Moving to zsh.” I found this one so useful, I thought I’d like to share it.

You can get the version of zsh with the ZSH_VERSION variable:

% echo $ZSH_VERSION
5.7.1

And you can get the version of macOS with the sw_vers command:

% sw_vers -productVersion
10.15.1

Comparing version strings is usually fraught with potential errors. Strings are compared by the character code for each character.‘2’ is alphabetically greater than ‘10’ when compared as strings, because the character code for 2 is greater than the character code for 1. So, a string comparison of macOS version numbers will return that 10.9.5 is greater than 10.15.1.

Zsh, however, provides a function is-at-least which helps with version string comparisons.

With a single argument, is-at-least will return if the current zsh version matches or is higher than a given number:

if ! is-at-least 2.6-17; then
  echo "is-at-least is not available"
fi

When you provide two arguments to is-at-least, then the second argument is compared (using version string rules) with the first and needs to match or be higher:

autoload is-at-least
if is-at-least 10.9 $(sw_vers -productVersion); then
  echo "can run Catalina installer"
else
  echo "cannot run Catalina installer"
fi

Note: when used in a script, you will probably have to autoload is-at-least before using it. In an interactive shell, it is often already loaded, because many other autoloader functions will have already loaded it.

Install shellcheck binary on macOS (updated)

A few months back I wrote a post on how to compile and build an installer for the shellcheck binary for macOS.

Just a few weeks later, the shellcheck project added a download for a pre-compiled binary for macOS. You can now download the binary with this link:

https://shellcheck.storage.googleapis.com/shellcheck-latest.darwin.x86_64.tar.xz

Ironically, macOS can unarchive xz archives when you double click them in the Finder, but there is no command line tool on macOS to unarchive them. In the previous post, I ran into the same problem, and there you can find instructions on how to install the xz tools on macOS.

After downloading and un-archiving, you can manually move the shellcheck binary to a suitable directory. The standard location is /usr/local/bin.

For manual installations, this is it! Much simpler than before. Thank you!

Note: if you want the man page as well, you still need to build it with pandoc from the source.

Build a pkg for managed deployment

If you are a MacAdmin and want to distribute shellcheck with your management system, you will need to build an installer package (pkg).

Instead of copying the binary to /usr/local/bin, place it in a payload folder in a project folder. Then build the pkg with pkgbuild:

% mkdir -p ShellcheckPkg/payload
% cp ~/Downloads/shellcheck-latest/shellcheck ShellcheckPkg/payload
% pkgbuild --root ShellcheckPkg/payload --identifier com.example.shellcheck --version 0.7.0 --install-location /usr/local/bin shellcheck-0.7.0.pkg

Replace the 0.7.0 with the actual version number.

Automated Package creation with autopkg

And because all of this isn’t really that difficult, I built autopkg recipes for Shellcheck You can find them in my recipe repository or with autopkg search shellcheck. Enjoy!

Associative arrays in zsh

This is an excerpt from my book “Moving to zsh” which is available for order on the Apple Books Store.

One of the advantages of zsh over bash 3 is the support of “associative arrays,” a data structure known as hash tables or dictionaries in other languages.

In associative arrays, you can store a piece of data, or value with an identifying ‘key’. For example, the associative array userinfo has multiple values, each identified with a key:

% echo $userinfo[name]
armin
% echo $userinfo[shell]
bash
% echo $userinfo[website]
scriptingosx.com

Note: bash 4 also added associative arrays, but they are implemented slightly differently.

Creating associative arrays

In zsh, before you can use a variable as an associative array, you have to declare it as one with

declare -A userinfo

This will tell the shell that the userinfo variable is an associative array. You can also use typeset -A as an alternative syntax. You can verify the type of the variable:

% echo ${(t)userinfo}
association

You can then set the key-value pairs of the userinfo associative array individually:

userinfo[name]="armin"
userinfo[shell]=bash
userinfo[website]="scriptingosx.com"

When you set the value for an existing key again, it will overwrite the existing value:

% echo $userinfo[shell]
bash
% userinfo[shell]=zsh
% echo $userinfo[shell]
zsh

Setting the values for each key is useful in some situations, but can be tedious. You can also set the entire associative array at once. There are two syntaxes for this in zsh:

userinfo=( name armin shell zsh website scriptingosx.com )

This format follows the format ( key1 value1 key2 value2 ...). The other syntax is more verbose and expressive:

userinfo=( [name]=armin [shell]=zsh [website]="scriptingosx.com" )

When you set the associative array variable this way, you are overwriting the entire array. For example, if you set the userinfo for ‘armin’ like above and then set it later like this, the website key and value pair will have been overwritten as well:

% userinfo=( [name]=beth [shell]=zsh )
% if [[ -z $userinfo[website] ]]; then echo no value; fi
no value

If you want to partially overwrite an existing associative array, while leaving the other key/value pairs intact, you can use the += operator:

% userinfo+=( [shell]=fish [website]=fishshell.com )
% echo $userinfo[name]                                           
beth
% echo $userinfo[shell]
fish
% echo $userinfo[website]
fishshell.com

To clear an associative array, you can use:

% userinfo=( )

Retrieving data from an associative array

We have already seen you can get the value for a given key with the ‘subscript’ notation:

% echo $userinfo[name]                                           
beth

When you access the $userinfo variable directly, you will get a normal array of the value:

% echo $userinfo
beth fish fishshell.com

You can also get an array of the keys with this syntax:

% echo ${(k)userinfo}
name shell website

or a list of both keys and values:

% echo ${(kv)userinfo} 
website fishshell.com shell fish name beth

You can use this to copy the data from one associative array to another:

% declare -A otherinfo
% otherinfo=( ${(kv)userinfo )
% echo $otherinfo[name]
beth

You can also use this to loop through all the keys and values of an associated array:

for key value in ${(kv)userinfo}; do
    echo "$key -> $value"
done

#output
website -> fishshell.com
shell -> fish
name -> beth

Limitations

Associative arrays have their uses, but are not as powerful as dictionaries in more powerful languages. In zsh, you cannot nest associative arrays in normal arrays, which limits their use for complex data structures.

There is also no functionality to transfer certain file formats, like XML or property lists directly in to associative arrays or back.

Shell scripting was never designed for complex data structures. When you encounter these limitations, you should move “up” to a higher level language, such as Python or Swift.

Download a full ‘Install macOS’ app with softwareupdate in Catalina

Catalina is out! While you are preparing for your upgrade, here’s a nice new feature for MacAdmins:

The softwareupdate command has gained a new option in Catalina:

% softwareupdate --fetch-full-installer

Will download the latest ‘Install macOS’ application to this Mac’s /Applications folder. This is extremely useful for many admin tasks.

The --fetch-full-installer flag has a sub-flag: --full-installer-version which allows you to download a specific version.

% softwareupdate --fetch-full-installer --full-installer-version 10.14.6

During my testing in the Catalina beta version I was able to download 10.15, 10.14.6, 10.14.5, and 10.13.6. I was not able to test if 10.13.6 would download the hardware specific build of 10.13.6 for the 2018 MacBook Pro, since I do not have that hardware.

I would assume that downloading an Installer application for a macOS version that is not supported on the hardware you are running the command on would fail. (Again I did not have such hardware available for testing.)

So far the only way to download the macOS Installer in some automated fashion was using Greg Neagle’s installinstallmacos.py script. That script still has some abilities that do not seem to be available to the softwareupdate command, but it is good to see Apple accepting the need for this kind of workflow.

Show the git status in the Terminal Window Title

I mentioned in my post on Customizing the zsh prompt, that you can get certain information from the git repository into the prompt or right prompt.

This subject has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

That post contains an example adapted from the ‘Pro git’ documentation which shows how to display the current branch and repo in the prompt.

Personally, I don’t like the Terminal window to be cluttered up with repeated information. While the information from the git repository does change over time, it does not change that frequently, and the information gets repeated over and over.

While I was researching the last post, which describes how to display the current working directory in Terminal’s window title bar, I learnt that you can also set a window title using a different escape code.

So, instead of repeating the git information on every prompt, we can show it in the window or tab title bar.

You can find the code to set this up in my dotfiles repository on Github. To use it make sure the git_vcs_setup file is in your fpath and load it in your .zshrc with

# Git status
autoload -U git_vcs_setup && git_vcs_setup

(More info on fpath and autoloading functions.)

Note that this file sets up two functions: update_terminal_window_title and update_terminal_tab_title. Terminal can have separate information titles for tabs and the entire window. When there are no tabs (or technically, just a single tab) Terminal displays both titles.

The status info above will show the repository, the current branch, a U when there are unstaged changes, and a S when there are staged, but uncommitted changes.

If you want more details in the status, you might want to consider using a more powerful solution to retrieve the git status, such as this project.

Make zsh show working directory in Terminal window title in macOS

This subject has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

While I was working on customizing my zsh configuration files for my zsh article series, I noticed that Terminal would not display the current working directory when using zsh. it worked fine when I switched back to bash.

Note: showing the working directory in the window or tab title is enabled by default in Terminal. You can configure what Terminal shows in the title in the Preferences Window for each profile. Select a profile and

Some sleuthing showed me that /etc/bashrc_Apple_Terminal sets up a update_terminal_cwd function which sends some escape codes to Terminal to keep the window title bar updated. This file is sourced by /etc/bashrc for the Terminal.app.

In macOS 10.14 Mojave and earlier, this configuration file has no equivalent for zsh. /etc/zshrc has code that would load /etc/zshrc_Apple_Terminal, if it existed. However, this file does not exist in macOS Mojave and earlier.

It does in Catalina, though, and it sets up a similar function that is added into the precmd hook. If you have access to the Catalina beta, you can just copy the /etc/zshrc_Apple_Terminal to your Mojave (or earlier) Mac and it will work.

Alternatively, you can write your own implementation, which is what I did, because I wanted it before the Catalina files existed. My solution exists of the function and its setup in its own file. This file is in a directory that is added to the fpath so I can simply load it with autoload.

You can see the code for the function file here.

Then, in my .zshrc I load the function file with:

# path to my zsh functions folder:
my_zsh_functions=$repo_dir/dotfiles/zshfunctions/

# include my zshfunctions dir in fpath:
if [[ -d $my_zsh_functions ]]; then
    fpath=( $my_zsh_functions $fpath )
fi

# only for Mojave and earlier
if [[ $(sw_vers -buildVersion) < "19" ]]; then 
    # this sets up the connection with the Apple Terminal Title Bar
    autoload -U update_terminal_pwd && update_terminal_pwd
fi

And from now, the title will be updated in Mojave (and earlier) and the function won’t even be loaded in Catalina.

This might seem a bit overkill, now that the functionality is in the Catalina file, but it might serve as a useful example when you want to customize, or re-implement similar behavior.

Notarize a Command Line Tool

The upcoming macOS 10.15 Catalina will require more apps and tools to be notarized. Apple has some loosened the requirements at last minute, but these changed limitations are only temporary, to give developers more time to adapt.

Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.

Howard Oakley has written up his experiences and that post was extremely helpful. But there were a few omissions and some steps that aren’t really necessary, so I decided to make my own write-up.

And yes, there is a script at the end…

Note: these instructions are for macOS 10.14.6 Mojave, Xcode 10.3 and Swift 5.0. It is very likely that the details will change over time.

Update 2019-09-24: Tested with Xcode 11 and it still works (the screen layout has changed for some of the options)

What do you need?

  • Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates)
  • Xcode 10.3 or 11
  • Developer ID Certificates (Application and Install)
  • Application Specific Password for your Developer account
  • a Command Line Tool Project that you want to sign and notarize

That’s a longish list. If you are already building command line tools in Xcode, you should have most of these covered already. We will walk through the list step-by-step:

Apple Developer Program Account

You need either the paid membership in the Apple Developer Program or be invited to an Apple Developer Enterprise Program team with access to the proper certificates.

You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.

Xcode

You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.

Developer ID Certificates

There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.

To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.

We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.

If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.

Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:

% security find-identity -p basic -v

This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.

You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"

The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.

Application Specific Password for your Developer Account

Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.

Create a new application specific password in Apple ID portal for your developer account.

You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:

  • Keychain Item Name: Developer-altool
  • Account Name: your developer account email
  • Password: the application-specific password you just created

This will create a developer specific password item that we can access safely from the tools.

If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.

A Command Line Tool Project

You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.

My sample project for this article will be named “hello.”

Preparing the Xcode Project

The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.

Choosing the proper signing certificates

Before you can notarize the command line tool, it needs to be signed with the correct certificates.

  1. in Xcode, select the blue project icon in the left sidebar
  2. select the black “terminal” icon with your project’s name under the “Targets” list entry
  3. make sure the ‘General’ tab is selected
  4. under ‘Signing’ disable ‘Automatically manage signing’
  5. under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
  6. under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Setting the certificates
Setting the certificates

Enable Hardened Runtime

Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.

  1. from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
  2. click on ‘All’ to show all available settings
  3. enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
  4. set the value in the project column (blue icon) to YES
Enable Hardened Runtime
Enable Hardened Runtime

Change the Install Build Location

If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:

  1. in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
  2. double click on the value in the Project column (blue icon), this will open a popup window
  3. change the value to $SRCROOT/build/pkgroot
Change the Installation Build location
Change the Installation Build location

If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)

Build the Binary

You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:

  1. open Terminal and change directory to the project folder
  2. % xcodebuild clean install

This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.

After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.

/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.

Build the pkg

Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.

We can use the pkgroot directory as our payload to build the installer package:

% pkgbuild --root build/pkgroot \
           --identifier "com.example.hello" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Armin Briegel (ABCD123456)" \
           build/hello-1.0.pkg

I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.

This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.

If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”

Notarizing the Installer Package

Xcode has a command line tool altool which you can use to upload your tool for notarization:

xcrun altool --notarize-app \
             --primary-bundle-id "com.example.com" \
             --username "username@example.com" \
             --password "@keychain:Developer-altool" \
             --asc-provider "ABCD123456" \
             --file "build/hello-1.0.pkg"

The username is your developer account email.

The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.

The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)

This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:

xcrun altool --notarization-info "Your-Request-UUID" \
             --username "username@example.com" \                                    
             --password "@keychain:Developer-altool"   

Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.

Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.

Completing the Process

You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.

If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.

You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:

% xcrun stapler staple build/hello-1.0.pkg

You can also use stapler to verify the process went well:

% xcrun stapler validate build/hello-1.0.pkg

But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:

% spctl --assess -vvv --type install build/hello-1.0.pkg

Automating the Process

Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.

The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.

You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.

Links and Videos

These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.

Going forward

Notarization is a key part of Apple’s security strategy going in macOS.

As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.

If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.

On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.

Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.