Wrangling Pythons

As I noted in my last Weekly News Summary, several open source projects for MacAdmins have completed their transition to Python 3. AutoPkg, JSSImport and outset announced Python 3 compatible versions last week and Munki already had the first Python 3 version last December.

Why?

Apple has included a version of Python 2 with Mac OS X since 10.2 (Jaguar). Python 3.0 was released in 2008 and it was not fully backwards compatible with Python 2. For this reason, Python 2 was maintained and updated alongside Python 3 for a long time. Python 2 was finally sunset on January 1, 2020. Nevertheless, presumably because of the compatibility issues, Apple has always pre-installed Python 2 with macOS and still does so in macOS 10.15 Catalina. With the announcement of Catalina, Apple also announced that in a “future version of macOS” there will be no pre-installed Python of any version.

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. (macOS 10.15 Catalina Release Notes)

This also applies to Perl and Ruby runtimes and other libraries. I will be focussing on Python because it is used more commonly for MacAdmin tools, but most of this post will apply equally to Perl and Ruby. Just mentally replace “Python” for your preferred language.

The final recommendation is what AutoPkg and Munki are following: they are bundling their own Python runtime.

How to get Python

There is a second bullet in the Catalina release notes, though:

Use of Python 2.7 isn’t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS won’t include Python 2.7. Instead, it’s recommended that you run python3 from within Terminal. (51097165)

This is great, right? Apple says there is a built-in Python 3! And it’s pre-installed? Just move all your scripts to Python 3 and you’ll be fine!

Unfortunately, not quite. The python3 binary does exist on a ‘clean’ macOS, but it is only a stub tool, that will prompt a user to download and install the Command Line Developer Tools (aka “Developer Command Line Tools” or “Command Line Tools for Xcode”). This is common for many tools that Apple considers to be of little interest to ‘normal,’ non-developer users. Another common example is git.

Dialog prompting to install the Command Line Tools
Dialog prompting to install the Command Line Tools

When you install Xcode, you will also get all the Command Line Developer Tools, including python3 and git. This is useful for developers, who may want to use Python scripts for build operation, or for individuals who just want to ‘play around’ or experiment with Python locally. For MacAdmins, it adds the extra burden of installing and maintaining either the Command Line Developer Tools or the full Xcode install.

Python Versions, a multitude of Snakes

After installing Xcode or the Command Line Developer Tools, you can check the version of python installed: (versions on macOS 10.15.3 with Xcode 11.3.1)

> python --version    
Python 2.7.16
> python3 --version    
Python 3.7.3

When you go on the download page for Python.org, you will get Python 3.8.1 (as of this writing). But, on that download page, you will also find download links for “specific versions” which include (as of this writing) versions 3.8.1, 3.7.6, 3.6.10, 3.5.9, and the deprecated 2.7.17.

The thing is, that Python isn’t merely split into two major release versions, which aren’t fully compatible with each other, but there are several minor versions of Python 3, which aren’t fully compatible with each other, but are still being maintained in parallel.

Developers (individuals, teams, and organisations) that use Python will often hold on to a specific minor (and sometimes even patch) version for a project to avoid issues and bugs that might appear when changing the run-time.

When you install the latest version of Munki, it will install a copy of the Python framework in /usr/local/munki/ and create a symbolic link to that python binary at /usr/local/munki/python. You can check its version as well:

 % /usr/local/munki/python --version
Python 3.7.4

All the Python code files for Munki will have a shebang (the first line in the code file) of

#!/usr/local/munki/python

This ensures that Munki code files use this particular instance of Python and no other copy of Python that may have been installed on the system.

The latest version of AutoPkg has a similar approach:

> /usr/local/autopkg/python --version    
Python 3.7.5

In both cases the python binary is a symbolic link. This allows the developer to change the symbolic link to point to a different Python framework. The shebangs in the all the code files point to the symbolic link, which can be changed to point to a different Python framework.

This is useful for testing and debugging. Could MacAdmins use this to point both tools to the same Python framework? Should they?

The Bridge to macOS

On top of all these different versions of Python itself, many scripts, apps, and tools written in Python rely on ‘Python modules.’ These are libraries (or frameworks) of code for a certain task, that can be downloaded and included with a Python installation to extend the functionality of Python.

The most relevant of these modules for MacAdmins is the “Python Objective-C Bridge.” This module allows Python code to access and use the native macOS Cocoa and CoreFoundation Frameworks. This not only allows for macOS native GUI applications to be written in Python (e.g. AutoDMG and Munki’s Managed Software Center [update: MSC was re-written in Swift last year]), but also allows short scripts to access system functions. This is sometimes necessary to get a data that matches what macOS applications “see” rather than what the raw unix tools see.

For example, the defaults tool can be used to read the value of property lists on disk. But those might not necessarily reflect the actual preference value an application sees, because that value might be controlled by a different plist file or configuration profile.

(Shameless self-promotion) Learn more about Property lists, Preferences and Profiles

You could build a tool with Swift or Objective-C that uses the proper frameworks to get the “real” preference value. Or you can use Python with the Objective-C bridge:

#!/usr/bin/python
from Foundation import CFPreferencesCopyAppValue
print CFPreferencesCopyAppValue("idleTime", "com.apple.screensaver")

Three simple lines of Python code. This will work with the pre-installed Python 2.7, because Apple also pre-installs the Python Objective-C bridge with that. When you try this with the Developer Tools python3 you get an error:

ModuleNotFoundError: No module named 'Foundation'

This is because the Developer Tools do not include the Objective-C bridge in the installation. You could easily add it with:

> sudo python3 -m pip install pyobjc

But again, while this command is “easy” enough for a single user on a single Mac, it is just the beginning of a Minoan labyrinth of management troubles.

Developers and MacAdmins, have to care about the version of the Python they install, as well as the list of modules and their versions, for each Python version.

It is as if the Medusa head kept growing more smaller snakes for every snake you cut off.

(Ok, I will ease off with Greek mythology metaphors.)

You can get a list of modules included with the AutoPkg and the Munki project with:

> /usr/local/munki/python -m pip list
> /usr/local/autopkg/python -m pip list

You will see that not only do Munki and AutoPkg include different versions of Python, but also a different list of modules. While Munki and AutoPkg share many modules, their versions might still differ.

Snake Herding Solutions

Apple’s advice in the Catalina Release Notes is good advice:

It’s recommended that you bundle the runtime within the app.

Rather than the MacAdmin managing a single version of Python and all the modules for every possible solution, each tool or application should provide its own copy of Python and its required modules.

If you want to build your own Python bundle installer, you can use this script from Greg Neagle.

This might seem wasteful. A full Python 3 Framework uses about 80MB of disk space, plus some extra for the modules. But it is the safest way to ensure that the tool or application gets the correct version of Python and all the modules. Anything else will quickly turn into a management nightmare.

This is the approach that Munki and AutoPkg have chosen. But what about smaller, single script solutions? For example simple Python scripts like quickpkg or prefs-tool?

Should I bundle my own Python framework with quickpkg or prefs-tool? I think that would be overkill and I am not planning to do that. I think the solution that Joseph Chilcote chose for the outset tool is a better approach for less complex Python scripts.

In this case, the project is written to run with Python 3 and generic enough to not require a specific version or extra modules. An admin who wants to use this script or tool, can change the shebang (the first line in the script) to point to either the Developer Tool python3, the python3 from the standard Python 3 installer or a custom Python version, such as the Munki python. A MacAdmin would have to ensure that the python binary in the shebang is present on the Mac when the tool runs.

You can also choose to provide your organization’s own copy Python with your chosen set of modules for all your management Python scripts and automations. You could build this with the relocatable Python tool and place it in a well-known location the clients. When updates for the Python run-time or modules are required, you can build and push them with your management system. (Thanks to Nathaniel Strauss for pointing out this needed clarifying.)

When you build such scripts and tools, it is important to document which Python versions (and module versions) you have tested the tool with.

(I still have to do that for my Python tools.)

What about /usr/bin/env python?

The env command will determine the path to the python binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the python binary is unknown.

This is useful when developers want to use the same script in different environments across different computers, user accounts, and platforms. However, this renders the actual version of python that will interpret the script completely unpredictable.

Not only is it impossible to predict which version of Python will interpret a script, but you cannot depend on any modules being installed (or their versions) either.

For MacAdmin management scripts and tools, a tighter control is necessary. You should use fixed, absolute paths in the shebang.

Conclusion

Managing Python runtimes might seem like a hopeless sisyphean task. I believe Apple made the right choice to not pre-install Python any more. Whatever version and pre-selection of module versions Apple would have chosen, it would only have been the correct combination for a few Python solutions and developers.

While it may seem wasteful to have a multitude of copies of the Python frameworks distributed through out the system, it is the easiest and most manageable solution to ensure that each tool or application works with the expected combination of run-time and modules.

Book Update – Moving to zsh v3

I have pushed an update for the “Moving to zsh” book.

Just a few changes and fixes that have accumulated over the past two weeks. Much of this has been from feedback of readers. Thanks to everyone who sent in their notes.

The update is free if you have already purchased the book. You should get a notification from the Books application to update. (On macOS, I have seen that it can help to delete the local download of the book to force the update.)

If you are enjoying the book, please rate it on the Books store, or (even better) leave a review. These really help, thank you!

Also, please recommend the book to friends, co-workers, and anyone else (not just MacAdmins) who might be facing the zsh transition as they upgrade to Catalina.

The changes in v3 are listed here. This list is also in the ‘Version History’ section in the book. There, you will get links to the relevant section of the book, so you can find the changes quickly.

  • Added a section explaining how to work with upper- or lower-case strings in zsh scripts
  • Added a section explaining the differences in the read built-in command
  • Clarified the section on Connected Variables
  • Fixed file names in the table for Configuration Files and added a note for how to use configuration files with python environments
  • As usual, several typos and clarifications (Thanks to many readers)

Random Terminal Background Colors

In an older post, I showed a trick to get random terminal backgrounds from a selection of colors. Others have used similar approaches.

While I have used this for a long time, the limited number of colors has always annoyed me. And with the introduction of Dark Mode in macOS it seemed just not useful anymore.

Mike’s approach to create the color files with a python script sent me down a rabbit hole to recreate this in Swift. I actually succeeded in creating such a Swift tool, but then, when I worked on connecting the tool with Terminal, I found an even simpler, and arguably better way to do this.

Surprisingly, it involved AppleScript.

Changing the Terminal Background color

Terminal has a powerful AppleScript library, which allows to read and change (among other things) the background color of a Terminal window or tab:

tell application "Terminal"
    get background color of selected tab of window 1
            --> {65535, 65533, 65534}
end tell

The background color is returned as a list of three RGB numbers ranging from 0 to 65535 (216 – 1). You can also set the background color:

tell application "Terminal"
    set background color of selected tab of window 1 to {60000, 45000, 45000}
end tell

This will set the background color of the current window to pastel pink (salmon?).

Armed with this knowledge it is fairly straight forward to write a script that will set the background color to a random color:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    
    set redValue to random number from 0 to maxValue
    set greenValue to random number from 0 to maxValue
    set blueValue to random number from 0 to maxValue
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

You can paste this code in Script Editor and hit the ‘play’ button and watch the Terminal window change. But since we want to use from within Terminal, we will take a different approach: paste the code into your favorite text editor and save it as a text file named randombackground (no extension).

Then open Terminal and change directory to where you save the file and set its executable bit:

> chmod +x randombackground

Now you can run this AppleScript file like any other script file from Terminal:

> ./randombackground

This is fun!

I am not the first to discover and use this, Daniel Jalkut and Erik Barzeski have documented this in 2006.

Enter Dark Mode

Fast forward back to 2018: Along with the rest of macOS, Terminal gained “Dark Mode” in macOS Mojave.

The default “Basic” window profile in Terminal has black text on a white background in light mode and white text on a black background in dark mode. There is some “magic” that happens when the system switches to Dark or Light mode.

However, once we customize the background color (or any other color) the magic does not work any more. When our random backgrounds are too dark in light mode (or vice versa), they don’t really look nice any more, and the text becomes hard to read or completely illegible.

So, we want to change the script to detect dark or light mode and limit the colors accordingly. You can detect dark mode in AppleScript with:

tell application "System Events"
  get dark mode of appearance preferences
end tell

This will return true for dark mode and false for light mode. We modify the script to use just a subrange of all available colors, depending on the mode:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    tell application "System Events"
        set mode to dark mode of appearance preferences
    end tell
    
    if mode then
        set rangestart to 0
        set rangeend to (maxValue * 0.4)
    else
        set rangestart to (maxValue * 0.6)
        set rangeend to maxValue
    end if
    
    set redValue to random number from rangestart to rangeend
    set greenValue to random number from rangestart to rangeend
    set blueValue to random number from rangestart to rangeend
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

When you run this from Terminal for the first time, it may prompt you to allow access to send events to “System Events.” Click ‘OK’ to confirm that:

Automatically setting the background color

Now we can randomize the color by running the command. For simplicity, you may want to put the script file somewhere in your PATH. I put mine in ~/bin/, a folder which a few useful tools and I also added to my PATH(in bash and in zsh).

It is still annoying that it doesn’t happen automatically when we create a new window or tab, but that is exactly what the shell configuration files are for. Add this code to your bash or zsh configuration file.

# random background color
if [[ $TERM_PROGRAM == "Apple_Terminal" ]]; then
    if [[ -x ~/bin/randombackground ]]; then
        ~/bin/randombackground
    fi
fi

Our script will likely fail when the shell is run in any other terminal application or context (such as over ssh). The first if clause checks if the shell is running in Terminal.app. Then the code check to see if the script exists and is executable, and then it executes the script.

This will result in random Terminal colors, matching your choice of dark or light mode.

Note: macOS Catalina added the option to automatically switch the theme depending on the time of day. This script will detect the mode correctly when creating a new window, but Terminal windows and tabs that are already open will retain their color. I am working on a solution…

Install shellcheck binary on macOS (updated)

A few months back I wrote a post on how to compile and build an installer for the shellcheck binary for macOS.

Just a few weeks later, the shellcheck project added a download for a pre-compiled binary for macOS. You can now download the binary with this link:

https://shellcheck.storage.googleapis.com/shellcheck-latest.darwin.x86_64.tar.xz

Ironically, macOS can unarchive xz archives when you double click them in the Finder, but there is no command line tool on macOS to unarchive them. In the previous post, I ran into the same problem, and there you can find instructions on how to install the xz tools on macOS.

Update 2020-03-26: I was unnecessarily complicating this. You can use tar to unarchive this:

tar -xf shellcheck-latest.darwin.x86_64.tar.xz

After downloading and un-archiving, you can manually move the shellcheck binary to a suitable directory. The standard location is /usr/local/bin.

For manual installations, this is it! Much simpler than before. Thank you!

Note: if you want the man page as well, you still need to build it with pandoc from the source.

Build a pkg for managed deployment

If you are a MacAdmin and want to distribute shellcheck with your management system, you will need to build an installer package (pkg).

Instead of copying the binary to /usr/local/bin, place it in a payload folder in a project folder. Then build the pkg with pkgbuild:

% mkdir -p ShellcheckPkg/payload
% cp ~/Downloads/shellcheck-latest/shellcheck ShellcheckPkg/payload
% pkgbuild --root ShellcheckPkg/payload --identifier com.example.shellcheck --version 0.7.0 --install-location /usr/local/bin shellcheck-0.7.0.pkg

Replace the 0.7.0 with the actual version number.

Automated Package creation with autopkg

And because all of this isn’t really that difficult, I built autopkg recipes for Shellcheck You can find them in my recipe repository or with autopkg search shellcheck. Enjoy!

Associative arrays in zsh

This is an excerpt from my book “Moving to zsh” which is available for order on the Apple Books Store.

One of the advantages of zsh over bash 3 is the support of “associative arrays,” a data structure known as hash tables or dictionaries in other languages.

In associative arrays, you can store a piece of data, or value with an identifying ‘key’. For example, the associative array userinfo has multiple values, each identified with a key:

% echo $userinfo[name]
armin
% echo $userinfo[shell]
bash
% echo $userinfo[website]
scriptingosx.com

Note: bash 4 also added associative arrays, but they are implemented slightly differently.

Creating associative arrays

In zsh, before you can use a variable as an associative array, you have to declare it as one with

declare -A userinfo

This will tell the shell that the userinfo variable is an associative array. You can also use typeset -A as an alternative syntax. You can verify the type of the variable:

% echo ${(t)userinfo}
association

You can then set the key-value pairs of the userinfo associative array individually:

userinfo[name]="armin"
userinfo[shell]=bash
userinfo[website]="scriptingosx.com"

When you set the value for an existing key again, it will overwrite the existing value:

% echo $userinfo[shell]
bash
% userinfo[shell]=zsh
% echo $userinfo[shell]
zsh

Setting the values for each key is useful in some situations, but can be tedious. You can also set the entire associative array at once. There are two syntaxes for this in zsh:

userinfo=( name armin shell zsh website scriptingosx.com )

This format follows the format ( key1 value1 key2 value2 ...). The other syntax is more verbose and expressive:

userinfo=( [name]=armin [shell]=zsh [website]="scriptingosx.com" )

When you set the associative array variable this way, you are overwriting the entire array. For example, if you set the userinfo for ‘armin’ like above and then set it later like this, the website key and value pair will have been overwritten as well:

% userinfo=( [name]=beth [shell]=zsh )
% if [[ -z $userinfo[website] ]]; then echo no value; fi
no value

If you want to partially overwrite an existing associative array, while leaving the other key/value pairs intact, you can use the += operator:

% userinfo+=( [shell]=fish [website]=fishshell.com )
% echo $userinfo[name]                                           
beth
% echo $userinfo[shell]
fish
% echo $userinfo[website]
fishshell.com

To clear an associative array, you can use:

% userinfo=( )

Retrieving data from an associative array

We have already seen you can get the value for a given key with the ‘subscript’ notation:

% echo $userinfo[name]                                           
beth

When you access the $userinfo variable directly, you will get a normal array of the value:

% echo $userinfo
beth fish fishshell.com

You can also get an array of the keys with this syntax:

% echo ${(k)userinfo}
name shell website

or a list of both keys and values:

% echo ${(kv)userinfo} 
website fishshell.com shell fish name beth

You can use this to copy the data from one associative array to another:

% declare -A otherinfo
% otherinfo=( ${(kv)userinfo )
% echo $otherinfo[name]
beth

You can also use this to loop through all the keys and values of an associated array:

for key value in ${(kv)userinfo}; do
    echo "$key -> $value"
done

#output
website -> fishshell.com
shell -> fish
name -> beth

Limitations

Associative arrays have their uses, but are not as powerful as dictionaries in more powerful languages. In zsh, you cannot nest associative arrays in normal arrays, which limits their use for complex data structures.

There is also no functionality to transfer certain file formats, like XML or property lists directly in to associative arrays or back.

Shell scripting was never designed for complex data structures. When you encounter these limitations, you should move “up” to a higher level language, such as Python or Swift.

EraseInstall Update 1.2.1

Just a “minor” update. The EraseInstall app now shows the progress that the startosinstall command gives in the command line. This also should help with some better error reporting when the startosinstall command errors out.

I say “minor” but small UI change required some major rewiring underneath. It also required us to dive deeper into how shell commands are executed from Swift than we wanted to.

We have also tested this version to work with macOS Catalina which was released yesterday.

We have more “major” features planned for the future!

You can download the latest EraseInstall Installer and Code here!

Notarize a Command Line Tool

There is an updated version of this post for the new tools in Xcode 13.

The upcoming macOS 10.15 Catalina will require more apps and tools to be notarized. Apple has somewhat loosened the requirements at last minute, but these changed limitations are only temporary, to give developers more time to adapt.

Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.

Howard Oakley has written up his experiences and that post was extremely helpful. But there were a few omissions and some steps that aren’t really necessary, so I decided to make my own write-up.

And yes, there is a script at the end…

Note: these instructions are for macOS 10.14.6 Mojave, Xcode 10.3 and Swift 5.0. It is very likely that the details will change over time.

Update 2019-09-24: Tested with Xcode 11 and it still works (the screen layout has changed for some of the options)

What do you need?

  • Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates)
  • Xcode 10.3 or 11
  • Developer ID Certificates (Application and Install)
  • Application Specific Password for your Developer account
  • a Command Line Tool Project that you want to sign and notarize

That’s a longish list. If you are already building command line tools in Xcode, you should have most of these covered already. We will walk through the list step-by-step:

Apple Developer Program Account

You need either the paid membership in the Apple Developer Program or be invited to an Apple Developer Enterprise Program team with access to the proper certificates.

You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.

Xcode

You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.

Developer ID Certificates

There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.

To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.

We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.

If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.

Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:

% security find-identity -p basic -v

This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.

You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"

The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.

Application Specific Password for your Developer Account

Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.

Create a new application specific password in Apple ID portal for your developer account.

You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:

  • Keychain Item Name: Developer-altool
  • Account Name: your developer account email
  • Password: the application-specific password you just created

This will create a developer specific password item that we can access safely from the tools.

If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.

A Command Line Tool Project

You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.

My sample project for this article will be named “hello.”

Preparing the Xcode Project

The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.

Choosing the proper signing certificates

Before you can notarize the command line tool, it needs to be signed with the correct certificates.

  1. in Xcode, select the blue project icon in the left sidebar
  2. select the black “terminal” icon with your project’s name under the “Targets” list entry
  3. make sure the ‘General’ tab is selected
  4. under ‘Signing’ disable ‘Automatically manage signing’
  5. under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
  6. under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Setting the certificates
Setting the certificates

Enable Hardened Runtime

Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.

  1. from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
  2. click on ‘All’ to show all available settings
  3. enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
  4. set the value in the project column (blue icon) to YES
Enable Hardened Runtime
Enable Hardened Runtime

Change the Install Build Location

If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:

  1. in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
  2. double click on the value in the Project column (blue icon), this will open a popup window
  3. change the value to $SRCROOT/build/pkgroot
Change the Installation Build location
Change the Installation Build location

If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)

Build the Binary

You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:

  1. open Terminal and change directory to the project folder
  2. % xcodebuild clean install

This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.

After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.

/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.

Build the pkg

Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.

We can use the pkgroot directory as our payload to build the installer package:

% pkgbuild --root build/pkgroot \
           --identifier "com.example.hello" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Armin Briegel (ABCD123456)" \
           build/hello-1.0.pkg

I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.

This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.

If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”

Notarizing the Installer Package

Xcode has a command line tool altool which you can use to upload your tool for notarization:

xcrun altool --notarize-app \
             --primary-bundle-id "com.example.com" \
             --username "username@example.com" \
             --password "@keychain:Developer-altool" \
             --asc-provider "ABCD123456" \
             --file "build/hello-1.0.pkg"

The username is your developer account email.

The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.

The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)

This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:

xcrun altool --notarization-info "Your-Request-UUID" \
             --username "username@example.com" \                                    
             --password "@keychain:Developer-altool"   

Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.

Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.

Completing the Process

You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.

If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.

You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:

% xcrun stapler staple build/hello-1.0.pkg

You can also use stapler to verify the process went well:

% xcrun stapler validate build/hello-1.0.pkg

But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:

% spctl --assess -vvv --type install build/hello-1.0.pkg

Automating the Process

Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.

The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.

You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.

Links and Videos

These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.

Going forward

Notarization is a key part of Apple’s security strategy going in macOS.

As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.

If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.

On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.

Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.

Check Installer Pkgs for deprecated scripts

macOS 10.15 Catalina will deprecate the built-in /bin/bash. I have talked about this at length.

The release notes for Catalina also tell us that other built-in scripting runtimes, namely Python, Perl, and Ruby. Will not be included in future macOS releases (post-Catalina) any more.

This means, that if you want to use bash, Python, Perl, or Ruby on macOS, you will have to install, and maintain your own version in the future.

However, scripts in installation packages, cannot rely on any of these interpreters being available in future, post-Catalina versions of macOS. Installer pkgs can be run in all kinds of environments and at all times, and you would not want them to fail, because a dependency is missing.

The good news is that we still have time. All the runtimes mentioned above are still present in Catalina, so the packages will continue to work for now. But if you are building installation scripts, you need to check if any of the installation scripts use one of these interpreters and fix them.

I recommend to use /bin/sh for installation scripts, since that will run in any macOS context, even the Recovery system.

If you are using third-party installer packages, you may also want to check them for these interpreters, and notify the developer that these packages will break in future versions of macOS.

To check a flat installer package, you would expand it with pkgutil --expand and then look at script files in the Scripts folder. This will work fine for a package or two, but gets tedious really quickly, especially with large distribution pkgs with many components (e.g. Office).

So… I wrote a script to do it. The script should handle normal component pkgs, distribution pkgs and the legacy bundle pkgs and mpkgs.

You can get the pkgcheck script from my Github repo.

What the script does

Once I had written the code to inspect all these types of pkgs, I realized I could grab all other kinds of information, as well. The pkgcheck.sh script will check for:

  • Signature and Notarization
  • Type of Package: Component, Distribution, legacy bundle or mpkg
  • Identifier and version (when present)
  • Install-location
  • for Distribution and mpkg types, shows the information for all components as well
  • for every script in a pkg or component, checks the first line of the script for shebangs of the deprecated interpreters (/bin/bash, /usr/bin/python, /usr/bin/perl, and /usr/bin/ruby) and print a warning when found

How to run pkgcheck.sh

Run the script with the target pkg file as an argument:

% ./pkgcheck.sh sample.pkg

You can give more than one file:

% ./pkgcheck.sh file1.pkg file2.pkg ...

When you pass a directory, pkgcheck.sh will recursively search for all files or bundle directories with the pkg or mpkg extension in that directory:

% ./pkgcheck.sh SamplePkgs

Features and Errors

There are a few more things that I think might be useful to check in this script. Most of all, I want to add an indicator whether a component is enabled by default or not. If you can think of any other valuable data to display, let me know. (Issue or Pull Request or just ping me on MacAdmins Slack)

I have tested the script against many pkgs that I came across. However, there are likely edge cases that I haven’t anticipated, which might break the script. If you run into any of those, let me know. (File an Issue or Pull Request.) Having the troublesome pkg would of course be a great help.

Note: the script will create a scratch directory for temporary file extractions. The script doesn’t actually expand the entire pkg file, only the Scripts sub-archive. The scratch folder will be cleaned out at the beginning of the next run, but not when the script ends, as you might want to do some further inspections.

Sample outputs

This is a sample pkg I build in my book, it has pre- and postinstall scripts using a /bin/bash shebang:

% ./pkgcheck.sh SourceCodePro-2.030d.pkg
SourceCodePro-2.030d
SamplePkgs/SourceCodePro-2.030d.pkg
Signature:      None
Notarized:      No
Type:           Flat Component PKG
Identifier:     com.example.SourceCodePro
Version:        2.030d
Location:       /
Contains 2 resource files
postinstall has shebang #!/bin/bash
preinstall has shebang #!/bin/bash

This is the experimental notarized pkg installer for desktoppr:

% ./pkgcheck.sh desktoppr-0.2.pkg
desktoppr-0.2
SamplePkgs/desktoppr-0.2.pkg
Signature:      Developer ID Installer: Armin Briegel (JME5BW3F3R)
Notarized:      Yes
Type:           Flat Component PKG
Identifier:     com.scriptingosx.desktoppr
Version:        0.2
Contains 0 resource files

And finally, this is a big one, the Microsoft Office installer: (they have some work to do to clean up those scripts)

% ./pkgcheck.sh Microsoft\ Office\ 16.27.19071500_Installer.pkg
Microsoft Office 16.27.19071500_Installer
SamplePkgs/Microsoft Office 16.27.19071500_Installer.pkg
Signature:      Developer ID Installer: Microsoft Corporation (UBF8T346G9)
Notarized:      No
Type:           Flat Distribution PKG
Contains 11 component pkgs

    Microsoft_Word_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Word.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 3 resource files

    Microsoft_Excel_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Excel.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_PowerPoint_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_PowerPoint.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_OneNote_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_OneNote.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_Outlook_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Outlook.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    OneDrive
    Type:           Flat Component PKG
    Identifier:     com.microsoft.OneDrive
    Version:        19.70.410
    Location:       /Applications
    Contains 30 resource files
    postinstall has shebang #!/bin/bash
    od_logging has shebang #!/bin/bash
    od_service has shebang #!/bin/bash
    od_migration has shebang #!/bin/bash
    preinstall has shebang #!/bin/bash

    Office16_all_autoupdate
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_AutoUpdate.app
    Version:        4.13.19071500
    Location:       /Library/Application Support/Microsoft/MAU2.0
    Contains 2 resource files
    postinstall has shebang #!/bin/bash
    preinstall has shebang #!/bin/bash

    Office16_all_licensing
    Type:           Flat Component PKG
    Identifier:     com.microsoft.pkg.licensing
    Version:        16.27.19071500
    Location:       /
    Contains 2 resource files
    dockutil has shebang #!/usr/bin/python

    Office_fonts
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.DFonts
    Version:        0
    Location:       /private/tmp/com.microsoft.package.DFonts
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

    Office_frameworks
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Frameworks
    Version:        0
    Location:       /private/tmp/com.microsoft.package.Frameworks
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

    Office_proofing
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Proofing_Tools
    Version:        0
    Location:       /private/tmp/com.microsoft.package.Proofing_Tools
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

Moving to zsh – part 7: Miscellanea

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

We have covered the general aspects of configuring your zsh environment and enabling some of its features to make your work more productive. However, there are zsh features that didn’t quite fit in earlier posts, but also don’t warrant a post of their own. So I am gathering them here.

multiIO

Terminal commands can take input from a file or a previous command (stdin) and have two different outputs: stdout and stderr. In bash you can redirect each of these to a single other destination.

For example, you can redirect the output of a command to a file:

% system_profiler SPHardwareDataType >hardwareinfo.txt

In zsh you can redirect to (and from) multiple sources. In the simplest form you can write the output to two files:

% system_profiler SPHardwareDataType >hardwareinfo.txt >computerinfo.txt

This is of course not a very realistic case. Since the pipe | is a form of redirection, you can combine output to a file with a pipe:

% system_profiler SPHardwareDataType >hardwareprofile.txt | cat

Instead of piping to cat, you can also redirect to stdout (or &1) as well as to a file:

% system_profiler SPHardwareDataType >&1 >hardwareprofile.txt

Note that the order of doing this is important. The construct >file.txt >&1 would redirect the output to file.txt and then redirect the output again to where stdout or 1 is going, so it would be redundant.

When combined with pipes and other commands multiIO can become very useful:

% system_profiler SPHardwareDataType >hardwareprofile.txt | awk '/Serial Number/ { print $4 }' >&1 >serialnumber.txt

You can use multiIO for input as well:

% sort </usr/share/calendar/calendar.freebsd </usr/share/calendar/calendar.computer

And while this not directly related, but somewhat close, in zsh, this

% <hardwareinfo.txt

is equivalent to more hardwareinfo.txt.

Recursive Globbing with **

You can use the ** to denote an arbitrary string that can span multiple directories in a path.

For example:

% echo Library/Preferences/**/com.apple.screensaver.*plist
Library/Preferences/ByHost/com.apple.screensaver.BBCCDDEE-AABB-CCDD-ABCD-00AABBCCDDEE.plist Library/Preferences/com.apple.screensaver.plist

In this case the ** matches nothing as well as /ByHost/.

Note: when used on large folder structures this glob can take a while. So use with care.

Connected array Variables

We already encountered the fpath variable in earlier posts. You can see its contents with the echo command:

% echo $fpath
/Users/armin/Projects/mac-zsh-completions/completions/ /Users/armin/Projects/dotfiles/zshfunctions /usr/local/share/zsh/site-functions /usr/share/zsh/site-functions /usr/share/zsh/5.3/functions

Interestingly enough, zsh also has an FPATH variable, which is a colon-separated list of directories:

% echo $FPATH
/Users/armin/Projects/mac-zsh-completions/completions/:/Users/armin/Projects/dotfiles/zshfunctions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.3/functions

Since the fpath variable is an array, I only changed the fpath variable in my zshrc.I never set or changed the FPATH, yet it reflects the changes made to the fpath variable.

When you see the type of both variables, you get an idea that something is going on:

% echo ${(t)fpath}
array-special
% echo ${(t)FPATH}
scalar-special

The fpath and FPATH are connected in zsh. Changes to one affect the other. This allows use of more flexible and powerful array operations through the fpath ‘aspect’ of the value, but also provides compatibility to tools that expect the traditional colon-separated format in FPATH.

You will not be surprised to hear that zsh uses the same ‘magic’ with the PATH variable and its array counterpart path.

This means that you can continue to use path_helper to get your PATH from the files in /etc/paths and /etc/paths.d. (Well, you don’t have to, because on macOS this is done for all users in /etc/zprofile.) But then you can manipulate the path variable with array functions, like:

path+=~/bin

You get the useful aspects of both syntaxes.

Suffix Aliases

I learnt this one after writing the aliases part.

Suffix aliases take effect on the last part of a path, so usually the file extension. A suffix alias will assign a command to use when you just type a file path in the command line.

For example, you can a suffix alias for the txt file extension:

alias -s txt="open -t"

When you then type a path ending with .txt and no command, zsh will execute open -t /path/to/file.txt.

The open -t command opens a file in the default application set for the txt file extension in Finder. You probably want to set the suffix alias to bbedit or atom or something like that rather than open -t.

You can use other command line tools for the suffix alias:

alias -s log="tail -f"

Then, typing /var/log/install.log will show the last lines of that file and update the output when the file changes. If you prefer the graphical user interface, you can use the open -a command to assign suffix aliases to applications:

alias -s log="open -a Console"

You can even create a suffix alias using a different alias:

alias pacifist="open -a Pacifist"
alias -s pkg=pacifist

Together with the AutoCD option, this can improve your application-shell interactions a lot.

Bindkey for History Search

Most of the keyboard shortcuts in zsh work the same way as they do in bash. I have found one change that has proven quite useful:

^[[A' up-line-or-search # up arrow bindkey

^[[B' down-line-or-search # down arrow 

These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.

There are many commands or ‘widgets’ you can assign to keystrokes with the bindkey command. You can find a list of default ‘widgets’ in the documentation.

Conclusion

This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.

After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.

The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.

In the next (and last) post, I will cover the changes when scripting with zsh vs bash.

Moving to zsh, part 6 – Customizing the zsh Prompt

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

The default bash prompt on macOS is quite elaborate. It shows the username, the hostname, and the current directory.

Calypso:~ armin$

On the other hand, the default bash prompt doesn’t show the previous command’s exit code, a piece of information I find very useful. I have written before how I re-configured my bash prompt to have the information I want:

Of course, I wanted to recreate the same experience in zsh.

Minimal Zsh Prompt

The only (visual) difference to my bash prompt is the % instead of the $.

Note: creating a file ~/.hushlogin will suppress the status message at the start of each Terminal session in zsh as well as in bash (or any other shell).

Basic Prompt Configuration

The basic zsh prompt configuration works similar to bash, even though it uses a different syntax. The different placeholders are described in detail in the zsh manual.

zsh uses the same shell variable PS1 to store the default prompt. However, the variable names PROMPT and prompt are synonyms for PS1 and you will see either of those three being used in various examples. I am going to use PROMPT.

The default prompt in zsh is %m%#. The %m shows the first element of the hostname, the %# shows a # when the current prompt has super-user privileges (e.g. after a sudo -s) and otherwise the % symbol (the default zsh prompt symbol).

The zsh default prompt is far shorter than the bash default, but even less useful. Since I work on the local system most of the time, the hostname bears no useful information, and repeating it every line is superfluous.

Note: you can argue that the hostname in the prompt is useful when you frequently have multiple terminal windows open to different hosts. This is true, but then the prompt is defined by the remote shell and its configuration files on the remote host. In your configuration file, you can test if the SSH_CLIENT variable is set and show a different prompt for remote sessions. There are more ways of showing the host in remote shell sessions, for example in the Terminal window title bar or with different window background colors.

In our first iteration, I want to show the current working directory instead of the hostname. When you look through the list of prompt placeholders in the zsh documentation, you find %d, %/, and %~. The first two do exactly the same. The last substitution will display a path that starts with the user’s home directory with the ~, so it will shorten /Users/armin/Projects/ to ~/Projects.

Note: in the end you want to set your PROMPT variable in the .zshrc file, so it will take effect in all your zsh sessions. For testing, however, you can just change the PROMPT variable in the interactive shell. This will give you immediate feedback, how your current setup works.

% PROMPT='%/ %# '
/Users/armin/Projects/dotfiles/zshfunctions % 

% PROMPT='%~ %# '
~/Projects/dotfiles/zshfunctions % 

Note the trailing space in the prompt string, to separate the final % or # from the command entry.

I prefer the shorter output of the %~ option, but it can still be quite long, depending on your working directory. zsh has a trick for this: when you insert a number n between the % and the ~, then only the last n elements of the path will be shown:

% PROMPT='%2~ %# '
dotfiles/zshfunctions %                       

When you do %1~ it will show only the name of the working directory or ~ if it is the home directory. (This also works with %/, e.g. %2/.)

Adding Color

Adding a bit of color or shades of gray to the prompt can make it more readable. In bash you need cryptic escape codes to switch the colors. zsh provides an easier way. To turn the directory in the path blue, you can use:

PROMPT='%F{blue}%1~%f %# '

The F stands for ‘Foreground color.’ zsh understands the colors black, red, green, yellow, blue, magenta, cyan and white. %F or %f resets to the default text color. Furthermore, Terminal.app represents itself as a 256-color terminal to the shell. You can verify this with

% echo $TERM
xterm-256color

You can access the 256 color pallet with %F{0} through %F{255}. There are tables showing which number maps to which color:

So, since I want a dark gray for my current working dir in my prompt, I chose 240, I also set it to bold with the %B code:

PROMPT='%B%F{240}%1~%f%b %# '

You can find a detailed list of the codes for visual effects in the documentation.

Dynamic Prompt

I wrote an entire post on how to get bash to show the color-coded exit code of the last command. As it turns out, this is much easier in zsh.

One of the prompt codes provides a ‘ternary conditional,’ which means it will show one of two expressions, depending on a condition. There are several conditions you can use. Once again the details can be found in the documentation.

There is one condition for the previous commands exit code:

%(?.<success expression>.<failure expression>)

This expression will use the <success expression> when the previous command exited successfully (exit code zero) and <failure expression> when the previous command failed (non-zero exit code). So it is quite easy to build an conditional prompt:

% PROMPT='%(?.√.?%?) %1~ %# ' 
√ ~ % false
?1 ~ % 

You can get the character with option-V on the US or international macOS keyboard layout. The last part of the ternary ?%? looks confusing. The first ? will print a literal question mark, and the second part %? will be replaced with previous command’s exit code.

You can add colors in the ternary expression as well:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# ' 

Another interesting conditional code is ! which returns whether the shell is privileged (i.e. running as root) or not. This allows us to change the default prompt symbol from % to something else, while maintaining the warning functionality when running as root:

% PROMPT='%1~ %(!.#.>) ' 
~ > sudo -s
~ # exit
~ > 

Complete Prompt

Here is the complete prompt we assembled, with all the parts explained:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# '
%(?.√.?%?) if return code ? is 0, show , else show ?%?
%? exit code of previous command
%1~ current working dir, shortening home to ~, show only last 1 element
%# # with root privileges, % otherwise
%B %b start/stop bold
%F{...} text (foreground) color, see table
%f reset to default textcolor

Right Sided Prompt

zsh also offers a right sided prompt. It uses the same placeholders as the ‘normal’ prompt. Use the RPROMPT variable to set the right side prompt:

% RPROMPT='%*'
√ zshfunctions %                    11:02:55

zsh will automatically hide the right prompt when the cursor reaches it when typing a long command. You can use all the other substitutions from the left side prompt, including colors and other visual markers in the right side prompt.

Git Integration

zsh includes some basic integration for version control systems. Once again there is a voluminous, but hard to understand description of it in the documentation.

I found a better, more specific example in the ‘Pro git’ documentation. This example will show the current branch on the right side prompt.

I have changed the example to include the repo name and the branch, and to change the color.

autoload -Uz vcs_info
precmd_vcs_info() { vcs_info }
precmd_functions+=( precmd_vcs_info )
setopt prompt_subst
RPROMPT=\$vcs_info_msg_0_
zstyle ':vcs_info:git:*' formats '%F{240}(%b)%r%f'
zstyle ':vcs_info:*' enable git

In this case %b and %r are placeholders for the VCS (version control system) system for the branch and the repository name.

There are git prompt solutions other than the built-in module, which deliver more information. There is a script in the git repository, and many of the larger zsh theme projects, such as ‘oh-my-zsh’ and ‘prezto’ have all kinds of git status widgets or modules or themes or what ever they call them.

Summary

You can spend (or waste) a lot of time on fine-tuning your prompt. Whether these modifications really improve your productivity is a matter of opinion.

In the next post, we will cover some miscellaneous odds and ends that haven’t yet really fit into any of preceding posts.