Make zsh show working directory in Terminal window title in macOS

While I was working on customizing my zsh configuration files for my zsh article series, I noticed that Terminal would not display the current working directory when using zsh. it worked fine when I switched back to bash.

Note: showing the working directory in the window or tab title is enabled by default in Terminal. You can configure what Terminal shows in the title in the Preferences Window for each profile. Select a profile and

Some sleuthing showed me that /etc/bashrc_Apple_Terminal sets up a update_terminal_cwd function which sends some escape codes to Terminal to keep the window title bar updated. This file is sourced by /etc/bashrc for the Terminal.app.

In macOS 10.14 Mojave and earlier, this configuration file has no equivalent for zsh. /etc/zshrc has code that would load /etc/zshrc_Apple_Terminal, if it existed. However, this file does not exist in macOS Mojave and earlier.

It does in Catalina, though, and it sets up a similar function that is added into the precmd hook. If you have access to the Catalina beta, you can just copy the /etc/zshrc_Apple_Terminal to your Mojave (or earlier) Mac and it will work.

Alternatively, you can write your own implementation, which is what I did, because I wanted it before the Catalina files existed. My solution exists of the function and its setup in its own file. This file is in a directory that is added to the fpath so I can simply load it with autoload.

You can see the code for the function file here.

Then, in my .zshrc I load the function file with:

# path to my zsh functions folder:
my_zsh_functions=$repo_dir/dotfiles/zshfunctions/

# include my zshfunctions dir in fpath:
if [[ -d $my_zsh_functions ]]; then
    fpath=( $my_zsh_functions $fpath )
fi

# only for Mojave and earlier
if [[ $(sw_vers -buildVersion) < "19" ]]; then 
    # this sets up the connection with the Apple Terminal Title Bar
    autoload -U update_terminal_pwd && update_terminal_pwd
fi

And from now, the title will be updated in Mojave (and earlier) and the function won’t even be loaded in Catalina.

This might seem a bit overkill, now that the functionality is in the Catalina file, but it might serve as a useful example when you want to customize, or re-implement similar behavior.

Notarize a Command Line Tool

The upcoming macOS 10.15 Catalina will require more apps and tools to be notarized. Apple has some loosened the requirements at last minute, but these changed limitations are only temporary, to give developers more time to adapt.

Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.

Howard Oakley has written up his experiences and that post was extremely helpful. But there were a few omissions and some steps that aren’t really necessary, so I decided to make my own write-up.

And yes, there is a script at the end…

Note: these instructions are for macOS 10.14.6 Mojave, Xcode 10.3 and Swift 5.0. It is very likely that the details will change over time.

What do you need?

  • Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates)
  • Xcode 10.3
  • Developer ID Certificates (Application and Install)
  • Application Specific Password for your Developer account
  • a Command Line Tool Project that you want to sign and notarize

That’s a longish list. If you are already building command line tools in Xcode, you should have most of these covered already. We will walk through the list step-by-step:

Apple Developer Program Account

You need either the paid membership in the Apple Developer Program or be invited to an Apple Developer Enterprise Program team with access to the proper certificates.

You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.

Xcode

You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.

Developer ID Certificates

There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.

To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.

We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.

If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.

Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:

% security find-identity -p basic -v

This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.

You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"

The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.

Application Specific Password for your Developer Account

Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.

Create a new application specific password in Apple ID portal for your developer account.

You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:

  • Keychain Item Name: Developer-altool
  • Account Name: your developer account email
  • Password: the application-specific password you just created

This will create a developer specific password item that we can access safely from the tools.

If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.

A Command Line Tool Project

You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.

My sample project for this article will be named “hello.”

Preparing the Xcode Project

The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.

Choosing the proper signing certificates

Before you can notarize the command line tool, it needs to be signed with the correct certificates.

  1. in Xcode, select the blue project icon in the left sidebar
  2. select the black “terminal” icon with your project’s name under the “Targets” list entry
  3. make sure the ‘General’ tab is selected
  4. under ‘Signing’ disable ‘Automatically manage signing’
  5. under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
  6. under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Setting the certificates
Setting the certificates

Enable Hardened Runtime

Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.

  1. from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
  2. click on ‘All’ to show all available settings
  3. enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
  4. set the value in the project column (blue icon) to YES
Enable Hardened Runtime
Enable Hardened Runtime

Change the Install Build Location

If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:

  1. in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
  2. double click on the value in the Project column (blue icon), this will open a popup window
  3. change the value to $SRCROOT/build/pkgroot
Change the Installation Build location
Change the Installation Build location

If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)

Build the Binary

You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:

  1. open Terminal and change directory to the project folder
  2. % xcodebuild clean install

This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.

After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.

/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.

Build the pkg

Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.

We can use the pkgroot directory as our payload to build the installer package:

% pkgbuild --root build/pkgroot \
           --identifier "com.example.hello" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Armin Briegel (ABCD123456)" \
           build/hello-1.0.pkg

I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.

This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.

If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”

Notarizing the Installer Package

Xcode has a command line tool altool which you can use to upload your tool for notarization:

xcrun altool --notarize-app \
             --primary-bundle-id "com.example.com" \
             --username "username@example.com" \
             --password "@keychain:Developer-altool" \
             --asc-provider "ABCD123456" \
             --file "build/hello-1.0.pkg"

The username is your developer account email.

The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.

The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)

This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:

xcrun altool --notarization-info "Your-Request-UUID" \
             --username "username@example.com" \                                    
             --password "@keychain:Developer-altool"   

Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.

Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.

Completing the Process

You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.

If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.

You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:

% xcrun stapler staple build/hello-1.0.pkg

You can also use stapler to verify the process went well:

% xcrun stapler validate build/hello-1.0.pkg

But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:

% spctl --assess -vvv --type install build/hello-1.0.pkg

Automating the Process

Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.

The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.

You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.

Links and Videos

These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.

Going forward

Notarization is a key part of Apple’s security strategy going in macOS.

As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.

If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.

On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.

Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.

Check Installer Pkgs for deprecated scripts

macOS 10.15 Catalina will deprecate the built-in /bin/bash. I have talked about this at length.

The release notes for Catalina also tell us that other built-in scripting runtimes, namely Python, Perl, and Ruby. Will not be included in future macOS releases (post-Catalina) any more.

This means, that if you want to use bash, Python, Perl, or Ruby on macOS, you will have to install, and maintain your own version in the future.

However, scripts in installation packages, cannot rely on any of these interpreters being available in future, post-Catalina versions of macOS. Installer pkgs can be run in all kinds of environments and at all times, and you would not want them to fail, because a dependency is missing.

The good news is that we still have time. All the runtimes mentioned above are still present in Catalina, so the packages will continue to work for now. But if you are building installation scripts, you need to check if any of the installation scripts use one of these interpreters and fix them.

I recommend to use /bin/sh for installation scripts, since that will run in any macOS context, even the Recovery system.

If you are using third-party installer packages, you may also want to check them for these interpreters, and notify the developer that these packages will break in future versions of macOS.

To check a flat installer package, you would expand it with pkgutil --expand and then look at script files in the Scripts folder. This will work fine for a package or two, but gets tedious really quickly, especially with large distribution pkgs with many components (e.g. Office).

So… I wrote a script to do it. The script should handle normal component pkgs, distribution pkgs and the legacy bundle pkgs and mpkgs.

You can get the pkgcheck script from my Github repo.

What the script does

Once I had written the code to inspect all these types of pkgs, I realized I could grab all other kinds of information, as well. The pkgcheck.sh script will check for:

  • Signature and Notarization
  • Type of Package: Component, Distribution, legacy bundle or mpkg
  • Identifier and version (when present)
  • Install-location
  • for Distribution and mpkg types, shows the information for all components as well
  • for every script in a pkg or component, checks the first line of the script for shebangs of the deprecated interpreters (/bin/bash, /usr/bin/python, /usr/bin/perl, and /usr/bin/ruby) and print a warning when found

How to run pkgcheck.sh

Run the script with the target pkg file as an argument:

% ./pkgcheck.sh sample.pkg

You can give more than one file:

% ./pkgcheck.sh file1.pkg file2.pkg ...

When you pass a directory, pkgcheck.sh will recursively search for all files or bundle directories with the pkg or mpkg extension in that directory:

% ./pkgcheck.sh SamplePkgs

Features and Errors

There are a few more things that I think might be useful to check in this script. Most of all, I want to add an indicator whether a component is enabled by default or not. If you can think of any other valuable data to display, let me know. (Issue or Pull Request or just ping me on MacAdmins Slack)

I have tested the script against many pkgs that I came across. However, there are likely edge cases that I haven’t anticipated, which might break the script. If you run into any of those, let me know. (File an Issue or Pull Request.) Having the troublesome pkg would of course be a great help.

Note: the script will create a scratch directory for temporary file extractions. The script doesn’t actually expand the entire pkg file, only the Scripts sub-archive. The scratch folder will be cleaned out at the beginning of the next run, but not when the script ends, as you might want to do some further inspections.

Sample outputs

This is a sample pkg I build in my book, it has pre- and postinstall scripts using a /bin/bash shebang:

% ./pkgcheck.sh SourceCodePro-2.030d.pkg
SourceCodePro-2.030d
SamplePkgs/SourceCodePro-2.030d.pkg
Signature:      None
Notarized:      No
Type:           Flat Component PKG
Identifier:     com.example.SourceCodePro
Version:        2.030d
Location:       /
Contains 2 resource files
postinstall has shebang #!/bin/bash
preinstall has shebang #!/bin/bash

This is the experimental notarized pkg installer for desktoppr:

% ./pkgcheck.sh desktoppr-0.2.pkg
desktoppr-0.2
SamplePkgs/desktoppr-0.2.pkg
Signature:      Developer ID Installer: Armin Briegel (JME5BW3F3R)
Notarized:      Yes
Type:           Flat Component PKG
Identifier:     com.scriptingosx.desktoppr
Version:        0.2
Contains 0 resource files

And finally, this is a big one, the Microsoft Office installer: (they have some work to do to clean up those scripts)

% ./pkgcheck.sh Microsoft\ Office\ 16.27.19071500_Installer.pkg
Microsoft Office 16.27.19071500_Installer
SamplePkgs/Microsoft Office 16.27.19071500_Installer.pkg
Signature:      Developer ID Installer: Microsoft Corporation (UBF8T346G9)
Notarized:      No
Type:           Flat Distribution PKG
Contains 11 component pkgs

    Microsoft_Word_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Word.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 3 resource files

    Microsoft_Excel_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Excel.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_PowerPoint_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_PowerPoint.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_OneNote_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_OneNote.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_Outlook_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Outlook.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    OneDrive
    Type:           Flat Component PKG
    Identifier:     com.microsoft.OneDrive
    Version:        19.70.410
    Location:       /Applications
    Contains 30 resource files
    postinstall has shebang #!/bin/bash
    od_logging has shebang #!/bin/bash
    od_service has shebang #!/bin/bash
    od_migration has shebang #!/bin/bash
    preinstall has shebang #!/bin/bash

    Office16_all_autoupdate
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_AutoUpdate.app
    Version:        4.13.19071500
    Location:       /Library/Application Support/Microsoft/MAU2.0
    Contains 2 resource files
    postinstall has shebang #!/bin/bash
    preinstall has shebang #!/bin/bash

    Office16_all_licensing
    Type:           Flat Component PKG
    Identifier:     com.microsoft.pkg.licensing
    Version:        16.27.19071500
    Location:       /
    Contains 2 resource files
    dockutil has shebang #!/usr/bin/python

    Office_fonts
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.DFonts
    Version:        0
    Location:       /private/tmp/com.microsoft.package.DFonts
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

    Office_frameworks
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Frameworks
    Version:        0
    Location:       /private/tmp/com.microsoft.package.Frameworks
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

    Office_proofing
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Proofing_Tools
    Version:        0
    Location:       /private/tmp/com.microsoft.package.Proofing_Tools
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

Shellcheck and zsh

The sad news is that the shellcheck binary does not really know how to deal with zsh scripts:

% shellcheck script.zsh

In script.zsh line 1:
#!/bin/zsh
^-- SC1071: ShellCheck only supports sh/bash/dash/ksh scripts. Sorry!

You can force shellcheck to interpret a zsh script with the --shell option:

% shellcheck --shell=bash dirlist.sh

This might show errors, such as Quote to prevent word splitting/globbing which aren’t really errors in zsh so its usefulness will be limited.

If you want a specific script to be always interpreted as a bash script by shellcheck you can also add this line:

# shellcheck shell=bash

in the script, right after the shebang line.

This can be useful if you use zsh in bash emulation mode with

emulate -LR bash

Moving to zsh, part 8 – Scripting zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class in Amsterdam on September 6. We will cover the ‘why’ and ‘how’ of zsh and show lots of practical examples how using zsh will improve your Terminal productivity. More info and sign-up on the Pro Warehouse webpage.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This is the final article in this series. (If I ever announce an eight part series again, please somebody intervene!) However, I am quite sure it will not be the last post on zsh

All the previous posts described how zsh works as an interactive shell. The interactive shell is of course the most direct way we use a shell and configuring the shell to your taste can bring a huge boost in usefulness and productivity.

The other, equally, important aspect of a shell is running script files. In the simplest perspective, script files are just series of interactive commands, but of course they will get complex very quickly.

sh, bash, or zsh?

Should you even script in zsh? The argument for bash has been that it has been pre-installed on every Mac OS X since 10.2. The same is true for zsh, with one exception: the zsh binary is not present on the Recovery system. It is also not present on a NetInstall or External Installation System, but these are less relevant in a modern deployment workflow, which has to work with Secure Boot Macs.

If you plan to run a script from Recovery, such as an installr or bootstrappr script or as part of an MDS workflow, your only choices are /bin/sh and /bin/bash. The current /bin/bash binary is 12 years old, and Apple is messaging its demise. I would not consider that a future proof choice. So, if your script may run in a Recovery context, i would recommend /bin/sh over either /bin/bash or /bin/zsh

Since installation packages can be run from the Recovery context as well, and you cannot really always predict in which context your package will be used, I would extend the recommendation to use /bin/sh for all installation scripts as well.

While sh is surely ubiquitous, it is also a ‘lowest common denominator’, so it is not a very comfortable scripting language to work in. I recommend using shellcheck to verify all your sh scripts for bashisms that might have crept in out of habit.

When you can ensure your script will only run on a full macOS installation, zsh is good choice over sh. It is pre-installed on macOS, and it offers better and safer language options than sh and some advantages over bash, too. Deployment scripts, scripts pushed from management systems, launch daemons and launch agents, and script you write to automate your admin workflows (such as building packages) would fall in this category.

You can also choose to stick with bash, but then you should start installing and using your own bash 5 binary instead of the built in /bin/bash. This will give you newer security updates and features and good feeling that when Apple does eventually yank the /bin/bash binary, your scripts will keep working.

Admins who want to keep using Python for their scripts are facing a similar problem. Once you choose to use a non-system version of bash (or python), it is your responsibility to install and update it on all your clients. But that is what system management tools are for. We will have to get used to managing our own tools as well as the users’ tools, instead of relying on Apple.

Shebang

To switch your script from using bash to zsh, you have to change the shebang in the first line from #!/bin/bash to #!/bin/zsh.

If you want to distinguish your zsh script files, the you can also change the script’s file extension from .sh to .zsh. This will be especially helpful while you transfer scripts from bash to zsh (or sh). The file extension will have no effect on which interpreter will be used to run the script. That is determined by the shebang, but the extension provides a visible clue in the Finder and Terminal.

zsh vs bash

Since zsh derives from the same Bourne shell family as bash does, most commands, syntax, and control structures will work just the same. zsh provides alternative syntax for some of the structures.

zsh has several options to control compatibility, not only for bash, but for other shells as well. We have already seen that options can be used to enable features specific for zsh. These options can significantly change how zsh interprets your scripts.

Because you can never quite anticipate in which environment your particular zsh will be launched in, it is good practice to reset the options at the beginning of your script with the emulate command:

emulate -LR zsh

After the emulate command, you can explicitly set the shell options your script requires.

The emulate command also provides a bash emulation:

emulate -LR bash

This will change the zsh options to closely emulate bash behavior. Rather than relying on this emulation mode, I would recommend actually using bash, even if you have to install and manage a newer version yourself.

Word Splitting in Variable Substitutions

Nearly all syntax from bash scripts will ‘just work’ in zsh as well. There are just a few important differences you have to be aware of.

The most significant difference, which will affect most scripts is how zsh treats word splitting in variable substitutions.

Recap: bash behavior

In bash substituted variables are split on whitespace when the substitution is not quoted. To demonstrate, we will use a function that counts the number of arguments passed into it. This way we can see whether a variable was split or not:

#!/bin/bash
export PATH=/usr/bin:/bin:/usr/sbin:/sbin

function countArguments() {
    echo "${#@}"
}

wordlist="one two three four five"

echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 5

echo "substitution with quotes"
countArguments "$wordlist"
# -> 1

In bash and sh the contents of the variable split into separate arguments when substituted without the quotes. Usually you do not want the splitting to occur. Hence the rule: “always quote variable substitutions!”

zsh behavior: no splitting

zsh will not split a variable when substituted. With zsh the contents of a variable will be kept in one piece:

#!/bin/zsh
emulate -LR zsh # reset zsh options
export PATH=/usr/bin:/bin:/usr/sbin:/sbin

function countArguments() {
    echo "${#@}"
}

wordlist="one two three four five"

echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 1

echo "substitution with quotes"
countArguments "$wordlist"
# -> 1

The positive effect of this is that you do not have to worry about quoting variables all the time, making zsh less error prone, and much more like other scripting and programming languages.

Splitting Arrays

The wordlist variable in our example above is a string. Because of this it returns a count of 1, since there is only one element, the string itself.

If you want to loop through multiple elements of a list

In bash this happens, whether you want to or not, unless you explicitly tell bash not to split by quoting the variable.

In zsh, you have to explicitly tell the shell to split a string into its components. If you do this naïvely, by wrapping the string variable in the parenthesis to declare and array, it will not work:

wordlist="one two three"
wordarray=( $wordlist )

for word in $wordarray; do
    echo "->$word<-"
done

#output
->one two three<-

Note: the for loop echoes every item in the array. I have added the -> characters to make the individual items more visible. In the subsequent examples, I will not repeat the for loop, but only show its output. So the above example will be shortened to:

wordarray=( $wordlist )
->one two three<-

There are few options to do this right.

Revert to sh behavior

First, you can tell zsh to revert to the bash or sh behavior and split on any whitespace. You can do this by pre-fixing the variable substitution with an =:

wordarray=( ${=wordlist} )
->one<-
->two<-
->three<-

Note: if you find yourself using the = frequently, you can also re-enable sh style word splitting with the shwordsplit option. This will of course affect all substitutions in the script until you disable the option again.

setopt shwordsplit
wordarray=( $wordlist )
->one<-
->two<-
->three<-

This option can be very useful when you quickly need to convert a bash script to a zsh script. But you will also re-enable all the problems you had with unintentional word splitting.

Splitting Lines

If you want to be more specific and split on particular characters, zsh has a special substitution syntax for that:

macOSversion=$(sw_vers -productBuild) # 10.14.6
versionParts=${(s/./)macOSVersion}
->10<-
->14<-
->6<-

If you want to split on a newline character \n the syntax is slightly different:

citytext="New York
Rio
Tokyo"

cityarray=( ${(ps/\n/)citytext} )
->New York<-
->Rio<-
->Tokyo<-

Since newline is a common character to split text on, there is a short cut:

cityarray=( ${(f)citytext} )

Since the newline character is a legal character in file names, you should use zero-terminated strings where possible:

foundDirs=$(find /Library -type d -maxdepth 1 -print0)
dirlist=${(ps/\0/)foundDirs}

Again, there is a shortcut for this:

dirlist=${(0)foundDirs}

Array index starts at 1

Once you have split text into an array, remember, that in zsh array indices start at 1:

% versionList=( ${(s/./)$(sw_vers -productVersion)} )
% echo ${versionList[1]}
10
% echo ${versionList[2]}
14
% echo ${versionList[3]}
6

If you think this is wrong and absolutely require a zero-based index, you can set the KSH_ARRAYS shell option:

% setopt KSH_ARRAYS
% echo ${versionList[0]}
10
% echo ${versionList[1]}
14
% echo ${versionList[2]}
6
% echo ${versionList[3]}

Conclusion

Switching your scripts from bash to zsh requires a bit more work than merely switching out the shebang. However, since /bin/bash will still be present in Catalina, you do not have to move all scripts immediately.

Moving to sh instead of zsh can be safer choice, especially for package installation scripts.

In zsh, there always seems to be some option to disable or enable a particular behavior.

This concludes my series on switching to zsh on macOS. I hope you found it helpful.

After having worked with zsh for a few weeks, I already find some of its features indispensable. I am looking forward to discovering and using more features over time. When I do, I will certainly share them here.

Moving to zsh – part 7: Miscellanea

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

We have covered the general aspects of configuring your zsh environment and enabling some of its features to make your work more productive. However, there are zsh features that didn’t quite fit in earlier posts, but also don’t warrant a post of their own. So I am gathering them here.

multiIO

Terminal commands can take input from a file or a previous command (stdin) and have two different outputs: stdout and stderr. In bash you can redirect each of these to a single other destination.

For example, you can redirect the output of a command to a file:

% system_profiler SPHardwareDataType >hardwareinfo.txt

In zsh you can redirect to (and from) multiple sources. In the simplest form you can write the output to two files:

% system_profiler SPHardwareDataType >hardwareinfo.txt >computerinfo.txt

This is of course not a very realistic case. Since the pipe | is a form of redirection, you can combine output to a file with a pipe:

% system_profiler SPHardwareDataType >hardwareprofile.txt | cat

Instead of piping to cat, you can also redirect to stdout (or &1) as well as to a file:

% system_profiler SPHardwareDataType >&1 >hardwareprofile.txt

Note that the order of doing this is important. The construct >file.txt >&1 would redirect the output to file.txt and then redirect the output again to where stdout or 1 is going, so it would be redundant.

When combined with pipes and other commands multiIO can become very useful:

% system_profiler SPHardwareDataType >hardwareprofile.txt | awk '/Serial Number/ { print $4 }' >&1 >serialnumber.txt

You can use multiIO for input as well:

% sort </usr/share/calendar/calendar.freebsd </usr/share/calendar/calendar.computer

And while this not directly related, but somewhat close, in zsh, this

% <hardwareinfo.txt

is equivalent to more hardwareinfo.txt.

Recursive Globbing with **

You can use the ** to denote an arbitrary string that can span multiple directories in a path.

For example:

% echo Library/Preferences/**/com.apple.screensaver.*plist
Library/Preferences/ByHost/com.apple.screensaver.BBCCDDEE-AABB-CCDD-ABCD-00AABBCCDDEE.plist Library/Preferences/com.apple.screensaver.plist

In this case the ** matches nothing as well as /ByHost/.

Note: when used on large folder structures this glob can take a while. So use with care.

Connected array Variables

We already encountered the fpath variable in earlier posts. You can see its contents with the echo command:

% echo $fpath
/Users/armin/Projects/mac-zsh-completions/completions/ /Users/armin/Projects/dotfiles/zshfunctions /usr/local/share/zsh/site-functions /usr/share/zsh/site-functions /usr/share/zsh/5.3/functions

Interestingly enough, zsh also has an FPATH variable, which is a colon-separated list of directories:

% echo $FPATH
/Users/armin/Projects/mac-zsh-completions/completions/:/Users/armin/Projects/dotfiles/zshfunctions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.3/functions

Since the fpath variable is an array, I only changed the fpath variable in my zshrc.I never set or changed the FPATH, yet it reflects the changes made to the fpath variable.

When you see the type of both variables, you get an idea that something is going on:

% echo ${(t)fpath}
array-special
% echo ${(t)FPATH}
scalar-special

The fpath and FPATH are connected in zsh. Changes to one affect the other. This allows use of more flexible and powerful array operations through the fpath ‘aspect’ of the value, but also provides compatibility to tools that expect the traditional colon-separated format in FPATH.

You will not be surprised to hear that zsh uses the same ‘magic’ with the PATH variable and its array counterpart path.

This means that you can continue to use path_helper to get your PATH from the files in /etc/paths and /etc/paths.d. (Well, you don’t have to, because on macOS this is done for all users in /etc/zprofile.) But then you can manipulate the path variable with array functions, like:

path+=~/bin

You get the useful aspects of both syntaxes.

Suffix Aliases

I learnt this one after writing the aliases part.

Suffix aliases take effect on the last part of a path, so usually the file extension. A suffix alias will assign a command to use when you just type a file path in the command line.

For example, you can a suffix alias for the txt file extension:

alias -s txt="open -t"

When you then type a path ending with .txt and no command, zsh will execute open -t /path/to/file.txt.

The open -t command opens a file in the default application set for the txt file extension in Finder. You probably want to set the suffix alias to bbedit or atom or something like that rather than open -t.

You can use other command line tools for the suffix alias:

alias -s log="tail -f"

Then, typing /var/log/install.log will show the last lines of that file and update the output when the file changes. If you prefer the graphical user interface, you can use the open -a command to assign suffix aliases to applications:

alias -s log="open -a Console"

You can even create a suffix alias using a different alias:

alias pacifist="open -a Pacifist"
alias -s pkg=pacifist

Together with the AutoCD option, this can improve your application-shell interactions a lot.

Bindkey for History Search

Most of the keyboard shortcuts in zsh work the same way as they do in bash. I have found one change that has proven quite useful:

bindkey $'^[[A' up-line-or-search    # up arrow
bindkey $'^[[B' down-line-or-search  # down arrow

These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.

There are many commands or ‘widgets’ you can assign to keystrokes with the bindkey command. You can find a list of default ‘widgets’ in the documentation.

Conclusion

This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.

After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.

The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.

In the next (and last) post, I will cover the changes when scripting with zsh vs bash.

Moving to zsh, part 6 – Customizing the zsh Prompt

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

The default bash prompt on macOS is quite elaborate. It shows the username, the hostname, and the current directory.

Calypso:~ armin$

On the other hand, the default bash prompt doesn’t show the previous command’s exit code, a piece of information I find very useful. I have written before how I re-configured my bash prompt to have the information I want:

Of course, I wanted to recreate the same experience in zsh.

Minimal Zsh Prompt

The only (visual) difference to my bash prompt is the % instead of the $.

Note: creating a file ~/.hushlogin will suppress the status message at the start of each Terminal session in zsh as well as in bash (or any other shell).

Basic Prompt Configuration

The basic zsh prompt configuration works similar to bash, even though it uses a different syntax. The different placeholders are described in detail in the zsh manual.

zsh uses the same shell variable PS1 to store the default prompt. However, the variable names PROMPT and prompt are synonyms for PS1 and you will see either of those three being used in various examples. I am going to use PROMPT.

The default prompt in zsh is %m%#. The %m shows the first element of the hostname, the %# shows a # when the current prompt has super-user privileges (e.g. after a sudo -s) and otherwise the % symbol (the default zsh prompt symbol).

The zsh default prompt is far shorter than the bash default, but even less useful. Since I work on the local system most of the time, the hostname bears no useful information, and repeating it every line is superfluous.

Note: you can argue that the hostname in the prompt is useful when you frequently have multiple terminal windows open to different hosts. This is true, but then the prompt is defined by the remote shell and its configuration files on the remote host. In your configuration file, you can test if the SSH_CLIENT variable is set and show a different prompt for remote sessions. There are more ways of showing the host in remote shell sessions, for example in the Terminal window title bar or with different window background colors.

In our first iteration, I want to show the current working directory instead of the hostname. When you look through the list of prompt placeholders in the zsh documentation, you find %d, %/, and %~. The first two do exactly the same. The last substitution will display a path that starts with the user’s home directory with the ~, so it will shorten /Users/armin/Projects/ to ~/Projects.

Note: in the end you want to set your PROMPT variable in the .zshrc file, so it will take effect in all your zsh sessions. For testing, however, you can just change the PROMPT variable in the interactive shell. This will give you immediate feedback, how your current setup works.

% PROMPT='%/ %# '
/Users/armin/Projects/dotfiles/zshfunctions % 

% PROMPT='%~ %# '
~/Projects/dotfiles/zshfunctions % 

Note the trailing space in the prompt string, to separate the final % or # from the command entry.

I prefer the shorter output of the %~ option, but it can still be quite long, depending on your working directory. zsh has a trick for this: when you insert a number n between the % and the ~, then only the last n elements of the path will be shown:

% PROMPT='%2~ %# '
dotfiles/zshfunctions %                       

When you do %1~ it will show only the name of the working directory or ~ if it is the home directory. (This also works with %/, e.g. %2/.)

Adding Color

Adding a bit of color or shades of gray to the prompt can make it more readable. In bash you need cryptic escape codes to switch the colors. zsh provides an easier way. To turn the directory in the path blue, you can use:

PROMPT='%F{blue}%1~%f %# '

The F stands for ‘Foreground color.’ zsh understands the colors black, red, green, yellow, blue, magenta, cyan and white. %F or %f resets to the default text color. Furthermore, Terminal.app represents itself as a 256-color terminal to the shell. You can verify this with

% echo $TERM
xterm-256color

You can access the 256 color pallet with %F{0} through %F{255}. There are tables showing which number maps to which color:

So, since I want a dark gray for my current working dir in my prompt, I chose 240, I also set it to bold with the %B code:

PROMPT='%B%F{240}%1~%f%b %# '

You can find a detailed list of the codes for visual effects in the documentation.

Dynamic Prompt

I wrote an entire post on how to get bash to show the color-coded exit code of the last command. As it turns out, this is much easier in zsh.

One of the prompt codes provides a ‘ternary conditional,’ which means it will show one of two expressions, depending on a condition. There are several conditions you can use. Once again the details can be found in the documentation.

There is one condition for the previous commands exit code:

%(?.<success expression>.<failure expression>)

This expression will use the <success expression> when the previous command exited successfully (exit code zero) and <failure expression> when the previous command failed (non-zero exit code). So it is quite easy to build an conditional prompt:

% PROMPT='%(?.√.?%?) %1~ %# ' 
√ ~ % false
?1 ~ % 

You can get the character with option-V on the US or international macOS keyboard layout. The last part of the ternary ?%? looks confusing. The first ? will print a literal question mark, and the second part %? will be replaced with previous command’s exit code.

You can add colors in the ternary expression as well:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# ' 

Another interesting conditional code is ! which returns whether the shell is privileged (i.e. running as root) or not. This allows us to change the default prompt symbol from % to something else, while maintaining the warning functionality when running as root:

% PROMPT='%1~ %(!.#.>) ' 
~ > sudo -s
~ # exit
~ > 

Complete Prompt

Here is the complete prompt we assembled, with all the parts explained:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# '
%(?.√.?%?) if return code ? is 0, show , else show ?%?
%? exit code of previous command
%1~ current working dir, shortening home to ~, show only last 1 element
%# # with root privileges, % otherwise
%B %b start/stop bold
%F{...} text (foreground) color, see table
%f reset to default textcolor

Right Sided Prompt

zsh also offers a right sided prompt. It uses the same placeholders as the ‘normal’ prompt. Use the RPROMPT variable to set the right side prompt:

% RPROMPT='%*'
√ zshfunctions %                    11:02:55

zsh will automatically hide the right prompt when the cursor reaches it when typing a long command. You can use all the other substitutions from the left side prompt, including colors and other visual markers in the right side prompt.

Git Integration

zsh includes some basic integration for version control systems. Once again there is a voluminous, but hard to understand description of it in the documentation.

I found a better, more specific example in the ‘Pro git’ documentation. This example will show the current branch on the right side prompt.

I have changed the example to include the repo name and the branch, and to change the color.

autoload -Uz vcs_info
precmd_vcs_info() { vcs_info }
precmd_functions+=( precmd_vcs_info )
setopt prompt_subst
RPROMPT=\$vcs_info_msg_0_
zstyle ':vcs_info:git:*' formats '%F{240}(%b)%r%f'
zstyle ':vcs_info:*' enable git

In this case %b and %r are placeholders for the VCS (version control system) system for the branch and the repository name.

There are git prompt solutions other than the built-in module, which deliver more information. There is a script in the git repository, and many of the larger zsh theme projects, such as ‘oh-my-zsh’ and ‘prezto’ have all kinds of git status widgets or modules or themes or what ever they call them.

Summary

You can spend (or waste) a lot of time on fine-tuning your prompt. Whether these modifications really improve your productivity is a matter of opinion.

In the next post, we will cover some miscellaneous odds and ends that haven’t yet really fit into any of preceding posts.

Build a shellcheck installer for macOS

Shellcheck is an invaluable tool for anyone who writes, well, shell scripts. The tool will point out common and less common errors, such as forgetting to quote a variable substitution, not putting that space before the closing ]], or not testing if a variable substitution used with rm -Rf might be empty.

You can use the tool’s website to check your code, but it is unwieldy, not to mention you or your organization might have concerns uploading all your code to some website, useful as it may be. You can install a shellcheck command line tool, but the instructions available for macOS will only work with Homebrew.

If you already have Homebrew installed, or don’t mind installing it and only need to install shellcheck on a single Mac, then you can use brew install shellcheck and skip ahead to the end, where I share how to use the shellcheck binary with BBEdit.

Relevant for MacAdmins

Homebrew may be a fine tool for individual users, but it is near-impossible to centrally manage a deployment of brew let alone a piece software installed with brew.

My attempts to build the shellcheck binary without homebrew lead to several dead-ends. Shellcheck is written in Haskell Cabal, which can be easily installed with… brew. There is also an official ghcup script, which is the ‘recommended way to install Haskell’, but this script fails on macOS because it requires a tool named xz. This is a (de-)compression tool, which is part of xz-utils, whose website does not explain how to install on macOS, but can be easily installed with… guess… brew

Even without brew, you have to install a number of tools that you probably don’t want hanging around on your production machine. I am sure cabal is a fine environment to build other tools as well, but I don’t need it on my production system.

I would recommend running these steps on a virtual machine. That will simplify ‘cleanup’ as well as the ‘start over’ process, should you need it.

How to build the shellcheck binary for macOS

Commands shown are correct at the time this article was written and may change in the future. Links to original sources are given, so you can check for changes.

Developer Tools

  • install Xcode or command line developer tools (CLI dev tools have the advantage of not being >6GB in size)
$ xcode-select --install

XZ Utils

Note: the macOS Archive Utility can deal with .xz archives just fine. There is really no need on macOS to have the xz-utils installed, but the GHC Cabal installer requires them.

$ cd ~/Downloads/xz-5.2.4
$ ./configure
$ sudo make install
  • check to see of xz was installed correctly
$ xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.4

Install Pandoc

We will require pandoc to build the man page for shellcheck.

Haskell Cabal

$ curl https://get-ghcup.haskell.org -sSf | sh
  • make sure the cabal environment is setup
$ source "$HOME/.ghcup/env"
$ cabal update

Shellcheck

$ git clone "https://github.com/koalaman/shellcheck.git"
  • build the shellcheck binary (this will take a while)
$ cd shellcheck
$ cabal install
  • this will build and install the shellcheck binary into ~/.cabal/bin, you can test if this worked with
$ ~/.cabal/bin/shellcheck --version
ShellCheck - shell script analysis tool
version: 0.6.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net

man page

  • there is also a man page in the repo, written in markdown format. The ReadMe says you can convert it to man format with pandoc:
$ pandoc -s -f markdown-smart -t man shellcheck.1.md -o shellcheck.1

Pkg it up

  • create a payload directory for the binary and man file and move the files there:
$ mkdir -p payload/local/bin
$ cp ~/.cabal/bin/shellcheck payload/local/bin/
$ mkdir -p payload/share/man/man1
$ cp shellcheck.1 payload/share/man/man1/
  • build the pkg installer from the payload with pkgbuild
$ pkgbuild --root payload --install-location /usr --identifier com.scriptingosx.pkg.shellcheck --version 0.6.0 shellcheck-0.6.0.pkg

Automation!

Of course, you can do this once and then forget about until the next update of shellcheck. While I wrote this post, I had to go through these steps more than once and got bored with it halfway through the second cycle. So, there is a script, which you can get at this Github repo.

Since this script will install xz-utils, cabal, and pandoc, I do not recommend to run this on your work machine. I use it with clean virtual machine. I suggest this workflow:

  • boot to clean vm
  • snapshot the vm
  • open Terminal
  • install developer command line tools: xcode-select --install
  • clone the script: git clone https://github.com/scriptingosx/BuildShellcheckPkg
  • run the script: cd BuildShellcheckPkg; ./buildShellCheckPkg.sh
  • confirm the prompts (once the cabal installation starts for good, the prompts are over)
  • wait…
  • wait…
  • start doing other work such as answering emails, MacAdmins Slack, Twitter,…
  • much later, remember there was something important, go to the vm, copy the brand new shellcheck pkg to your work machine
  • revert the vm to snapshot
  • buy ‘Packaging for Apple Administrators’ to show your gratitude and learn how to wield pkgbuild yourself

Using shellcheck

Using the shellcheck command line tool is very straightforward. Just pass one (or more) shell script files as arguments. There are some flags and options that you can read about in the --help option or the man page.

$ shellcheck serial.sh

In serial.sh line 11:
serial_number=$(serial) # stores output of function serial in variable
^-----------^ SC2034: serial_number appears unused. Verify use (or export if used externally).

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- serial_number appears unused. Ver...

Use shellcheck with BBEdit

My favored text editor – BBEdit – has a tool that will display the output of tools like shellcheck. This allows you to click on an error or warning and see and edit the code right there in a BBEdit window.

To get this, pipe the output of shellcheck into bbresults:

$ shellcheck -f gcc serial.sh | bbresults

I have created a function in my .bash_profile and .zshrc, so I don’t have to remember the details:

function bbshellcheck {
    shellcheck -f gcc "$@" | bbresults
}

If you do not know what the .bash_profile or functions are, then you can read about them here:

Note: you cannot use shellcheck to check zsh scripts. As of now it only works for sh or bash scripts. You can tell shellcheck to ignore the #!/bin/zsh shebang and interpret the script as using bash syntax, which should work in most cases, but will not always have good results.

$ shellcheck -s bash script.zsh

Moving to zsh, part 5: Completions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class in Amsterdam on September 6! Visit our website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

What are Completions?

Man shells use the tab key (⇥) for completion. When you press that key, the shell tries to guess what you are typing and will complete it, or if the beginning of what you typed is ambiguous, suggest from a list of possible completions.

For example when you want to cd to your Documents folder, you can save typing:

% cd ~/Doc⇥
% cd ~/Documents/

When you hit the tab key, the system will complete the path to the Documents folder.

When the completion is ambiguous, the shell will list possible completions:

% cd ~/D⇥
Desktop/    Documents/  Downloads/

At this point, you can add a character or two to get to a unique completion, and hit the tab key again. In zsh you can also hit the tab key repeatedly to cycle through the suggested completions. In this example, the first tab keystroke will show the list, the second will complete ~/Desktop/, the third completes ~/Documents, and so on.

You can use tab completion commands as well:

% system⇥
system_profiler     systemkeychain      systemsetup         systemsoundserverd  systemstats
% system_⇥
% system_profiler 

Not having to type path and file names saves time and avoids errors, especially with complex paths with spaces and other special characters:

% cd ~/Li⇥
% cd ~/Library/Appl⇥
% cd ~/Library/Application S⇥
Application Scripts/  Application Support/
% cd ~/Library/Application Su⇥
% cd ~/Library/Application Support/

Using tab completion is a huge productivity boost when using a shell.

Turning It On

In the default configuration, tab completion in zsh is very basic. It will complete commands and paths, but not much else. But you can enable a very powerful, and useful completion system.

zsh comes with a tool you can use to setup this completion system. When you run the compinstall command it will lead you through a complex and hard to understand list of menus which explains the options and will generate the code necessary to set this configuration up and add it to your .zshrc file or another configuration file of your choice.

Since the commands to configure the completion are quite arcane and hard to understand, this is a good way to get something to start out with. I will explain some of these options and commands in detail.

Whether you use compinstall or not, to turn on the more powerful completion system, you need to add at least this command to yourzsh` configuration file:

autoload -Uz compinit && compinit

This will initialize the zsh completion system. The details of this system are documented here.

If you want to configure the system, the configuration commands (usually zstyle commands) should be added to the zsh configuration file before you enable the system. (This only matters for a few configurations, but as a general rule it is safer.)

All of these completion rules need to be loaded and prepared. zsh’s completion system creates a cache in the file ~/.zcompdump. The first time you run compinit it might take a noticeable time, but subsequent runs should use this cache and be much faster.

Sometimes, especially when building and debugging your own completion files, you may need to delete this file to force a rebuild:

% rm -f ~/.zcompdump
% compinit

Case Insensitive Completion

Since the macOS file systems are usually case-insensitive, I prefer my tab-completion to be case-insensitive as well. For bash you configure that in the ~/.inputrc. In zsh you modify the completion systems behavior with this (monstrous) command:

# case insensitive path-completion
zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*'

I have seen many varieties for this configuration in different websites, but this is what compinstall adds when I select case-insensitive completion, so I am going with that.

Partial Completion

This is a particularly nice feature. You can type fragments of each path segment and the completion will try to complete them all at once:

% cd /u/lo/b⇥
% cd /usr/local/bin
% cd ~/L/P/B⇥
% ~/Library/Preferences/ByHost/

If the fragments are ambiguous, there are different strategies to what the completion system suggests. I have configured these like this:

# partial completion suggestions
zstyle ':completion:*' list-suffixes
zstyle ':completion:*' expand prefix suffix

Commands with built-in completion

zsh comes with several completion definitions for many commands. For example, when you type cp and then hit tab, the system will correctly assume you want to complete a file path and show the suggestions from the current working directory.

However, when you type cp -⇥ the completion can tell from the - that you want to add an option to the command and suggest a list of options for cp, with short descriptions.

% cp -⇥ 
-H  -- follow symlinks on the command line in recursive mode
-L  -- follow all symlinks in recursive mode
-P  -- do not follow symlinks in recursive mode (default)
-R  -- copy directories recursively
-X  -- don't copy extended attributes or resource forks
-a  -- archive mode, same as -RpP
-f  -- force overwriting existing file
-i  -- confirm before overwriting existing file
-n  -- don't overwrite existing file
-p  -- preserve timestamps, mode, owner, flags, ACLs, and extended attributes
-v  -- show file names as they are copied

As the context of command prompt you are assembling changes, you may get different completion suggestions. For example, the completion for ssh will suggest host names:

% ssh armin@⇥

zsh comes with completion definitions for many common commands. Nevertheless, it can be helpful to just hit tab, especially when wondering about options.

On macOS completions are stored in /usr/share/zsh/5.3/functions (replace the5.3 with 5.7.1 in Catalina). This directory stores many functions used with zsh and is in the default fpath. All the files in that directory that start with an underscore _ contain the completion definitions command. So, the file _cp contains the definition for the cp command. (Some of the definition files contain the definitions for multiple commands.)

Completions for macOS Commands

There are even a few macOS specific command that come with the default zsh installation.

% system_profiler ⇥⇥

macOS High Sierra and macOS Mojave come with zsh 5.3, which is now nearly two years old. zsh 5.3 contains less macOS specific completion definitions than the current zsh 5.7.1 which will is the pre-installed zsh in macOS Catalina. Some of the completions in 5.3 have also been updated in 5.7.1.

Tool zsh 5.3 zsh 5.7.1
caffeinate
defaults
fink
fs_usage
hdiutil
mdfind
mdls
mdutil
networksetup
nvram
open
osascript
otool
pbcopy/pbpaste
plutil
say
sc_usage
scselect
scutil
softwareupdate
sw_vers
swift
system_profiler
xcode_select

Load bash completions

Since the default shell on macOS has been bash for so long, there are quite a few bash completion definitions for macOS commands and third party tools available. For example Tony Williams’ bash completion for autopkg (post, Github).

You do not have to rewrite these completions, since the zsh completion system can use bash completion scripts as well: (add this to your zsh configuration file)

# load bashcompinit for some old bash completions
autoload bashcompinit && bashcompinit

[[ -r ~/Projects/autopkg_complete/autopkg ]] && source ~/Projects/autopkg_complete/autopkg

When you have multiple bash completion scripts you want to load, you only need to load bashcompinit once.

Build your own completions

Once you start using completions, you will want to have them everywhere. While many built-in completions exists, there are still many commands that lack a good definition.

Some commands, like the swift command line tool, have a built-in option to generate the completion syntax. You can then store that in a file and put it in your fpath:

% swift package completion-tool generate-zsh-script >_swift

Note: in the case of swift, its definition will conflict with the _openstack definition in zsh 5.3. You can fix this with the command compdef _swift swift after loading the completion system.

Some commands provide a list of options and arguments with the -h/--help option. If this list follows a certain syntax, you can get a decent completion working with

% compdef _gnu_generic <command>

One example on macOS, where this has decent results is the xed command which opens a file or folder in Xcode.

But for best results, you will often have to build the description yourself. Unfortunately this is not a simple task. The syntax is meticulously, but also quite abstractly documented in the zsh documentation for the Completion System. I also found the ‘howto’ documentation in the zsh-completions repository very useful, as well as the ‘zsh Completion Style Guide.’

To avoid everyone re-inventing the wheel, I have started a repository on Github for macOS specific completion files. The page has the instructions on how to install them and I will welcome pull requests with contributions. Since I am just starting to learn this as well, I am sure there are improvements that can be made on the completions I have built so far and there are several commands where you can test your skills and build a new one.

I suggest the #zsh channel on the MacAdmins Slack for discussion.

Next

In the next post in this series, we will discuss how to configure zsh’s command line prompt.

Install Bash 5 on macOS with Patches

I recently posted an article on how to download, install, and build a macOS installer pkg for bash 5. In that first version of this post I ignored patches, minor updates to the bash source code and binary. But as the patches to bash 5 are accumulating, I cannot ignore them much longer.

This post will extend the instructions in the original post.

After downloading and expanding the bash-5.0.tar.gz, create a patches folder:

$ cd path/to/bash-5.0
$ mkdir patches
$ cd patches

You can download the patches for bash-5.0 here. As of this writing, there are seven patches for bash-5.0 labelled bash50-001 through bash50-007. You can download all at once with:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' -O

(Adapt the numbers when there are more patches in the future.)

Then move up one directory level to the bash-5.0 root directory and apply the patches using the patch command.

$ cd ..
$ patch -p0 -i patches/bash50-001
$ patch -p0 -i patches/bash50-002

(etc.)

You can download and patch with a single step. Make sure your working directory is the bash-5.0 with all the code and run:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' | patch -p0

From here, you can continue with the remaining build steps from the original post. The next step will be running ./configure.

The script to build the pkg installer has also been updated in the repository to download and apply the patches before building.