This is an excerpt from my book “Moving to zsh” which is available for order on the Apple Books Store.
One of the advantages of zsh over bash 3 is the support of “associative arrays,” a data structure known as hash tables or dictionaries in other languages.
In associative arrays, you can store a piece of data, or value with an identifying ‘key’. For example, the associative array userinfo has multiple values, each identified with a key:
In zsh, before you can use a variable as an associative array, you have to declare it as one with
declare -A userinfo
This will tell the shell that the userinfo variable is an associative array. You can also use typeset -A as an alternative syntax. You can verify the type of the variable:
% echo ${(t)userinfo}
association
You can then set the key-value pairs of the userinfo associative array individually:
Setting the values for each key is useful in some situations, but can be tedious. You can also set the entire associative array at once. There are two syntaxes for this in zsh:
userinfo=( name armin shell zsh website scriptingosx.com )
This format follows the format ( key1 value1 key2 value2 ...). The other syntax is more verbose and expressive:
When you set the associative array variable this way, you are overwriting the entire array. For example, if you set the userinfo for ‘armin’ like above and then set it later like this, the website key and value pair will have been overwritten as well:
% userinfo=( [name]=beth [shell]=zsh )
% if [[ -z $userinfo[website] ]]; then echo no value; fi
no value
If you want to partially overwrite an existing associative array, while leaving the other key/value pairs intact, you can use the += operator:
You can also use this to loop through all the keys and values of an associated array:
for key value in ${(kv)userinfo}; do
echo "$key -> $value"
done
#output
website -> fishshell.com
shell -> fish
name -> beth
Limitations
Associative arrays have their uses, but are not as powerful as dictionaries in more powerful languages. In zsh, you cannot nest associative arrays in normal arrays, which limits their use for complex data structures.
There is also no functionality to transfer certain file formats, like XML or property lists directly in to associative arrays or back.
Shell scripting was never designed for complex data structures. When you encounter these limitations, you should move “up” to a higher level language, such as Python or Swift.
Just a “minor” update. The EraseInstall app now shows the progress that the startosinstall command gives in the command line. This also should help with some better error reporting when the startosinstall command errors out.
I say “minor” but small UI change required some major rewiring underneath. It also required us to dive deeper into how shell commands are executed from Swift than we wanted to.
We have also tested this version to work with macOS Catalina which was released yesterday.
We have more “major” features planned for the future!
The upcoming macOS 10.15 Catalina will require more apps and tools to be notarized. Apple has somewhat loosened the requirements at last minute, but these changed limitations are only temporary, to give developers more time to adapt.
Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.
Howard Oakley has written up his experiences and that post was extremely helpful. But there were a few omissions and some steps that aren’t really necessary, so I decided to make my own write-up.
And yes, there is a script at the end…
Note: these instructions are for macOS 10.14.6 Mojave, Xcode 10.3 and Swift 5.0. It is very likely that the details will change over time.
Update 2019-09-24: Tested with Xcode 11 and it still works (the screen layout has changed for some of the options)
What do you need?
Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates)
Xcode 10.3 or 11
Developer ID Certificates (Application and Install)
Application Specific Password for your Developer account
a Command Line Tool Project that you want to sign and notarize
That’s a longish list. If you are already building command line tools in Xcode, you should have most of these covered already. We will walk through the list step-by-step:
You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.
Xcode
You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.
Developer ID Certificates
There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.
To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.
We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.
If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.
Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:
% security find-identity -p basic -v
This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.
You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"
The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.
Application Specific Password for your Developer Account
Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.
Create a new application specific password in Apple ID portal for your developer account.
You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:
Keychain Item Name: Developer-altool
Account Name: your developer account email
Password: the application-specific password you just created
This will create a developer specific password item that we can access safely from the tools.
If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.
A Command Line Tool Project
You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.
My sample project for this article will be named “hello.”
Preparing the Xcode Project
The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.
Choosing the proper signing certificates
Before you can notarize the command line tool, it needs to be signed with the correct certificates.
in Xcode, select the blue project icon in the left sidebar
select the black “terminal” icon with your project’s name under the “Targets” list entry
make sure the ‘General’ tab is selected
under ‘Signing’ disable ‘Automatically manage signing’
under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Enable Hardened Runtime
Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.
from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
click on ‘All’ to show all available settings
enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
set the value in the project column (blue icon) to YES
Change the Install Build Location
If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:
in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
double click on the value in the Project column (blue icon), this will open a popup window
change the value to $SRCROOT/build/pkgroot
If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)
Build the Binary
You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:
open Terminal and change directory to the project folder
% xcodebuild clean install
This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.
After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.
/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.
Build the pkg
Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.
We can use the pkgroot directory as our payload to build the installer package:
I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.
This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.
If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”
Notarizing the Installer Package
Xcode has a command line tool altool which you can use to upload your tool for notarization:
The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.
The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)
This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:
Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.
Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.
Completing the Process
You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.
If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.
You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:
% xcrun stapler staple build/hello-1.0.pkg
You can also use stapler to verify the process went well:
% xcrun stapler validate build/hello-1.0.pkg
But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:
Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.
The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.
You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.
Links and Videos
These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.
Notarization is a key part of Apple’s security strategy going in macOS.
As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.
If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.
On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.
Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.
The release notes for Catalina also tell us that other built-in scripting runtimes, namely Python, Perl, and Ruby. Will not be included in future macOS releases (post-Catalina) any more.
This means, that if you want to use bash, Python, Perl, or Ruby on macOS, you will have to install, and maintain your own version in the future.
However, scripts in installation packages, cannot rely on any of these interpreters being available in future, post-Catalina versions of macOS. Installer pkgs can be run in all kinds of environments and at all times, and you would not want them to fail, because a dependency is missing.
The good news is that we still have time. All the runtimes mentioned above are still present in Catalina, so the packages will continue to work for now. But if you are building installation scripts, you need to check if any of the installation scripts use one of these interpreters and fix them.
I recommend to use /bin/sh for installation scripts, since that will run in any macOS context, even the Recovery system.
If you are using third-party installer packages, you may also want to check them for these interpreters, and notify the developer that these packages will break in future versions of macOS.
To check a flat installer package, you would expand it with pkgutil --expand and then look at script files in the Scripts folder. This will work fine for a package or two, but gets tedious really quickly, especially with large distribution pkgs with many components (e.g. Office).
So… I wrote a script to do it. The script should handle normal component pkgs, distribution pkgs and the legacy bundle pkgs and mpkgs.
Once I had written the code to inspect all these types of pkgs, I realized I could grab all other kinds of information, as well. The pkgcheck.sh script will check for:
Signature and Notarization
Type of Package: Component, Distribution, legacy bundle or mpkg
Identifier and version (when present)
Install-location
for Distribution and mpkg types, shows the information for all components as well
for every script in a pkg or component, checks the first line of the script for shebangs of the deprecated interpreters (/bin/bash, /usr/bin/python, /usr/bin/perl, and /usr/bin/ruby) and print a warning when found
How to run pkgcheck.sh
Run the script with the target pkg file as an argument:
% ./pkgcheck.sh sample.pkg
You can give more than one file:
% ./pkgcheck.sh file1.pkg file2.pkg ...
When you pass a directory, pkgcheck.sh will recursively search for all files or bundle directories with the pkg or mpkg extension in that directory:
% ./pkgcheck.sh SamplePkgs
Features and Errors
There are a few more things that I think might be useful to check in this script. Most of all, I want to add an indicator whether a component is enabled by default or not. If you can think of any other valuable data to display, let me know. (Issue or Pull Request or just ping me on MacAdmins Slack)
I have tested the script against many pkgs that I came across. However, there are likely edge cases that I haven’t anticipated, which might break the script. If you run into any of those, let me know. (File an Issue or Pull Request.) Having the troublesome pkg would of course be a great help.
Note: the script will create a scratch directory for temporary file extractions. The script doesn’t actually expand the entire pkg file, only the Scripts sub-archive. The scratch folder will be cleaned out at the beginning of the next run, but not when the script ends, as you might want to do some further inspections.
Sample outputs
This is a sample pkg I build in my book, it has pre- and postinstall scripts using a /bin/bash shebang:
As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)
We have covered the general aspects of configuring your zsh environment and enabling some of its features to make your work more productive. However, there are zsh features that didn’t quite fit in earlier posts, but also don’t warrant a post of their own. So I am gathering them here.
multiIO
Terminal commands can take input from a file or a previous command (stdin) and have two different outputs: stdout and stderr. In bash you can redirect each of these to a single other destination.
For example, you can redirect the output of a command to a file:
Note that the order of doing this is important. The construct >file.txt >&1 would redirect the output to file.txt and then redirect the output again to where stdout or 1 is going, so it would be redundant.
When combined with pipes and other commands multiIO can become very useful:
Since the fpath variable is an array, I only changed the fpath variable in my zshrc.I never set or changed the FPATH, yet it reflects the changes made to the fpath variable.
When you see the type of both variables, you get an idea that something is going on:
The fpath and FPATH are connected in zsh. Changes to one affect the other. This allows use of more flexible and powerful array operations through the fpath ‘aspect’ of the value, but also provides compatibility to tools that expect the traditional colon-separated format in FPATH.
You will not be surprised to hear that zsh uses the same ‘magic’ with the PATH variable and its array counterpart path.
This means that you can continue to use path_helper to get your PATH from the files in /etc/paths and /etc/paths.d. (Well, you don’t have to, because on macOS this is done for all users in /etc/zprofile.) But then you can manipulate the path variable with array functions, like:
path+=~/bin
You get the useful aspects of both syntaxes.
Suffix Aliases
I learnt this one after writing the aliases part.
Suffix aliases take effect on the last part of a path, so usually the file extension. A suffix alias will assign a command to use when you just type a file path in the command line.
For example, you can a suffix alias for the txt file extension:
alias -s txt="open -t"
When you then type a path ending with .txt and no command, zsh will execute open -t /path/to/file.txt.
The open -t command opens a file in the default application set for the txt file extension in Finder. You probably want to set the suffix alias to bbedit or atom or something like that rather than open -t.
You can use other command line tools for the suffix alias:
alias -s log="tail -f"
Then, typing /var/log/install.log will show the last lines of that file and update the output when the file changes. If you prefer the graphical user interface, you can use the open -a command to assign suffix aliases to applications:
alias -s log="open -a Console"
You can even create a suffix alias using a different alias:
alias pacifist="open -a Pacifist"
alias -s pkg=pacifist
Together with the AutoCD option, this can improve your application-shell interactions a lot.
Bindkey for History Search
Most of the keyboard shortcuts in zsh work the same way as they do in bash. I have found one change that has proven quite useful:
^[[A' up-line-or-search # up arrow bindkey^[[B' down-line-or-search # down arrow
These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.
This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.
After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.
The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.
As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)
The default bash prompt on macOS is quite elaborate. It shows the username, the hostname, and the current directory.
Calypso:~ armin$
On the other hand, the default bash prompt doesn’t show the previous command’s exit code, a piece of information I find very useful. I have written before how I re-configured my bash prompt to have the information I want:
Of course, I wanted to recreate the same experience in zsh.
The only (visual) difference to my bash prompt is the % instead of the $.
Note: creating a file ~/.hushlogin will suppress the status message at the start of each Terminal session in zsh as well as in bash (or any other shell).
Basic Prompt Configuration
The basic zsh prompt configuration works similar to bash, even though it uses a different syntax. The different placeholders are described in detail in the zsh manual.
zsh uses the same shell variable PS1 to store the default prompt. However, the variable names PROMPT and prompt are synonyms for PS1 and you will see either of those three being used in various examples. I am going to use PROMPT.
The default prompt in zsh is %m%#. The %m shows the first element of the hostname, the %# shows a # when the current prompt has super-user privileges (e.g. after a sudo -s) and otherwise the % symbol (the default zsh prompt symbol).
The zsh default prompt is far shorter than the bash default, but even less useful. Since I work on the local system most of the time, the hostname bears no useful information, and repeating it every line is superfluous.
Note: you can argue that the hostname in the prompt is useful when you frequently have multiple terminal windows open to different hosts. This is true, but then the prompt is defined by the remote shell and its configuration files on the remote host. In your configuration file, you can test if the SSH_CLIENT variable is set and show a different prompt for remote sessions. There are more ways of showing the host in remote shell sessions, for example in the Terminal window title bar or with different window background colors.
In our first iteration, I want to show the current working directory instead of the hostname. When you look through the list of prompt placeholders in the zsh documentation, you find %d, %/, and %~. The first two do exactly the same. The last substitution will display a path that starts with the user’s home directory with the ~, so it will shorten /Users/armin/Projects/ to ~/Projects.
Note: in the end you want to set your PROMPT variable in the .zshrc file, so it will take effect in all your zsh sessions. For testing, however, you can just change the PROMPT variable in the interactive shell. This will give you immediate feedback, how your current setup works.
Note the trailing space in the prompt string, to separate the final % or # from the command entry.
I prefer the shorter output of the %~ option, but it can still be quite long, depending on your working directory. zsh has a trick for this: when you insert a number n between the % and the ~, then only the last n elements of the path will be shown:
% PROMPT='%2~ %# '
dotfiles/zshfunctions %
When you do %1~ it will show only the name of the working directory or ~ if it is the home directory. (This also works with %/, e.g. %2/.)
Adding Color
Adding a bit of color or shades of gray to the prompt can make it more readable. In bash you need cryptic escape codes to switch the colors. zsh provides an easier way. To turn the directory in the path blue, you can use:
PROMPT='%F{blue}%1~%f %# '
The F stands for ‘Foreground color.’ zsh understands the colors black, red, green, yellow, blue, magenta, cyan and white. %F or %f resets to the default text color. Furthermore, Terminal.app represents itself as a 256-color terminal to the shell. You can verify this with
% echo $TERM
xterm-256color
You can access the 256 color pallet with %F{0} through %F{255}. There are tables showing which number maps to which color:
One of the prompt codes provides a ‘ternary conditional,’ which means it will show one of two expressions, depending on a condition. There are several conditions you can use. Once again the details can be found in the documentation.
There is one condition for the previous commands exit code:
%(?.<success expression>.<failure expression>)
This expression will use the <success expression> when the previous command exited successfully (exit code zero) and <failure expression> when the previous command failed (non-zero exit code). So it is quite easy to build an conditional prompt:
% PROMPT='%(?.√.?%?) %1~ %# '
√ ~ % false
?1 ~ %
You can get the √ character with option-V on the US or international macOS keyboard layout. The last part of the ternary ?%? looks confusing. The first ? will print a literal question mark, and the second part %? will be replaced with previous command’s exit code.
You can add colors in the ternary expression as well:
Another interesting conditional code is ! which returns whether the shell is privileged (i.e. running as root) or not. This allows us to change the default prompt symbol from % to something else, while maintaining the warning functionality when running as root:
% PROMPT='%1~ %(!.#.>) '
~ > sudo -s
~ # exit
~ >
Complete Prompt
Here is the complete prompt we assembled, with all the parts explained:
zsh also offers a right sided prompt. It uses the same placeholders as the ‘normal’ prompt. Use the RPROMPT variable to set the right side prompt:
% RPROMPT='%*'
√ zshfunctions % 11:02:55
zsh will automatically hide the right prompt when the cursor reaches it when typing a long command. You can use all the other substitutions from the left side prompt, including colors and other visual markers in the right side prompt.
Git Integration
zsh includes some basic integration for version control systems. Once again there is a voluminous, but hard to understand description of it in the documentation.
I found a better, more specific example in the ‘Pro git’ documentation. This example will show the current branch on the right side prompt.
I have changed the example to include the repo name and the branch, and to change the color.
In this case %b and %r are placeholders for the VCS (version control system) system for the branch and the repository name.
There are git prompt solutions other than the built-in module, which deliver more information. There is a script in the git repository, and many of the larger zsh theme projects, such as ‘oh-my-zsh’ and ‘prezto’ have all kinds of git status widgets or modules or themes or what ever they call them.
Summary
You can spend (or waste) a lot of time on fine-tuning your prompt. Whether these modifications really improve your productivity is a matter of opinion.
In the next post, we will cover some miscellaneous odds and ends that haven’t yet really fit into any of preceding posts.
As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)
What are Completions?
Man shells use the tab key (⇥) for completion. When you press that key, the shell tries to guess what you are typing and will complete it, or if the beginning of what you typed is ambiguous, suggest from a list of possible completions.
For example when you want to cd to your Documents folder, you can save typing:
% cd ~/Doc⇥
% cd ~/Documents/
When you hit the tab key, the system will complete the path to the Documents folder.
When the completion is ambiguous, the shell will list possible completions:
% cd ~/D⇥
Desktop/ Documents/ Downloads/
At this point, you can add a character or two to get to a unique completion, and hit the tab key again. In zsh you can also hit the tab key repeatedly to cycle through the suggested completions. In this example, the first tab keystroke will show the list, the second will complete ~/Desktop/, the third completes ~/Documents, and so on.
Not having to type path and file names saves time and avoids errors, especially with complex paths with spaces and other special characters:
% cd ~/Li⇥ % cd ~/Library/Appl⇥
% cd ~/Library/Application S⇥
Application Scripts/ Application Support/
% cd ~/Library/Application Su⇥
% cd ~/Library/Application Support/
Using tab completion is a huge productivity boost when using a shell.
Turning It On
In the default configuration, tab completion in zsh is very basic. It will complete commands and paths, but not much else. But you can enable a very powerful, and useful completion system.
zsh comes with a tool you can use to setup this completion system. When you run the compinstall command it will lead you through a complex and hard to understand list of menus which explains the options and will generate the code necessary to set this configuration up and add it to your .zshrc file or another configuration file of your choice.
Since the commands to configure the completion are quite arcane and hard to understand, this is a good way to get something to start out with. I will explain some of these options and commands in detail.
Whether you use compinstall or not, to turn on the more powerful completion system, you need to add at least this command to your zsh configuration file:
autoload -Uz compinit && compinit
This will initialize the zsh completion system. The details of this system are documented here.
If you want to configure the system, the configuration commands (usually zstyle commands) should be added to the zsh configuration file before you enable the system. (This only matters for a few configurations, but as a general rule it is safer.)
All of these completion rules need to be loaded and prepared. zsh’s completion system creates a cache in the file ~/.zcompdump. The first time you run compinit it might take a noticeable time, but subsequent runs should use this cache and be much faster.
Sometimes, especially when building and debugging your own completion files, you may need to delete this file to force a rebuild:
% rm -f ~/.zcompdump % compinit
Case Insensitive Completion
Since the macOS file systems are usually case-insensitive, I prefer my tab-completion to be case-insensitive as well. For bash you configure that in the ~/.inputrc. In zsh you modify the completion systems behavior with this (monstrous) command:
I have seen many varieties for this configuration in different websites, but this is what compinstall adds when I select case-insensitive completion, so I am going with that.
Partial Completion
This is a particularly nice feature. You can type fragments of each path segment and the completion will try to complete them all at once:
% cd /u/lo/b⇥
% cd /usr/local/bin
% cd ~/L/P/B⇥
% ~/Library/Preferences/ByHost/
If the fragments are ambiguous, there are different strategies to what the completion system suggests. I have configured these like this:
zsh comes with several completion definitions for many commands. For example, when you type cp and then hit tab, the system will correctly assume you want to complete a file path and show the suggestions from the current working directory.
However, when you type cp -⇥ the completion can tell from the - that you want to add an option to the command and suggest a list of options for cp, with short descriptions.
% cp -⇥
-H -- follow symlinks on the command line in recursive mode
-L -- follow all symlinks in recursive mode
-P -- do not follow symlinks in recursive mode (default)
-R -- copy directories recursively
-X -- don't copy extended attributes or resource forks
-a -- archive mode, same as -RpP
-f -- force overwriting existing file
-i -- confirm before overwriting existing file
-n -- don't overwrite existing file
-p -- preserve timestamps, mode, owner, flags, ACLs, and extended attributes
-v -- show file names as they are copied
As the context of command prompt you are assembling changes, you may get different completion suggestions. For example, the completion for ssh will suggest host names:
% ssh armin@⇥
zsh comes with completion definitions for many common commands. Nevertheless, it can be helpful to just hit tab, especially when wondering about options.
On macOS completions are stored in /usr/share/zsh/5.3/functions (replace the5.3 with 5.7.1 in Catalina). This directory stores many functions used with zsh and is in the default fpath. All the files in that directory that start with an underscore _ contain the completion definitions command. So, the file _cp contains the definition for the cp command. (Some of the definition files contain the definitions for multiple commands.)
Completions for macOS Commands
There are even a few macOS specific command that come with the default zsh installation.
% system_profiler ⇥⇥
macOS High Sierra and macOS Mojave come with zsh 5.3, which is now nearly two years old. zsh 5.3 contains less macOS specific completion definitions than the current zsh 5.7.1 which will is the pre-installed zsh in macOS Catalina. Some of the completions in 5.3 have also been updated in 5.7.1.
Tool
zsh 5.3
zsh 5.7.1
caffeinate
√
defaults
√
√
fink
√
√
fs_usage
√
hdiutil
√
√
mdfind
√
mdls
√
mdutil
√
networksetup
√
nvram
√
open
√
√
osascript
√
otool
√
pbcopy/pbpaste
√
plutil
√
say
√
sc_usage
√
scselect
√
scutil
√
softwareupdate
√
√
sw_vers
√
swift
√
system_profiler
√
√
xcode_select
√
Load bash completions
Since the default shell on macOS has been bash for so long, there are quite a few bash completion definitions for macOS commands and third party tools available. For example Tony Williams’ bash completion for autopkg (post, Github).
You do not have to rewrite these completions, since the zsh completion system can use bash completion scripts as well: (add this to your zsh configuration file)
# load bashcompinit for some old bash completions
autoload bashcompinit && bashcompinit
[[ -r ~/Projects/autopkg_complete/autopkg ]] && source ~/Projects/autopkg_complete/autopkg
When you have multiple bash completion scripts you want to load, you only need to load bashcompinit once.
Build your own completions
Once you start using completions, you will want to have them everywhere. While many built-in completions exists, there are still many commands that lack a good definition.
Some commands, like the swift command line tool, have a built-in option to generate the completion syntax. You can then store that in a file and put it in your fpath:
% swift package completion-tool generate-zsh-script >_swift
Note: in the case of swift, its definition will conflict with the _openstack definition in zsh 5.3. You can fix this with the command compdef _swift swift after loading the completion system.
Some commands provide a list of options and arguments with the -h/--help option. If this list follows a certain syntax, you can get a decent completion working with
% compdef _gnu_generic <command>
One example on macOS, where this has decent results is the xed command which opens a file or folder in Xcode.
To avoid everyone re-inventing the wheel, I have started a repository on Github for macOS specific completion files. The page has the instructions on how to install them and I will welcome pull requests with contributions. Since I am just starting to learn this as well, I am sure there are improvements that can be made on the completions I have built so far and there are several commands where you can test your skills and build a new one.
I suggest the #zsh channel on the MacAdmins Slack for discussion.
This post will extend the instructions in the original post.
After downloading and expanding the bash-5.0.tar.gz, create a patches folder:
$ cd path/to/bash-5.0
$ mkdir patches
$ cd patches
You can download the patches for bash-5.0 here. As of this writing, there are seven patches for bash-5.0 labelled bash50-001 through bash50-007. You can download all at once with:
zsh (I believe it is pronounced zee-shell, though zish is fun to say) will succeed bash as the default shell. bash has been the default shell since Mac OS X 10.3 Panther.
The bash binary bundled with macOS has been stuck on version 3.2 for a long time now. bash v4 was released in 2009 and bash v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3. bash v3 is still GPL v2.
zsh, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default. zsh has been available as on macOS for a long time. The zsh version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current zsh 5.7.1.
Is bash gone!?
No.
macOS Catalina still has the same /bin/bash (version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.
Many scripts in macOS, management systems, and Apple and third party installers rely on /bin/bash. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.
Users that have /bin/bash as their default shell on Catalina will see a prompt at the start of each Terminal session stating that zsh is now the recommended default shell. If you want to continue using /bin/bash, you can supress this message by setting an environment variable in your .bash_profile or .bashrc.
Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to bash, but didn’t really care if you remained on tcsh. In fact, tcsh is still present on macOS.
Apple’s messaging should tell us, that the days of /bin/bashare numbered. Probably not very soon, but eventually keeping a more than ten year old version of bash on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.
Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’ dash has been added to the lineup of shells. dash is designed to be a minimal implementation of the Posix standard shell sh. So far, in macOS (including Catalina),sh invokes bash in sh-compatibility mode.
As Apple’s support article mentions, Catalina also adds a new mechanism for users and admins to change which shell handles sh invocations. MacAdmins or users can change the symbolic link stored in /var/select/sh to point to a shell other than /bin/bash. This changes which shell interprets scripts the #!/bin/sh shebang or scripts invoked with sh -c. Changing the interpreter for sh should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.
All of these changes are indicators that Apple is preparing to remove /bin/bash at some, yet indeterminate, time in the future.
Do I need to wait for Catalina to switch to zsh?
No, zsh is available Mojave and on older macOS versions. You can start testing zsh or even switch your default shell already.
If you want to just see how zsh works, you can just open Terminal and type zsh:
$ zsh
MacBook%
The main change you will see is that the prompt looks different. zsh uses the % character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in bash.
If you want to already switch your default shell to zsh you can use the chsh command:
$ chsh -s /bin/zsh
This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.
How is zsh different?
Like bash (‘Bourne again shell’ ), zshderives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.
The main difference between bash and zsh is configuration. Since zsh ignores the bash configuration files (.bash_profile or .bashrc) you cannot simply copy customized bash settings over to zsh. zsh has much more options and points to change zsh configuration and behavior. There is an entire eco-system of configuration tools and themes called oh-my-zsh which is very popular.
zsh also offers better configuration for auto-completion which is far easier than in bash.
I am planning a separate post, describing how to transfer (and translate) your configurations from bash to zsh.
What about scripting?
Since zsh has been present on macOS for a long time, you could start moving your scripts from bash to zsh right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to #!/bin/zsh.
You will gain some features where zsh is superior to bash v3, such as arrays and associative arrays (dictionaries).
There is one exception where I would now recommend to use /bin/sh for your scripts: the Recovery system does not contain the /bin/zsh shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider older macOS installations where zsh is definitely not present in Recovery.
When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as Twocanoes MDS, installr or bootstrappr, then you cannot rely on /bin/zsh.
Since we now know that bash is eventually going away, the only common choice left is /bin/sh.
When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using /bin/sh as the shebang from now on.
I used to recommend using /bin/bash for everything MacAdmin related. /bin/sh is definitely a step down in functionality, but it seems like the safest choice for continued support.
Summary
Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’ bash with zsh, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!
Someone on the MacAdmins Slack recently asked how you could assign a global keyboard short cut to open Terminal on macOS.
Note: alternative terminal applications such as iTerm2 may have this built-in.
macOS has an option to assign custom global keystrokes to pretty much anything, but it is not obvious how to get there.
First, open the Automator application. In the chooser for a new Workflow, choose ‘Quick Action’ (on Mojave) or ‘Service’ on earlier versions of macOS.
In the new workflow configure the input to be ‘no input’ and the application to be ‘any application.’
Then search for ‘Launch Application’ action in the library pane on the left and add it to your workflow by double-clicking or dragging.
The popup menu where you can slect an application in the action will only show applications from the /Applications folder. Choose ‘Other…’ and select Terminal in the ’/Applications/Utilities` folder.
Save the workflow. Give it a meaningful name such as ‘Open Terminal.’ Since you chose Quick Action or Service, this workflow will be saved in ~/Library/Services.
Open System Preferences > Keyboard. Click the ‘Shortcuts’ tab and select ‘Services’ from the list on the left side. (Even on Mojave, it is still called ‘Services’.)
Scroll all the way down the list of services under the ‘General’ heading, you should find the service you just created. Select it and click ‘Add Shortcut’ to assign a global shortcut.
You are done!
When the active application uses the same keystroke, the application’s definition will precede your global shortcut.
Of course, you don’t have stop at launching applications. You can assign a global keyboard shortcut to any Automator workflow this way. Since Automator workflows can include AppleScript, Python or shell scripts, you can do pretty much anything this way!
However, most Apple users don’t bother with shortcuts to launch apps. Just invoke Spotlight with command-space and start typing term and hit return.