Scripting OS X

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

• Part 1: Moving to zsh
• Part 2: Configuration Files
• Part 3: Shell Options
• Part 4: Aliases and Functions
• Part 5: Completions
• Part 6: Customizing the zsh Prompt
• Part 7: Miscellanea (this article)
• Part 8: Scripting zsh

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

We have covered the general aspects of configuring your zsh environment and enabling some of its features to make your work more productive. However, there are zsh features that didn’t quite fit in earlier posts, but also don’t warrant a post of their own. So I am gathering them here.

multiIO

Terminal commands can take input from a file or a previous command (stdin) and have two different outputs: stdout and stderr. In bash you can redirect each of these to a single other destination.

For example, you can redirect the output of a command to a file:

% system_profiler SPHardwareDataType >hardwareinfo.txt

In zsh you can redirect to (and from) multiple sources. In the simplest form you can write the output to two files:

% system_profiler SPHardwareDataType >hardwareinfo.txt >computerinfo.txt

This is of course not a very realistic case. Since the pipe | is a form of redirection, you can combine output to a file with a pipe:

% system_profiler SPHardwareDataType >hardwareprofile.txt | cat

Instead of piping to cat, you can also redirect to stdout (or &1) as well as to a file:

% system_profiler SPHardwareDataType >&1 >hardwareprofile.txt

Note that the order of doing this is important. The construct >file.txt >&1 would redirect the output to file.txt and then redirect the output again to where stdout or 1 is going, so it would be redundant.

When combined with pipes and other commands multiIO can become very useful:

% system_profiler SPHardwareDataType >hardwareprofile.txt | awk '/Serial Number/ { print $4 }' >&1 >serialnumber.txt

You can use multiIO for input as well:

% sort </usr/share/calendar/calendar.freebsd </usr/share/calendar/calendar.computer

And while this not directly related, but somewhat close, in zsh, this

% <hardwareinfo.txt

is equivalent to more hardwareinfo.txt.

Recursive Globbing with **

You can use the ** to denote an arbitrary string that can span multiple directories in a path.

For example:

% echo Library/Preferences/**/com.apple.screensaver.*plist
Library/Preferences/ByHost/com.apple.screensaver.BBCCDDEE-AABB-CCDD-ABCD-00AABBCCDDEE.plist Library/Preferences/com.apple.screensaver.plist

In this case the ** matches nothing as well as /ByHost/.

Note: when used on large folder structures this glob can take a while. So use with care.

Connected array Variables

We already encountered the fpath variable in earlier posts. You can see its contents with the echo command:

% echo $fpath
/Users/armin/Projects/mac-zsh-completions/completions/ /Users/armin/Projects/dotfiles/zshfunctions /usr/local/share/zsh/site-functions /usr/share/zsh/site-functions /usr/share/zsh/5.3/functions

Interestingly enough, zsh also has an FPATH variable, which is a colon-separated list of directories:

% echo $FPATH
/Users/armin/Projects/mac-zsh-completions/completions/:/Users/armin/Projects/dotfiles/zshfunctions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.3/functions

Since the fpath variable is an array, I only changed the fpath variable in my zshrc.I never set or changed the FPATH, yet it reflects the changes made to the fpath variable.

When you see the type of both variables, you get an idea that something is going on:

% echo ${(t)fpath}
array-special
% echo ${(t)FPATH}
scalar-special

The fpath and FPATH are connected in zsh. Changes to one affect the other. This allows use of more flexible and powerful array operations through the fpath ‘aspect’ of the value, but also provides compatibility to tools that expect the traditional colon-separated format in FPATH.

You will not be surprised to hear that zsh uses the same ‘magic’ with the PATH variable and its array counterpart path.

This means that you can continue to use path_helper to get your PATH from the files in /etc/paths and /etc/paths.d. (Well, you don’t have to, because on macOS this is done for all users in /etc/zprofile.) But then you can manipulate the path variable with array functions, like:

path+=~/bin

You get the useful aspects of both syntaxes.

Suffix Aliases

I learnt this one after writing the aliases part.

Suffix aliases take effect on the last part of a path, so usually the file extension. A suffix alias will assign a command to use when you just type a file path in the command line.

For example, you can a suffix alias for the txt file extension:

alias -s txt="open -t"

When you then type a path ending with .txt and no command, zsh will execute open -t /path/to/file.txt.

The open -t command opens a file in the default application set for the txt file extension in Finder. You probably want to set the suffix alias to bbedit or atom or something like that rather than open -t.

You can use other command line tools for the suffix alias:

alias -s log="tail -f"

Then, typing /var/log/install.log will show the last lines of that file and update the output when the file changes. If you prefer the graphical user interface, you can use the open -a command to assign suffix aliases to applications:

alias -s log="open -a Console"

You can even create a suffix alias using a different alias:

alias pacifist="open -a Pacifist"
alias -s pkg=pacifist

Together with the AutoCD option, this can improve your application-shell interactions a lot.

Bindkey for History Search

Most of the keyboard shortcuts in zsh work the same way as they do in bash. I have found one change that has proven quite useful:

^[[A' up-line-or-search # up arrow bindkey

^[[B' down-line-or-search # down arrow 

These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.

There are many commands or ‘widgets’ you can assign to keystrokes with the bindkey command. You can find a list of default ‘widgets’ in the documentation.

Conclusion

This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.

After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.

The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.

In the next (and last) post, I will cover the changes when scripting with zsh vs bash.

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

• Part 1: Moving to zsh
• Part 2: Configuration Files
• Part 3: Shell Options
• Part 4: Aliases and Functions
• Part 5: Completions
• Part 6: Customizing the zsh Prompt (this article)
• Part 7: Miscellanea
• Part 8: Scripting zsh

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

The default bash prompt on macOS is quite elaborate. It shows the username, the hostname, and the current directory.

Calypso:~ armin$

On the other hand, the default bash prompt doesn’t show the previous command’s exit code, a piece of information I find very useful. I have written before how I re-configured my bash prompt to have the information I want:

• Minimal Terminal Prompt
• Show Exit Code in your bash Prompt

Of course, I wanted to recreate the same experience in zsh.

The only (visual) difference to my bash prompt is the % instead of the $.

Note: creating a file ~/.hushlogin will suppress the status message at the start of each Terminal session in zsh as well as in bash (or any other shell).

Basic Prompt Configuration

The basic zsh prompt configuration works similar to bash, even though it uses a different syntax. The different placeholders are described in detail in the zsh manual.

zsh uses the same shell variable PS1 to store the default prompt. However, the variable names PROMPT and prompt are synonyms for PS1 and you will see either of those three being used in various examples. I am going to use PROMPT.

The default prompt in zsh is %m%#. The %m shows the first element of the hostname, the %# shows a # when the current prompt has super-user privileges (e.g. after a sudo -s) and otherwise the % symbol (the default zsh prompt symbol).

The zsh default prompt is far shorter than the bash default, but even less useful. Since I work on the local system most of the time, the hostname bears no useful information, and repeating it every line is superfluous.

Note: you can argue that the hostname in the prompt is useful when you frequently have multiple terminal windows open to different hosts. This is true, but then the prompt is defined by the remote shell and its configuration files on the remote host. In your configuration file, you can test if the SSH_CLIENT variable is set and show a different prompt for remote sessions. There are more ways of showing the host in remote shell sessions, for example in the Terminal window title bar or with different window background colors.

In our first iteration, I want to show the current working directory instead of the hostname. When you look through the list of prompt placeholders in the zsh documentation, you find %d, %/, and %~. The first two do exactly the same. The last substitution will display a path that starts with the user’s home directory with the ~, so it will shorten /Users/armin/Projects/ to ~/Projects.

Note: in the end you want to set your PROMPT variable in the .zshrc file, so it will take effect in all your zsh sessions. For testing, however, you can just change the PROMPT variable in the interactive shell. This will give you immediate feedback, how your current setup works.

% PROMPT='%/ %# '
/Users/armin/Projects/dotfiles/zshfunctions % 

% PROMPT='%~ %# '
~/Projects/dotfiles/zshfunctions % 

Note the trailing space in the prompt string, to separate the final % or # from the command entry.

I prefer the shorter output of the %~ option, but it can still be quite long, depending on your working directory. zsh has a trick for this: when you insert a number n between the % and the ~, then only the last n elements of the path will be shown:

% PROMPT='%2~ %# '
dotfiles/zshfunctions %                       

When you do %1~ it will show only the name of the working directory or ~ if it is the home directory. (This also works with %/, e.g. %2/.)

Adding Color

Adding a bit of color or shades of gray to the prompt can make it more readable. In bash you need cryptic escape codes to switch the colors. zsh provides an easier way. To turn the directory in the path blue, you can use:

PROMPT='%F{blue}%1~%f %# '

The F stands for ‘Foreground color.’ zsh understands the colors black, red, green, yellow, blue, magenta, cyan and white. %F or %f resets to the default text color. Furthermore, Terminal.app represents itself as a 256-color terminal to the shell. You can verify this with

% echo $TERM
xterm-256color

You can access the 256 color pallet with %F{0} through %F{255}. There are tables showing which number maps to which color:

• 256 Colors – Cheat Sheet – Xterm, HEX, RGB, HSL
• 256 Terminal colors and their 24bit equivalent (or similar)

So, since I want a dark gray for my current working dir in my prompt, I chose 240, I also set it to bold with the %B code:

PROMPT='%B%F{240}%1~%f%b %# '

You can find a detailed list of the codes for visual effects in the documentation.

Dynamic Prompt

I wrote an entire post on how to get bash to show the color-coded exit code of the last command. As it turns out, this is much easier in zsh.

One of the prompt codes provides a ‘ternary conditional,’ which means it will show one of two expressions, depending on a condition. There are several conditions you can use. Once again the details can be found in the documentation.

There is one condition for the previous commands exit code:

%(?.<success expression>.<failure expression>)

This expression will use the <success expression> when the previous command exited successfully (exit code zero) and <failure expression> when the previous command failed (non-zero exit code). So it is quite easy to build an conditional prompt:

% PROMPT='%(?.√.?%?) %1~ %# ' 
√ ~ % false
?1 ~ % 

You can get the √ character with option-V on the US or international macOS keyboard layout. The last part of the ternary ?%? looks confusing. The first ? will print a literal question mark, and the second part %? will be replaced with previous command’s exit code.

You can add colors in the ternary expression as well:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# ' 

Another interesting conditional code is ! which returns whether the shell is privileged (i.e. running as root) or not. This allows us to change the default prompt symbol from % to something else, while maintaining the warning functionality when running as root:

% PROMPT='%1~ %(!.#.>) ' 
~ > sudo -s
~ # exit
~ > 

Complete Prompt

Here is the complete prompt we assembled, with all the parts explained:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# '

%(?.√.?%?)	if return code ? is 0, show √, else show ?%?
%?	exit code of previous command
%1~	current working dir, shortening home to ~, show only last 1 element
%#	# with root privileges, % otherwise
%B %b	start/stop bold
%F{...}	text (foreground) color, see table
%f	reset to default textcolor

Right Sided Prompt

zsh also offers a right sided prompt. It uses the same placeholders as the ‘normal’ prompt. Use the RPROMPT variable to set the right side prompt:

% RPROMPT='%*'
√ zshfunctions %                    11:02:55

zsh will automatically hide the right prompt when the cursor reaches it when typing a long command. You can use all the other substitutions from the left side prompt, including colors and other visual markers in the right side prompt.

Git Integration

zsh includes some basic integration for version control systems. Once again there is a voluminous, but hard to understand description of it in the documentation.

I found a better, more specific example in the ‘Pro git’ documentation. This example will show the current branch on the right side prompt.

I have changed the example to include the repo name and the branch, and to change the color.

autoload -Uz vcs_info
precmd_vcs_info() { vcs_info }
precmd_functions+=( precmd_vcs_info )
setopt prompt_subst
RPROMPT=\$vcs_info_msg_0_
zstyle ':vcs_info:git:*' formats '%F{240}(%b)%r%f'
zstyle ':vcs_info:*' enable git

In this case %b and %r are placeholders for the VCS (version control system) system for the branch and the repository name.

There are git prompt solutions other than the built-in module, which deliver more information. There is a script in the git repository, and many of the larger zsh theme projects, such as ‘oh-my-zsh’ and ‘prezto’ have all kinds of git status widgets or modules or themes or what ever they call them.

Summary

You can spend (or waste) a lot of time on fine-tuning your prompt. Whether these modifications really improve your productivity is a matter of opinion.

In the next post, we will cover some miscellaneous odds and ends that haven’t yet really fit into any of preceding posts.

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

• Part 1: Moving to zsh
• Part 2: Configuration Files
• Part 3: Shell Options
• Part 4: Aliases and Functions
• Part 5: Completions (this article)
• Part 6: Customizing the zsh Prompt
• Part 7: Miscellanea
• Part 8: Scripting zsh

I am preparing a book on this topic, reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can pre-order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

What are Completions?

Man shells use the tab key (⇥) for completion. When you press that key, the shell tries to guess what you are typing and will complete it, or if the beginning of what you typed is ambiguous, suggest from a list of possible completions.

For example when you want to cd to your Documents folder, you can save typing:

% cd ~/Doc⇥

% cd ~/Documents/


When you hit the tab key, the system will complete the path to the Documents folder.

When the completion is ambiguous, the shell will list possible completions:

% cd ~/D⇥

Desktop/    Documents/  Downloads/


At this point, you can add a character or two to get to a unique completion, and hit the tab key again. In zsh you can also hit the tab key repeatedly to cycle through the suggested completions. In this example, the first tab keystroke will show the list, the second will complete ~/Desktop/, the third completes ~/Documents, and so on.

You can use tab completion commands as well:

% system⇥

system_profiler     systemkeychain      systemsetup         systemsoundserverd  systemstats

% system_⇥

% system_profiler 


Not having to type path and file names saves time and avoids errors, especially with complex paths with spaces and other special characters:

% cd ~/Li⇥
% cd ~/Library/Appl⇥

% cd ~/Library/Application S⇥

Application Scripts/  Application Support/

% cd ~/Library/Application Su⇥

% cd ~/Library/Application Support/


Using tab completion is a huge productivity boost when using a shell.

Turning It On

In the default configuration, tab completion in zsh is very basic. It will complete commands and paths, but not much else. But you can enable a very powerful, and useful completion system.

zsh comes with a tool you can use to setup this completion system. When you run the compinstall command it will lead you through a complex and hard to understand list of menus which explains the options and will generate the code necessary to set this configuration up and add it to your .zshrc file or another configuration file of your choice.

Since the commands to configure the completion are quite arcane and hard to understand, this is a good way to get something to start out with. I will explain some of these options and commands in detail.

Whether you use compinstall or not, to turn on the more powerful completion system, you need to add at least this command to your zsh configuration file:

autoload -Uz compinit && compinit


This will initialize the zsh completion system. The details of this system are documented here.

If you want to configure the system, the configuration commands (usually zstyle commands) should be added to the zsh configuration file before you enable the system. (This only matters for a few configurations, but as a general rule it is safer.)

All of these completion rules need to be loaded and prepared. zsh’s completion system creates a cache in the file ~/.zcompdump. The first time you run compinit it might take a noticeable time, but subsequent runs should use this cache and be much faster.

Sometimes, especially when building and debugging your own completion files, you may need to delete this file to force a rebuild:

% rm -f ~/.zcompdump
% compinit


Case Insensitive Completion

Since the macOS file systems are usually case-insensitive, I prefer my tab-completion to be case-insensitive as well. For bash you configure that in the ~/.inputrc. In zsh you modify the completion systems behavior with this (monstrous) command:

# case insensitive path-completion

zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*'


I have seen many varieties for this configuration in different websites, but this is what compinstall adds when I select case-insensitive completion, so I am going with that.

Partial Completion

This is a particularly nice feature. You can type fragments of each path segment and the completion will try to complete them all at once:

% cd /u/lo/b⇥

% cd /usr/local/bin


% cd ~/L/P/B⇥

% ~/Library/Preferences/ByHost/


If the fragments are ambiguous, there are different strategies to what the completion system suggests. I have configured these like this:

# partial completion suggestions

zstyle ':completion:*' list-suffixes
zstyle ':completion:*' expand prefix suffix


Commands with built-in completion

zsh comes with several completion definitions for many commands. For example, when you type cp and then hit tab, the system will correctly assume you want to complete a file path and show the suggestions from the current working directory.

However, when you type cp -⇥ the completion can tell from the - that you want to add an option to the command and suggest a list of options for cp, with short descriptions.

% cp -⇥
 
-H  -- follow symlinks on the command line in recursive mode
 
-L  -- follow all symlinks in recursive mode

 -P  -- do not follow symlinks in recursive mode (default)
 
-R  -- copy directories recursively

 -X  -- don't copy extended attributes or resource forks
 
-a  -- archive mode, same as -RpP

 -f  -- force overwriting existing file
 
-i  -- confirm before overwriting existing file

 -n  -- don't overwrite existing file
 
-p  -- preserve timestamps, mode, owner, flags, ACLs, and extended attributes
 -v  -- show file names as they are copied


As the context of command prompt you are assembling changes, you may get different completion suggestions. For example, the completion for ssh will suggest host names:

% ssh armin@⇥


zsh comes with completion definitions for many common commands. Nevertheless, it can be helpful to just hit tab, especially when wondering about options.

On macOS completions are stored in /usr/share/zsh/5.3/functions (replace the5.3 with 5.7.1 in Catalina). This directory stores many functions used with zsh and is in the default fpath. All the files in that directory that start with an underscore _ contain the completion definitions command. So, the file _cp contains the definition for the cp command. (Some of the definition files contain the definitions for multiple commands.)

Completions for macOS Commands

There are even a few macOS specific command that come with the default zsh installation.

% system_profiler ⇥⇥


macOS High Sierra and macOS Mojave come with zsh 5.3, which is now nearly two years old. zsh 5.3 contains less macOS specific completion definitions than the current zsh 5.7.1 which will is the pre-installed zsh in macOS Catalina. Some of the completions in 5.3 have also been updated in 5.7.1.

Tool	zsh 5.3	zsh 5.7.1
caffeinate		√
defaults	√	√
fink	√	√
fs_usage		√
hdiutil	√	√
mdfind		√
mdls		√
mdutil		√
networksetup		√
nvram		√
open	√	√
osascript		√
otool		√
pbcopy/pbpaste		√
plutil		√
say		√
sc_usage		√
scselect		√
scutil		√
softwareupdate	√	√
sw_vers		√
swift		√
system_profiler	√	√
xcode_select		√

Load bash completions

Since the default shell on macOS has been bash for so long, there are quite a few bash completion definitions for macOS commands and third party tools available. For example Tony Williams’ bash completion for autopkg (post, Github).

You do not have to rewrite these completions, since the zsh completion system can use bash completion scripts as well: (add this to your zsh configuration file)

# load bashcompinit for some old bash completions

autoload bashcompinit && bashcompinit


[[ -r ~/Projects/autopkg_complete/autopkg ]] && source ~/Projects/autopkg_complete/autopkg


When you have multiple bash completion scripts you want to load, you only need to load bashcompinit once.

Build your own completions

Once you start using completions, you will want to have them everywhere. While many built-in completions exists, there are still many commands that lack a good definition.

Some commands, like the swift command line tool, have a built-in option to generate the completion syntax. You can then store that in a file and put it in your fpath:

% swift package completion-tool generate-zsh-script >_swift


Note: in the case of swift, its definition will conflict with the _openstack definition in zsh 5.3. You can fix this with the command compdef _swift swift after loading the completion system.

Some commands provide a list of options and arguments with the -h/--help option. If this list follows a certain syntax, you can get a decent completion working with

% compdef _gnu_generic <command>


One example on macOS, where this has decent results is the xed command which opens a file or folder in Xcode.

But for best results, you will often have to build the description yourself. Unfortunately this is not a simple task. The syntax is meticulously, but also quite abstractly documented in the zsh documentation for the Completion System. I also found the ‘howto’ documentation in the zsh-completions repository very useful, as well as the ‘zsh Completion Style Guide.’

To avoid everyone re-inventing the wheel, I have started a repository on Github for macOS specific completion files. The page has the instructions on how to install them and I will welcome pull requests with contributions. Since I am just starting to learn this as well, I am sure there are improvements that can be made on the completions I have built so far and there are several commands where you can test your skills and build a new one.

I suggest the #zsh channel on the MacAdmins Slack for discussion.

Next

In the next post in this series, we will discuss how to configure zsh’s command line prompt.