Moving to zsh – part 7: Miscellanea

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

We have covered the general aspects of configuring your zsh environment and enabling some of its features to make your work more productive. However, there are zsh features that didn’t quite fit in earlier posts, but also don’t warrant a post of their own. So I am gathering them here.

multiIO

Terminal commands can take input from a file or a previous command (stdin) and have two different outputs: stdout and stderr. In bash you can redirect each of these to a single other destination.

For example, you can redirect the output of a command to a file:

% system_profiler SPHardwareDataType >hardwareinfo.txt

In zsh you can redirect to (and from) multiple sources. In the simplest form you can write the output to two files:

% system_profiler SPHardwareDataType >hardwareinfo.txt >computerinfo.txt

This is of course not a very realistic case. Since the pipe | is a form of redirection, you can combine output to a file with a pipe:

% system_profiler SPHardwareDataType >hardwareprofile.txt | cat

Instead of piping to cat, you can also redirect to stdout (or &1) as well as to a file:

% system_profiler SPHardwareDataType >&1 >hardwareprofile.txt

Note that the order of doing this is important. The construct >file.txt >&1 would redirect the output to file.txt and then redirect the output again to where stdout or 1 is going, so it would be redundant.

When combined with pipes and other commands multiIO can become very useful:

% system_profiler SPHardwareDataType >hardwareprofile.txt | awk '/Serial Number/ { print $4 }' >&1 >serialnumber.txt

You can use multiIO for input as well:

% sort </usr/share/calendar/calendar.freebsd </usr/share/calendar/calendar.computer

And while this not directly related, but somewhat close, in zsh, this

% <hardwareinfo.txt

is equivalent to more hardwareinfo.txt.

Recursive Globbing with **

You can use the ** to denote an arbitrary string that can span multiple directories in a path.

For example:

% echo Library/Preferences/**/com.apple.screensaver.*plist
Library/Preferences/ByHost/com.apple.screensaver.BBCCDDEE-AABB-CCDD-ABCD-00AABBCCDDEE.plist Library/Preferences/com.apple.screensaver.plist

In this case the ** matches nothing as well as /ByHost/.

Note: when used on large folder structures this glob can take a while. So use with care.

Connected array Variables

We already encountered the fpath variable in earlier posts. You can see its contents with the echo command:

% echo $fpath
/Users/armin/Projects/mac-zsh-completions/completions/ /Users/armin/Projects/dotfiles/zshfunctions /usr/local/share/zsh/site-functions /usr/share/zsh/site-functions /usr/share/zsh/5.3/functions

Interestingly enough, zsh also has an FPATH variable, which is a colon-separated list of directories:

% echo $FPATH
/Users/armin/Projects/mac-zsh-completions/completions/:/Users/armin/Projects/dotfiles/zshfunctions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.3/functions

Since the fpath variable is an array, I only changed the fpath variable in my zshrc.I never set or changed the FPATH, yet it reflects the changes made to the fpath variable.

When you see the type of both variables, you get an idea that something is going on:

% echo ${(t)fpath}
array-special
% echo ${(t)FPATH}
scalar-special

The fpath and FPATH are connected in zsh. Changes to one affect the other. This allows use of more flexible and powerful array operations through the fpath ‘aspect’ of the value, but also provides compatibility to tools that expect the traditional colon-separated format in FPATH.

You will not be surprised to hear that zsh uses the same ‘magic’ with the PATH variable and its array counterpart path.

This means that you can continue to use path_helper to get your PATH from the files in /etc/paths and /etc/paths.d. (Well, you don’t have to, because on macOS this is done for all users in /etc/zprofile.) But then you can manipulate the path variable with array functions, like:

path+=~/bin

You get the useful aspects of both syntaxes.

Suffix Aliases

I learnt this one after writing the aliases part.

Suffix aliases take effect on the last part of a path, so usually the file extension. A suffix alias will assign a command to use when you just type a file path in the command line.

For example, you can a suffix alias for the txt file extension:

alias -s txt="open -t"

When you then type a path ending with .txt and no command, zsh will execute open -t /path/to/file.txt.

The open -t command opens a file in the default application set for the txt file extension in Finder. You probably want to set the suffix alias to bbedit or atom or something like that rather than open -t.

You can use other command line tools for the suffix alias:

alias -s log="tail -f"

Then, typing /var/log/install.log will show the last lines of that file and update the output when the file changes. If you prefer the graphical user interface, you can use the open -a command to assign suffix aliases to applications:

alias -s log="open -a Console"

You can even create a suffix alias using a different alias:

alias pacifist="open -a Pacifist"
alias -s pkg=pacifist

Together with the AutoCD option, this can improve your application-shell interactions a lot.

Bindkey for History Search

Most of the keyboard shortcuts in zsh work the same way as they do in bash. I have found one change that has proven quite useful:

bindkey

These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.

There are many commands or ‘widgets’ you can assign to keystrokes with the bindkey command. You can find a list of default ‘widgets’ in the documentation.

Conclusion

This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.

After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.

The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.

In the next (and last) post, I will cover the changes when scripting with zsh vs bash.

^[[A' up-line-or-search # up arrow bindkey

These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.

There are many commands or ‘widgets’ you can assign to keystrokes with the bindkey command. You can find a list of default ‘widgets’ in the documentation.

Conclusion

This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.

After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.

The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.

In the next (and last) post, I will cover the changes when scripting with zsh vs bash.

^[[B' down-line-or-search # down arrow

These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.

There are many commands or ‘widgets’ you can assign to keystrokes with the bindkey command. You can find a list of default ‘widgets’ in the documentation.

Conclusion

This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.

After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.

The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.

In the next (and last) post, I will cover the changes when scripting with zsh vs bash.

Moving to zsh, part 6 – Customizing the zsh Prompt

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

The default bash prompt on macOS is quite elaborate. It shows the username, the hostname, and the current directory.

Calypso:~ armin$

On the other hand, the default bash prompt doesn’t show the previous command’s exit code, a piece of information I find very useful. I have written before how I re-configured my bash prompt to have the information I want:

Of course, I wanted to recreate the same experience in zsh.

Minimal Zsh Prompt

The only (visual) difference to my bash prompt is the % instead of the $.

Note: creating a file ~/.hushlogin will suppress the status message at the start of each Terminal session in zsh as well as in bash (or any other shell).

Basic Prompt Configuration

The basic zsh prompt configuration works similar to bash, even though it uses a different syntax. The different placeholders are described in detail in the zsh manual.

zsh uses the same shell variable PS1 to store the default prompt. However, the variable names PROMPT and prompt are synonyms for PS1 and you will see either of those three being used in various examples. I am going to use PROMPT.

The default prompt in zsh is %m%#. The %m shows the first element of the hostname, the %# shows a # when the current prompt has super-user privileges (e.g. after a sudo -s) and otherwise the % symbol (the default zsh prompt symbol).

The zsh default prompt is far shorter than the bash default, but even less useful. Since I work on the local system most of the time, the hostname bears no useful information, and repeating it every line is superfluous.

Note: you can argue that the hostname in the prompt is useful when you frequently have multiple terminal windows open to different hosts. This is true, but then the prompt is defined by the remote shell and its configuration files on the remote host. In your configuration file, you can test if the SSH_CLIENT variable is set and show a different prompt for remote sessions. There are more ways of showing the host in remote shell sessions, for example in the Terminal window title bar or with different window background colors.

In our first iteration, I want to show the current working directory instead of the hostname. When you look through the list of prompt placeholders in the zsh documentation, you find %d, %/, and %~. The first two do exactly the same. The last substitution will display a path that starts with the user’s home directory with the ~, so it will shorten /Users/armin/Projects/ to ~/Projects.

Note: in the end you want to set your PROMPT variable in the .zshrc file, so it will take effect in all your zsh sessions. For testing, however, you can just change the PROMPT variable in the interactive shell. This will give you immediate feedback, how your current setup works.

% PROMPT='%/ %# '
/Users/armin/Projects/dotfiles/zshfunctions % 

% PROMPT='%~ %# '
~/Projects/dotfiles/zshfunctions % 

Note the trailing space in the prompt string, to separate the final % or # from the command entry.

I prefer the shorter output of the %~ option, but it can still be quite long, depending on your working directory. zsh has a trick for this: when you insert a number n between the % and the ~, then only the last n elements of the path will be shown:

% PROMPT='%2~ %# '
dotfiles/zshfunctions %                       

When you do %1~ it will show only the name of the working directory or ~ if it is the home directory. (This also works with %/, e.g. %2/.)

Adding Color

Adding a bit of color or shades of gray to the prompt can make it more readable. In bash you need cryptic escape codes to switch the colors. zsh provides an easier way. To turn the directory in the path blue, you can use:

PROMPT='%F{blue}%1~%f %# '

The F stands for ‘Foreground color.’ zsh understands the colors black, red, green, yellow, blue, magenta, cyan and white. %F or %f resets to the default text color. Furthermore, Terminal.app represents itself as a 256-color terminal to the shell. You can verify this with

% echo $TERM
xterm-256color

You can access the 256 color pallet with %F{0} through %F{255}. There are tables showing which number maps to which color:

So, since I want a dark gray for my current working dir in my prompt, I chose 240, I also set it to bold with the %B code:

PROMPT='%B%F{240}%1~%f%b %# '

You can find a detailed list of the codes for visual effects in the documentation.

Dynamic Prompt

I wrote an entire post on how to get bash to show the color-coded exit code of the last command. As it turns out, this is much easier in zsh.

One of the prompt codes provides a ‘ternary conditional,’ which means it will show one of two expressions, depending on a condition. There are several conditions you can use. Once again the details can be found in the documentation.

There is one condition for the previous commands exit code:

%(?.<success expression>.<failure expression>)

This expression will use the <success expression> when the previous command exited successfully (exit code zero) and <failure expression> when the previous command failed (non-zero exit code). So it is quite easy to build an conditional prompt:

% PROMPT='%(?.√.?%?) %1~ %# ' 
√ ~ % false
?1 ~ % 

You can get the character with option-V on the US or international macOS keyboard layout. The last part of the ternary ?%? looks confusing. The first ? will print a literal question mark, and the second part %? will be replaced with previous command’s exit code.

You can add colors in the ternary expression as well:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# ' 

Another interesting conditional code is ! which returns whether the shell is privileged (i.e. running as root) or not. This allows us to change the default prompt symbol from % to something else, while maintaining the warning functionality when running as root:

% PROMPT='%1~ %(!.#.>) ' 
~ > sudo -s
~ # exit
~ > 

Complete Prompt

Here is the complete prompt we assembled, with all the parts explained:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# '
%(?.√.?%?) if return code ? is 0, show , else show ?%?
%? exit code of previous command
%1~ current working dir, shortening home to ~, show only last 1 element
%# # with root privileges, % otherwise
%B %b start/stop bold
%F{...} text (foreground) color, see table
%f reset to default textcolor

Right Sided Prompt

zsh also offers a right sided prompt. It uses the same placeholders as the ‘normal’ prompt. Use the RPROMPT variable to set the right side prompt:

% RPROMPT='%*'
√ zshfunctions %                    11:02:55

zsh will automatically hide the right prompt when the cursor reaches it when typing a long command. You can use all the other substitutions from the left side prompt, including colors and other visual markers in the right side prompt.

Git Integration

zsh includes some basic integration for version control systems. Once again there is a voluminous, but hard to understand description of it in the documentation.

I found a better, more specific example in the ‘Pro git’ documentation. This example will show the current branch on the right side prompt.

I have changed the example to include the repo name and the branch, and to change the color.

autoload -Uz vcs_info
precmd_vcs_info() { vcs_info }
precmd_functions+=( precmd_vcs_info )
setopt prompt_subst
RPROMPT=\$vcs_info_msg_0_
zstyle ':vcs_info:git:*' formats '%F{240}(%b)%r%f'
zstyle ':vcs_info:*' enable git

In this case %b and %r are placeholders for the VCS (version control system) system for the branch and the repository name.

There are git prompt solutions other than the built-in module, which deliver more information. There is a script in the git repository, and many of the larger zsh theme projects, such as ‘oh-my-zsh’ and ‘prezto’ have all kinds of git status widgets or modules or themes or what ever they call them.

Summary

You can spend (or waste) a lot of time on fine-tuning your prompt. Whether these modifications really improve your productivity is a matter of opinion.

In the next post, we will cover some miscellaneous odds and ends that haven’t yet really fit into any of preceding posts.

Moving to zsh, part 5: Completions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class in Amsterdam on September 6! Visit our website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

What are Completions?

Man shells use the tab key (⇥) for completion. When you press that key, the shell tries to guess what you are typing and will complete it, or if the beginning of what you typed is ambiguous, suggest from a list of possible completions.

For example when you want to cd to your Documents folder, you can save typing:

% cd ~/Doc⇥
% cd ~/Documents/

When you hit the tab key, the system will complete the path to the Documents folder.

When the completion is ambiguous, the shell will list possible completions:

% cd ~/D⇥
Desktop/    Documents/  Downloads/

At this point, you can add a character or two to get to a unique completion, and hit the tab key again. In zsh you can also hit the tab key repeatedly to cycle through the suggested completions. In this example, the first tab keystroke will show the list, the second will complete ~/Desktop/, the third completes ~/Documents, and so on.

You can use tab completion commands as well:

% system⇥
system_profiler     systemkeychain      systemsetup         systemsoundserverd  systemstats
% system_⇥
% system_profiler 

Not having to type path and file names saves time and avoids errors, especially with complex paths with spaces and other special characters:

% cd ~/Li⇥
% cd ~/Library/Appl⇥
% cd ~/Library/Application S⇥
Application Scripts/  Application Support/
% cd ~/Library/Application Su⇥
% cd ~/Library/Application Support/

Using tab completion is a huge productivity boost when using a shell.

Turning It On

In the default configuration, tab completion in zsh is very basic. It will complete commands and paths, but not much else. But you can enable a very powerful, and useful completion system.

zsh comes with a tool you can use to setup this completion system. When you run the compinstall command it will lead you through a complex and hard to understand list of menus which explains the options and will generate the code necessary to set this configuration up and add it to your .zshrc file or another configuration file of your choice.

Since the commands to configure the completion are quite arcane and hard to understand, this is a good way to get something to start out with. I will explain some of these options and commands in detail.

Whether you use compinstall or not, to turn on the more powerful completion system, you need to add at least this command to yourzsh` configuration file:

autoload -Uz compinit && compinit

This will initialize the zsh completion system. The details of this system are documented here.

If you want to configure the system, the configuration commands (usually zstyle commands) should be added to the zsh configuration file before you enable the system. (This only matters for a few configurations, but as a general rule it is safer.)

All of these completion rules need to be loaded and prepared. zsh’s completion system creates a cache in the file ~/.zcompdump. The first time you run compinit it might take a noticeable time, but subsequent runs should use this cache and be much faster.

Sometimes, especially when building and debugging your own completion files, you may need to delete this file to force a rebuild:

% rm -f ~/.zcompdump
% compinit

Case Insensitive Completion

Since the macOS file systems are usually case-insensitive, I prefer my tab-completion to be case-insensitive as well. For bash you configure that in the ~/.inputrc. In zsh you modify the completion systems behavior with this (monstrous) command:

# case insensitive path-completion
zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*'

I have seen many varieties for this configuration in different websites, but this is what compinstall adds when I select case-insensitive completion, so I am going with that.

Partial Completion

This is a particularly nice feature. You can type fragments of each path segment and the completion will try to complete them all at once:

% cd /u/lo/b⇥
% cd /usr/local/bin
% cd ~/L/P/B⇥
% ~/Library/Preferences/ByHost/

If the fragments are ambiguous, there are different strategies to what the completion system suggests. I have configured these like this:

# partial completion suggestions
zstyle ':completion:*' list-suffixes
zstyle ':completion:*' expand prefix suffix

Commands with built-in completion

zsh comes with several completion definitions for many commands. For example, when you type cp and then hit tab, the system will correctly assume you want to complete a file path and show the suggestions from the current working directory.

However, when you type cp -⇥ the completion can tell from the - that you want to add an option to the command and suggest a list of options for cp, with short descriptions.

% cp -⇥ 
-H  -- follow symlinks on the command line in recursive mode
-L  -- follow all symlinks in recursive mode
-P  -- do not follow symlinks in recursive mode (default)
-R  -- copy directories recursively
-X  -- don't copy extended attributes or resource forks
-a  -- archive mode, same as -RpP
-f  -- force overwriting existing file
-i  -- confirm before overwriting existing file
-n  -- don't overwrite existing file
-p  -- preserve timestamps, mode, owner, flags, ACLs, and extended attributes
-v  -- show file names as they are copied

As the context of command prompt you are assembling changes, you may get different completion suggestions. For example, the completion for ssh will suggest host names:

% ssh armin@⇥

zsh comes with completion definitions for many common commands. Nevertheless, it can be helpful to just hit tab, especially when wondering about options.

On macOS completions are stored in /usr/share/zsh/5.3/functions (replace the5.3 with 5.7.1 in Catalina). This directory stores many functions used with zsh and is in the default fpath. All the files in that directory that start with an underscore _ contain the completion definitions command. So, the file _cp contains the definition for the cp command. (Some of the definition files contain the definitions for multiple commands.)

Completions for macOS Commands

There are even a few macOS specific command that come with the default zsh installation.

% system_profiler ⇥⇥

macOS High Sierra and macOS Mojave come with zsh 5.3, which is now nearly two years old. zsh 5.3 contains less macOS specific completion definitions than the current zsh 5.7.1 which will is the pre-installed zsh in macOS Catalina. Some of the completions in 5.3 have also been updated in 5.7.1.

Tool zsh 5.3 zsh 5.7.1
caffeinate
defaults
fink
fs_usage
hdiutil
mdfind
mdls
mdutil
networksetup
nvram
open
osascript
otool
pbcopy/pbpaste
plutil
say
sc_usage
scselect
scutil
softwareupdate
sw_vers
swift
system_profiler
xcode_select

Load bash completions

Since the default shell on macOS has been bash for so long, there are quite a few bash completion definitions for macOS commands and third party tools available. For example Tony Williams’ bash completion for autopkg (post, Github).

You do not have to rewrite these completions, since the zsh completion system can use bash completion scripts as well: (add this to your zsh configuration file)

# load bashcompinit for some old bash completions
autoload bashcompinit && bashcompinit

[[ -r ~/Projects/autopkg_complete/autopkg ]] && source ~/Projects/autopkg_complete/autopkg

When you have multiple bash completion scripts you want to load, you only need to load bashcompinit once.

Build your own completions

Once you start using completions, you will want to have them everywhere. While many built-in completions exists, there are still many commands that lack a good definition.

Some commands, like the swift command line tool, have a built-in option to generate the completion syntax. You can then store that in a file and put it in your fpath:

% swift package completion-tool generate-zsh-script >_swift

Note: in the case of swift, its definition will conflict with the _openstack definition in zsh 5.3. You can fix this with the command compdef _swift swift after loading the completion system.

Some commands provide a list of options and arguments with the -h/--help option. If this list follows a certain syntax, you can get a decent completion working with

% compdef _gnu_generic <command>

One example on macOS, where this has decent results is the xed command which opens a file or folder in Xcode.

But for best results, you will often have to build the description yourself. Unfortunately this is not a simple task. The syntax is meticulously, but also quite abstractly documented in the zsh documentation for the Completion System. I also found the ‘howto’ documentation in the zsh-completions repository very useful, as well as the ‘zsh Completion Style Guide.’

To avoid everyone re-inventing the wheel, I have started a repository on Github for macOS specific completion files. The page has the instructions on how to install them and I will welcome pull requests with contributions. Since I am just starting to learn this as well, I am sure there are improvements that can be made on the completions I have built so far and there are several commands where you can test your skills and build a new one.

I suggest the #zsh channel on the MacAdmins Slack for discussion.

Next

In the next post in this series, we will discuss how to configure zsh’s command line prompt.

Install Bash 5 on macOS with Patches

I recently posted an article on how to download, install, and build a macOS installer pkg for bash 5. In that first version of this post I ignored patches, minor updates to the bash source code and binary. But as the patches to bash 5 are accumulating, I cannot ignore them much longer.

This post will extend the instructions in the original post.

After downloading and expanding the bash-5.0.tar.gz, create a patches folder:

$ cd path/to/bash-5.0
$ mkdir patches
$ cd patches

You can download the patches for bash-5.0 here. As of this writing, there are seven patches for bash-5.0 labelled bash50-001 through bash50-007. You can download all at once with:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' -O

(Adapt the numbers when there are more patches in the future.)

Then move up one directory level to the bash-5.0 root directory and apply the patches using the patch command.

$ cd ..
$ patch -p0 -i patches/bash50-001
$ patch -p0 -i patches/bash50-002

(etc.)

You can download and patch with a single step. Make sure your working directory is the bash-5.0 with all the code and run:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' | patch -p0

From here, you can continue with the remaining build steps from the original post. The next step will be running ./configure.

The script to build the pkg installer has also been updated in the repository to download and apply the patches before building.

Moving to zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

zsh (I believe it is pronounced zee-shell, though zish is fun to say) will succeed bash as the default shell. bash has been the default shell since Mac OS X 10.3 Panther.

Why?

The bash binary bundled with macOS has been stuck on version 3.2 for a long time now. bash v4 was released in 2009 and bash v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3. bash v3 is still GPL v2.

zsh, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default. zsh has been available as on macOS for a long time. The zsh version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current zsh 5.7.1.

Is bash gone!?

No.

macOS Catalina still has the same /bin/bash (version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.

Many scripts in macOS, management systems, and Apple and third party installers rely on /bin/bash. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.

Users that have /bin/bash as their default shell on Catalina will see a prompt at the start of each Terminal session stating that zsh is now the recommended default shell. If you want to continue using /bin/bash, you can supress this message by setting an environment variable in your .bash_profile or .bashrc.

export BASH_SILENCE_DEPRECATION_WARNING=1

You can also download and install a newer version of bash yourself. Keep in mind that custom bash installations reside in a different directory, usually /usr/local/bin/bash.

Will bash remain indefinitely?

Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to bash, but didn’t really care if you remained on tcsh. In fact, tcsh is still present on macOS.

Apple’s messaging should tell us, that the days of /bin/bash are numbered. Probably not very soon, but eventually keeping a more than ten year old version of bash on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.

Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’ dash has been added to the lineup of shells. dash is designed to be a minimal implementation of the Posix standard shell sh. So far, in macOS (including Catalina),sh invokes bash in sh-compatibility mode.

As Apple’s support article mentions, Catalina also adds a new mechanism for users and admins to change which shell handles sh invocations. MacAdmins or users can change the symbolic link stored in /var/select/sh to point to a shell other than /bin/bash. This changes which shell interprets scripts the #!/bin/sh shebang or scripts invoked with sh -c. Changing the interpreter for sh should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.

All of these changes are indicators that Apple is preparing to remove /bin/bash at some, yet indeterminate, time in the future.

Do I need to wait for Catalina to switch to zsh?

No, zsh is available Mojave and on older macOS versions. You can start testing zsh or even switch your default shell already.

If you want to just see how zsh works, you can just open Terminal and type zsh:

$ zsh
MacBook%

The main change you will see is that the prompt looks different. zsh uses the % character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in bash.

If you want to already switch your default shell to zsh you can use the chsh command:

$ chsh -s /bin/zsh

This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.

How is zsh different?

Like bash (‘Bourne again shell’ ), zshderives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.

The main difference between bash and zsh is configuration. Since zsh ignores the bash configuration files (.bash_profile or .bashrc) you cannot simply copy customized bash settings over to zsh. zsh has much more options and points to change zsh configuration and behavior. There is an entire eco-system of configuration tools and themes called oh-my-zsh which is very popular.

zsh also offers better configuration for auto-completion which is far easier than in bash.

I am planning a separate post, describing how to transfer (and translate) your configurations from bash to zsh.

What about scripting?

Since zsh has been present on macOS for a long time, you could start moving your scripts from bash to zsh right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to #!/bin/zsh.

You will gain some features where zsh is superior to bash v3, such as arrays and associative arrays (dictionaries).

There is one exception where I would now recommend to use /bin/sh for your scripts: the Recovery system does not contain the /bin/zsh shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider older macOS installations where zsh is definitely not present in Recovery.

When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as Twocanoes MDS, installr or bootstrappr, then you cannot rely on /bin/zsh.

Since we now know that bash is eventually going away, the only common choice left is /bin/sh.

When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using /bin/sh as the shebang from now on.

I used to recommend using /bin/bash for everything MacAdmin related. /bin/sh is definitely a step down in fucntionality, but it seems like the safest choice for continued support.

Summary

Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’ bash with zsh, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!

This git repo has been shared by many on MacAdmins Slack: rothgar/mastering-zsh, I will certainly dive into that and share about my experiences here!

Next

In the next part we will look at the configuration files for zsh.

Open Apps with custom Shortcuts in macOS

Someone on the MacAdmins Slack recently asked how you could assign a global keyboard short cut to open Terminal on macOS.

Note: alternative terminal applications such as iTerm2 may have this built-in.

macOS has an option to assign custom global keystrokes to pretty much anything, but it is not obvious how to get there.

  • First, open the Automator application. In the chooser for a new Workflow, choose ‘Quick Action’ (on Mojave) or ‘Service’ on earlier versions of macOS.
The new Workflow chooser in Mojave
The new Workflow chooser in Mojave
  • In the new workflow configure the input to be ‘no input’ and the application to be ‘any application.’
  • Then search for ‘Launch Application’ action in the library pane on the left and add it to your workflow by double-clicking or dragging.
  • The popup menu where you can slect an application in the action will only show applications from the /Applications folder. Choose ‘Other…’ and select Terminal in the ’/Applications/Utilities` folder.
Configure your workflow
Configure your workflow
  • Save the workflow. Give it a meaningful name such as ‘Open Terminal.’ Since you chose Quick Action or Service, this workflow will be saved in ~/Library/Services.
  • Open System Preferences > Keyboard. Click the ‘Shortcuts’ tab and select ‘Services’ from the list on the left side. (Even on Mojave, it is still called ‘Services’.)
  • Scroll all the way down the list of services under the ‘General’ heading, you should find the service you just created. Select it and click ‘Add Shortcut’ to assign a global shortcut.
Keyboard Shortcut Preferences
Keyboard Shortcut Preferences
  • You are done!

When the active application uses the same keystroke, the application’s definition will precede your global shortcut.

Of course, you don’t have stop at launching applications. You can assign a global keyboard shortcut to any Automator workflow this way. Since Automator workflows can include AppleScript, Python or shell scripts, you can do pretty much anything this way!

However, most Apple users don’t bother with shortcuts to launch apps. Just invoke Spotlight with command-space and start typing term and hit return.

On Smart Quotes and Terminal

Typography is a wonderful art and has a long history. When humans turned from manual typesetting to machines, type writers and then computers, some compromises had to be made. One of these compromises was to use simple straight quote symbols for opening and closing the quote, rather than different quote symbols for opening and closing.

Note: which kind of quotes are used for opening and closing dependent on the language or and some convention. English uses upper quotes “…”, German opens with a lower quote: „…“, French uses ‘guillemets:’ «…», and Japanese uses hooks: 「…」

Quotation Marks have funny names in many languages. Germans call them “Gänsefüßchen,” or “little goose feet.”

See this Wikipedia article for more details.

macOS, iOS and other modern operating systems have a feature which replaces the simple or straight quote symbols with the typographic quotes. So, you type "Hello!" and the quotes are automatically replaced with the proper (depending on localization) typographic quotes. This is called “smart quotes.”

This is pretty nice, but can be troublesome when dealing with Terminal and text editors. Scripting languages and shells always use straight quotes, and cannot deal with typographic quotes.

Now, if someone sends you a command or a script that uses quotes, and it goes through an app that replaces them with smart quotes, then bash and Terminal will fail miserably.

There is not much you can do, other than be aware of this and check pasted code carefully. There is something you can do to make this easier, though.

The default monospace font used in Terminal on macOS are ‘Menlo’ or ‘SF Mono,’ depending on the macOS version. Now these are fine typefaces, but their typographic quotes are not very curly at all, making them very hard to distinguish from the ‘dumb’ straight quotes that Terminal expects. The classic ‘Monaco’ typeface on the other hand has beautiful curly typographic quotes, making them very distinct from the the straight quote.

My favorite mono space typeface ‘Source Code Pro’ also has nice curly typographic quotes. I have built this table with many common monospace typefaces and their quotes.

Quotation Mark Comparison
Quotation Mark Comparison

Now this shouldn’t be your only criteria in choosing your Terminal font, but it may be something that helps avoid quote errors.

Install Bash 5 on macOS

The default bash on macOS is still bash v3:

$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.

Just recently, bash v5 was released. The discrepancy comes from the fact that bash has been licensed as GPL v3 since version 4. Apple does not include GPL v3 licensed tools with macOS.

However, nothing is keeping you from downloading and installing the latest bash version.

New features include, among many other things, associated arrays (i.e. dictionaries) and better auto-completion setup.

While you would think this is a common desire, most pages I have found will simply point to Homebrew to download and install a newer bash version.

The main challenge with using brew is that it does not work on the scale that MacAdmins require. brew is designed for single user installation, where the user has administrator privileges. brew’s workflows do not scale to large deployments controlled with a management system.

Ideally, there would be package installer for the latest bash version. Unfortunately, the bash project does not provide one.

In this post, I will show how you can install the latest bash version without brew and how to build an installer package for deployment.

Manual Installation

This requires Xcode or the Developer Command Line Tools to be installed.

First, download the source for the latest bash version from this page. As of this writing the latest version is bash-5.0 and the file you want is bash-5.0.tar.gz. Once downloaded, you can expand the archive in Finder by double-clicking.

Update: I have a post with some updated instructions to include the patches to bash 5.0.

Open a Terminal window and change directory to the newly expanded bash-5.0 directory. Then run the configure script there.

$ cd ~/Downloads/bash-5.0
$ ./configure

The configure process will take a while, there will be plenty of messages showing progress.

Once the configure process is complete. You can build bash with the make command.

$ make

This will build the bash binary and the supporting files in the current directory. That’s not where we want it in the end, but it is probably a good idea see if the build process works. This will (again) take a while. There will be some odd looking warnings, but you can ignore those.

When make succeeds, you can actually install bash v5 with

$ sudo make install

This will build and install the bash binary and supporting files in /usr/local/bin and /usr/local. sudo is required to modify /usr/local.

If you were just looking for a way to install bash v5 without brew, you are done!

There is more useful information in the rest of the post, though, so keep reading!

How the new and the old bash interact

By default, the bash v5 binary is called bash and will be installed in /usr/local/bin. The macOS default PATH lists /usr/local/bin before /bin where the default bash v3 binary, also called bash, is located.

This means, that when a user types bash in to a shell, the version in /usr/local/bin will be preferred over the pre-installed bash v3.

You can test this behavior in Terminal. Since the default shell has not yet been changed from /bin/bash the Terminal still opens to bash v3. You can test this by showing the BASH_VERSION environment variable:

$ echo $BASH_VERSION
3.2.57(1)-release

But when you then run bash it will invoke /usr/local/bin/bash, so it will run the new bash v5. It will show this in the prompt, but you can also verify the BASH_VERSION.

$ bash
bash-5.0$ echo $BASH_VERSION
5.0.0(2)-release

This might be the setup you want, when you want to use bash v5 always. It might lead to some unexpected behavior for some users, though.

One option to avoid this ambiguity is to rename the binary in /usr/local/bin to bash5. But then other tools such as env (mentioned below) will not find the binary any more.

Note: the PATH in other contexts will likely not contain /usr/local/bin and further confuse matters.

bash v5 and Scripting

Scripts using bash, should have the full path to the binary in the shebang. This way, the script author can control whether a script is executed by the default bash v3 (/bin/bash) or the newer bash v5 (/usr/local/bin/bash or /usr/local/bin/bash5).

It is often recommended to use the env command in the shebang:

#!/usr/bin/env bash

The env command will determine the path to the bash binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the bash binary is unknown, in other words across multiple Unix and Unix-like platforms. However, this renders the actual version of bash that will interpret the script unpredictable.

For example, assume you have bash v5 installed in the default configuration (as /usr/local/bin/bash. A script with the shebang #!/usr/bin/env bash launched in the user environment (i.e. from Terminal) will use the newer bash, as /usr/local/bin comes before /bin in the search order.

When you launch the same script in a different context, e.g. as an installation script, an AppleScript, or a management system, /usr/local/bin will likely not be part of the PATH in that environment. Then the env shebang will choose /bin/bash (v3). The script will be interpreted and might behave differently.

Administrators prefer certainty in their managed environments. Administrators should know the location and versions of the binaries on their systems. For management scripts, you should avoid env and use the proper full path to the desired interpreter binary.

The solutions to resolve the ambiguity are

  • use the full path to the binary in the shebang
  • manage and update the additional custom version of bash with a management system
  • (optional) rename the newer bash binary to bash5 or bash4 (this also allows you to have bash v4 and bash v5 available on the same system)
  • Scripting OS X: On the Shebang
  • Scripting OS X: Setting the PATH in Scripts

Changing a user’s default Shell to bash v5

Even though we have installed bash v5, the default shell of a new Terminal window will still use the built-in bash v3.

The path to the default shell is stored in the user record. You can directly change the UserShell attribute with dscl, in the ‘Advanced Options’ of the ‘Users & Groups’ preference pane, or in Directory Utility.

There is also a command to set the default shell:

$ chsh -s /usr/local/bin/bash
Changing shell for armin.
Password for armin: 
chsh: /usr/local/bin/bash: non-standard shell

The chsh (change shell) command will check for allowed shells in the /etc/shells file. You can easily append a line with /usr/local/bin/bash to this file, and then chsh will work fine.

$ chsh -s /usr/local/bin/bash
Changing shell for armin.
Password for armin: 

Note: if you choose to rename the bash binary, you have to use the changed name in /etc/shells and with chsh.

Remember that just running chsh will not change the shell in the current Terminal window. It is best to close the old Terminal window and open a new one to get the new shell.

Packaging bash v5 for mass deployment

While these steps to install and configure bash v5 on a single Mac are simple enough, they would not work well with a management system for hundreds or thousands of Macs. We want to wrap all the files that make install creates into a package installer payload.

The --help option of the configure script yields this useful information:

By default, make install' will install all the files in/usr/local/bin,/usr/local/libetc. You can specify an installation prefix other than/usr/localusing–prefix, for instance–prefix=$HOME`.

When we run the configure script with the --prefix option it creates a folder suitable as a payload for a package installer. We can then use pkgbuild to build to create an installer pkg:

$ cd ~/Downloads/bash-5.0
$ mkdir payload
$ ./configure --prefix=/Users/armin/Downloads/bash-5.0/payload
$ make install
$ pkgbuild --root payload --install-location /usr/local --identifier org.gnu.bash --version 5.0 bash-5.0.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to bash-5.0.pkg

(Note: the --prefix argument requires an absolute path.)

Automate the package creation

So, we have our workflow for building an installer package to distribute and configure bash v5:

  • download the archive
  • extract the archive
  • run configure with the --prefix argument
  • run make install to create the files in a payload folder
  • optional: rename the resulting bash binary to bash5 to avoid conflicts
  • add a postinstall script that adds /usr/local/bin/bash[5] to /etc/shells if not yet present
  • build the installer with pkgbuild

This sounds like a workflow ripe for automation. You can get the script from this repository.

You can pass a different (valid) bash version number as an argument to the script, e.g. 4.4.18. (I did not test anything significantly older.) The script does not autodetect the latest version and defaults to version 5.0 when no argument is given. When an update to bash v5 is published, you will have to modify the version line or run the script with an argument.

I have not (yet) figured out how to detect the latest version from the download web page. An autopkg recipe will have to wait for that. (If someone else wants to tackle that, please do!)

Apple Two-Factor Authentication for a Secondary Apple ID

Apple sent an email to developers, stating that later this months, two-factor authentication will be required for Apple IDs used for developer accounts.

If you, like me, use separate Apple IDs for your personal iCloud and your developer accounts, this will pose some kind of challenge. There is a solution, however Apple does not document it very well.

Update: Apple now has a very detailed support page for this topic.

Two-factor authentication for the primary account

Assumption: you have two-factor authentication (2FA) enabled on your primary, personal Apple ID, and are logged in to that account on your Mac(s) and iOS devices. If you haven’t done that yet, do it now. 2FA does increase your account security significantly.

You can enable 2FA on any device logged in to the account in the iCloud Settings or preference pane. As part of the setup you can provide one or more phone numbers as a fall back mechanism. If no devices can be prompted through Apple’s built-in 2FA, it will send an SMS to the trusted phone numbers. You can use the same phone number for multiple Apple IDs, but there seems to be some limit on how often you can do that.

Enable 2FA for the secondary account

Assumption: The secondary account is your developer Apple ID, you don’t use it for iCloud storage, device backups, mail etc. You use it to log in to developer.apple.com and iTunes Connect, and to get all the certificates and other resources you need as a developer.

The challenge here is that you can only enable 2FA on the first account logged in to iCloud on a device. You could log out of your primary iCloud account, and the log in with the secondary, but this will disrupt a lot of things on your device. I’d rather avoid that.

On a Mac, you can have a separate iCloud account for each local user. So, it is easiest to create a second user account, log out of your first account, log in to the new second account and set up iCloud and 2FA for the developer Apple ID on this second local account.

You can sign in to the secondary Apple ID enable 2FA in System Preferences -> iCloud -> Account Details -> Security as described in Apple’s Support Article.

Follow the prompts to set up 2FA, you can re-use the same phone number as a trusted number. (There seem to be limits to how often you use the same phone number, but two accounts works fine for me.)

Once 2FA is set up, we don’t need the second user account on the Mac any more. Sign out of iCloud, log out of the second account and back in to your normal user account.

If you are ok with using SMS authentication (Apple calls this ‘two-step authentication’, rather than ‘two-factor authentication’) then you are done. However, many will argue codes over SMS are not good enough for secondary authentication, so we want go to ‘full’ 2FA.

Use the secondary Apple ID

As it turns out, you can be logged in to multiple iCloud accounts on the same device or account. Certain services, such as iCloud storage, or the Photo Library, will only work with the primary iCloud account, but other services, including 2FA, will work for all iCloud accounts.

On your iOS device go to Settings > Passwords & Accounts > Add Account, and choose to add another iCloud account. You probably want to turn off all services, like Mail, Calendar, etc. secondary account.

Second iCloud Account on iPhone
Second iCloud Account on iPhone

On the Mac you can do the same in System Preferences > Internet Accounts. You can use both your Mac and iOS devices for 2FA.

Second iCloud Account on macOS
Second iCloud Account on macOS

Now the secondary Apple ID will prompt the devices you are logged in as for 2FA.

2FA prompt on the iPhone
2FA prompt on the iPhone

PLIST Editor can Open Signed Mobileconfigs

Last year I posted about a useful app to read and edit property list files: PLIST Editor.

In its last update (1.15) the app gained a feature which is tremendously useful to MacAdmins: it can now open signed property lists.

MacAdmins regularly encounter signed property lists when they download configuration profiles (.mobileconfig) from a management server. Since the cryptographic signature is binary data wrapping the property list, most property list editors, such as Xcode or PlistEdit Pro, choke on the signature.

The new PLIST Editor (1.15) now detects a signed mobileconfig and will automatically unwrap the plist data from the signature and display a notification.

PLIST Editor Message

When you click on the notification, you will get more detailed information on the signature itself.

Once you edit and save the mobileconfig, the signature will obviously be removed. You can re-sign the edited mobileconfig file with a tool like Hancock or in the Terminal with:

$ security cms -S -N "Identity Name" -i profile.mobileconfig -o signed.mobileconfig

(Learn more on in my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’)

This addition will be very useful for my workflow and think for other MacAdmins as well.

PLIST Editor is available in the Mac App Store for US$3.99 (price will vary depending on region).