Install Bash 5 on macOS with Patches

I recently posted an article on how to download, install, and build a macOS installer pkg for bash 5. In that first version of this post I ignored patches, minor updates to the bash source code and binary. But as the patches to bash 5 are accumulating, I cannot ignore them much longer.

This post will extend the instructions in the original post.

After downloading and expanding the bash-5.0.tar.gz, create a patches folder:

$ cd path/to/bash-5.0
$ mkdir patches
$ cd patches

You can download the patches for bash-5.0 here. As of this writing, there are seven patches for bash-5.0 labelled bash50-001 through bash50-007. You can download all at once with:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' -O

(Adapt the numbers when there are more patches in the future.)

Then move up one directory level to the bash-5.0 root directory and apply the patches using the patch command.

$ cd ..
$ patch -p0 -i patches/bash50-001
$ patch -p0 -i patches/bash50-002

(etc.)

You can download and patch with a single step. Make sure your working directory is the bash-5.0 with all the code and run:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' | patch -p0

From here, you can continue with the remaining build steps from the original post. The next step will be running ./configure.

The script to build the pkg installer has also been updated in the repository to download and apply the patches before building.

Moving to zsh, part 4: Aliases and Functions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

Aliases

Aliases in zsh work just like aliases in bash. You declare an alias with the alias (built-in) command and it will work as a text replacement at the beginning of the command prompt:

alias ll='ls -al'

You can just copy your alias declarations from your .bash_profile or .bashrc to your .zshrc. I had aliases for .. and cd.. which are now handled by Auto CD and shell correction respectively, so I didn’t bother to move those. (part 3: ‘Shell Options’)

After the alias is declared, you can use it at the beginning of a command. When you try to use the alias anywhere else in the command, the alias will not work:

% sudo ll
sudo: ll: command not found

Global Aliases

This is where zsh has an advantage. You can declare an alias as a ‘global’ alias, and then will be replaced anywhere in the command line:

% alias -g badge='tput bel'
% sudo badge        #<beeps> with privilege

Identifying Aliases

There is one more feature of zsh that is useful with aliases. The which command will show if a command stems from an alias substitution:

% which ll
ll: aliased to ls -l

However, when you try this with global aliases, the substitution occurs before the which command can evaluate the alias, which leads to an unexpected result:

% which badge
/usr/bin/tput
bel not found

You can suppress the alias substitution by escaping the first character or by quoting the entire alias name:

% which \badge
badge: globally aliased to tput bel
% which 'badge'
badge: globally aliased to tput bel

Functions

As with aliases, functions in your zsh configuration will work just as they did in bash.

function vnc() {
    open vnc://"$USER"@"$1"
}

This code in your zsh configuration file will define the vnc function and make it available in the shell.

Autoload Functions

However, zsh has some features which make using functions more flexible. There is (once again) a bit of configuration required to get this working.

Instead of declaring the function directly the configuration file, you can put the function in a separate file. zsh has a built-in variable called fpath which is an array of paths where zsh will look for files defining a function. You can add your own directory to this search path:

fpath+=~/Projects/dotfiles/zshfunctions

Just having a file in the directory is not enough. You still have to tell zsh that you want to use this particular function:

autoload vnc

This command tells zsh: “’Declare a function named vnc. To execute it, load a file named vnc, it is somewhere in the fpath.”

Note: you often see the -U or -Uz option added to the autoload command. These options help avoid conflicts with your personal settings. They suppress alias substitution and ksh-style loading of functions, respectively.

The vnc file in my zshfunctions directory can look like this:

# uses the arguments as hostnames for `open vnc://` (Screen Sharing)
# uses the $USER username as default account name

for x in $@; do 
    open vnc://"$USER"@"$x"
done

The vnc function will open a Screen Sharing session with the current user name pre-filled in.

Initializing Autoload Functions

You could also put the code in the function file into a function block:

function vnc {
    for x in $@; do 
        open vnc://"$vnc_user"@"$x"
    done
}

# initialization code
vnc_user="remote_admin"
alias screen_sharing='vnc'

The function name should match the function name declared with autoload.

When you have additional code outside the function, the autoload behavior changes. When the function is called for the first time, the function will be defined and the code outside the function will be run. The function itself will not be executed on the first run. On subsequent calls, the function will be executed and the code outside the function is ignored.

You can use this to provide setup and initialization code for the function. You can even have more functions defined in the function file. The above example declares and sets a variable to use for account name and an alias for the vnc command.

Since you have to run the function once for the initialization, you often see this syntax in the zsh configuration file:

autoload vnc && vnc

Which means ‘declare the function and if that succeeds run it.’

In some functions, the initialization code will already launch the function itself:

function vnc {
    ...
}
# initialization
vnc_user="remote_admin"
vnc()

Since the behavior will vary from each autoloaded function to the next, be sure to study any documentation or the function’s code.

Identifying Functions

Finally, the which command will show the function code:

 % which vnc
vnc () {
    for x in $@
    do
        open vnc://"$USER"@"$x"
    done
}

The functions command without any parameters, will print all functions (there will be a lot of them). Use functions + to just list the function names.

Debugging Functions

When you are working on complex autoloaded functions, you will at some point have to do some debugging. You can enable tracing for functions with

% functions -t vnc
% vnc Client.local
+vnc:1> x=Client.local
+vnc:2> open vnc://armin@Client.local

You can disable tracing for this function with functions +t vnc.

Next

In the next part we will enable, use and configure tab completions.

Moving to zsh, part 3: Shell Options

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

Now that we have chosen a file to configure our zsh, we need to decide on ‘what’ to configure and ‘how.’ In this post, I want to talk about zsh’s shell options.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

In the previous post, I listed some features that I would like to transfer from my bash configuration. While researching how to implement these options in zsh, I found a few, new and interesting options in zsh.

The settings from bash which I want in zsh were:

  • case-insensitive globbing
  • command history, shared across windows and sessions

Note: bash in this series of posts specifically refers to the version of bash that comes with macOS as /bin/bash (v3.2.57).

Note 2: Mono-typed lines starting with a % show commands and results from zsh. Mono-typed lines starting with $ show commands and results in bash

What are Shell Options?

Shell options are preferences for the shell’s behavior. You are using shell options in bash, when you enable ‘trace mode’ for scripts with the set -x command or the bash -x option. (Note: this also works with zsh scripts.)

zsh has a lot of shell options. Many of these options serve the purpose of enabling (or disabling) compatibility with other shells. There are also many options which are specific to zsh.

You can set an option with the setopt command. For compatibility with other shells the setopt command and set -o have the same effect (set an option by name). The following commands set the same option:

set -o AUTO_CD
setopt AUTO_CD

The names or labels of the options are commonly written in all capitals in the documentation but in lowercase when listed with the setopt tool. The labels of the options are case insensitive and any underscores in the label are ignored. So, these commands set the same option:

setopt AUTO_CD
setopt autocd
setopt auto_cd
setopt autoCD

There are quite a few ways to negate or unset an option. First you can use unsetopt or set +o. Alternatively, you can prefix with NO or no to negate an option. The following commands all have the same effect of turning off the previously set option AUTO_CD

unsetopt AUTO_CD
set +o AUTO_CD
unsetopt autocd
setopt NO_AUTO_CD
setopt noautocd

Any options you change will only take effect in the current instance of zsh. When you want to change the settings for all new shells, you have to put the commands in one of the configuration files (usually .zshrc).

Showing the current Options

You can list the existing shell options with the setopt command:

% setopt
combiningchars
interactive
login
monitor
shinstdin
zle

This list only shows options are changed from the default set of options for zsh. These options are marked with <D> (default for all shell emulations) or <Z> (default for zsh) in the documentation or the zshoptions man page.

You can also get a list of all default zsh options with the command:

% emulate -lLR zsh

Some zsh Options I use

As I have mentioned before in my posts on bash configuration, I prefer minimal configuration changes, so I do not feel all awkward and lost when I have to work on an ‘un-configured’ Mac.

These configurations are a personal choice and you should pick and choose your own. You can find a full list of zsh options in the zsh Manual or with man zshoptions.

On the other hand, exploring the options allows us to explore a few useful zsh features.

Case Insensitive Globbing

Note: ‘Globbing’ is a unix/shell term that refers to the expansion of wildcard characters, such as * and ? into full file paths and names. I.e. ~/D* is expanded into /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Since the file system on macOS is (usually) case-insensitive, I prefer globbing and tab-completion to be case-insensitive as well.

The zsh option which controls this is CASE_GLOB. Since we want globbing to be case-insensitive, we want to turn the option off, so:

setopt NO_CASE_GLOB

You can test this in the shell:

% ls ~/d*<tab>

In zsh tab completion will replace the wildcard with the actual result. So after the tab you will see:

% ls /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Using tab completion this way to see and possibly edit the actual replacement for wildcards is a useful safety net.

In bash hit the tab key will list possible completions, but not substitute them in the command prompt.

If you do not like this behavior in zsh then you can change to behavior similar to bash with:

setopt GLOB_COMPLETE

Automatic CD

Sometimes you enter the path to a directory, but forget the leading cd:

$ Library/Preferences/
bash: Library/Preferences/: is a directory

% Library/Preferences
zsh: permission denied: Library/Preferences

With AUTO_CD enabled in zsh, the shell will automatically change directory:

% Library/Preferences
% pwd
/Users/armin/Library/Preferences

This works with relative and absolute paths, including the ..:

% ..
% pwd
/Users/armin/Library
% ../Desktop 
% pwd
/Users/armin/Desktop

I have an alias in my .bash_profile that sets the .. command to cd ... Auto CD replaces that functionality and more.

Enable Auto CD with:

setopt AUTO_CD

Shell History

Shells commonly remember previously executed commands and allows you to recall them with the up and down arrow keys, search or special history commands.

Most of those keys work the same in zsh. However, there are a few things you need to configure for zsh history to work as you are used to with bash on macOS.

By default, zsh does not save its history when the shell exits. The history is ‘forgotten’ when you close a Terminal window or tab. To make zsh save its history to a file when it exits, you need to set a variable in the shell:

HISTFILE=${ZDOTDIR:-$HOME}/.zsh_history

Note: this is not a shell option but shell variable or parameter. I will cover some more of those later, You can find a list of variables used by zsh in the documentation.

The HISTFILE variable tells zsh where to store the history data. The syntax ${ZDOTDIR:-$HOME} means it will use the value of ZDOTDIR when it is set or default to the value of HOME otherwise. When a user has set the ZDOTDIR variable to group their configurations files in a specific directory, the history will be stored there as well.

By default zsh simply writes each command in its own line in the history file. You can view the file’s contents with any text editor or list the last few commands:

% tail -n 10 ~/.zsh_history

You can make zsh add a bit more data (timestamp in unix epoch time and elapsed time of the command) by setting the EXTENDED_HISTORY shell option.

setopt EXTENDED_HISTORY

You can set limits on how many commands the shell should remember in the session and in the history file with the HISTSIZE and SAVEHIST variables:

SAVEHIST=5000
HISTSIZE=2000

When the shell reaches this limit the oldest commands will be removed from memory or the history file.

By default, when you exit zsh (for example, by closing the window or tab) this particular instance of zsh will overwrite an existing history file with its history. So when you have multiple Terminal windows or tabs open, they will all overwrite each others’ histories eventually.

You can tell zsh to use a single, shared history file across the sessions and append to it rather than overwrite:

# share history across multiple zsh sessions
setopt SHARE_HISTORY
# append to history
setopt APPEND_HISTORY

Furthermore, you can tell zsh to update the history file after every command, rather than waiting for the shell to exit:

# adds commands as they are typed, not at shell exit
setopt INC_APPEND_HISTORY

When you use a shared history file, it will grow very quickly, and you may want to use some options to clean out duplicates and blanks:

# expire duplicates first
setopt HIST_EXPIRE_DUPS_FIRST 
# do not store duplications
setopt HIST_IGNORE_DUPS
#ignore duplicates when searching
setopt HIST_FIND_NO_DUPS
# removes blank lines from history
setopt HIST_REDUCE_BLANKS

(some of these are redundant)

Most of the time you will access the history with the up arrow key to recall the last command, or maybe a few more steps. You can search through the history with ctrl-R

In zsh, you can also use the !! history substitution, which will be replaced with the entire last command. This is most commonly used in combination with sudo:

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

By default, the shell will show the command it is substituting before it is run. But at that point, it is too late to make any changes. When you set the HIST_VERIFY option, zsh will show the substituted command in the prompt instead, giving you a chance to edit or cancel it, or just confirm it.

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
% sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

This works for other history substitutions such as !$ or !*, as well. You can find all of zsh’s history expansions in the documentation.

Correction

When you mistype a command or path, the shell is usually unforgiving. In zsh you can enable correction. Then, the shell will make a guess of what you meant to type and ask whether you want do that instead:

% systemprofiler 
zsh: correct 'systemprofiler' to 'system_profiler' [nyae]?

Your options are to

  • n: execute as typed
  • y: accept and execute the suggested correction
  • a: abort and do nothing
  • e: return to the prompt to continue editing

I have found this far less annoying and far more useful than I expected. Especially, since it works together with AUTO_CD:

% Dekstop
zsh: correct 'Dekstop' to 'Desktop' [nyae]?

You enable zsh correction with these options:

setopt CORRECT
setopt CORRECT_ALL

Reverting to defaults

Most of the changes mentioned here affect the interactive shell and will have little impact on zsh scripts. However, there are some options that do affect the behavior of things like variable substitutions which will affect scripts.

You can revert the options for the current shell to the default settings with the following command:

emulate -LR zsh

We encountered this command earlier when we listed the default settings. The -l option will list the settings rather than apply them.

If in doubt, it may be useful to add this at the beginning of your zsh scripts.

Next

In the next part we will take a look at aliases and functions.

Moving to zsh, part 2: Configuration Files

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

In part one I talked about Apple’s motivation to switch the default shell and urge existing users to change to zsh.

Since I am new to zsh as well, I am planning to document my process of transferring my personal bash setup and learning the odds and ends of zsh.

Many websites and tutorials leap straight to projects like oh-my-zsh or prezto where you can choose from hundreds of pre-customized and pre-configured themes.

While these projects are very impressive and certainly show off the flexibility and power of zsh customization, I feel this will actually prevent an understanding of how zsh works and how it differs from bash. So, I am planning to build my own configuration ‘by hand’ first.

At first, I actually took a look at my current bash_profile and cleaned it up. There were many aliases and functions which I do not use or broke in some macOS update. I the end, this is what I want to re-create in zsh:

Most of these should be fairly easy to transfer. Some might be… interesting.

But first, where do we put our custom zsh configuration?

zsh Configuration Files

bash has a list of possible files that it tries in predefined order. I have the description in my post on the bash_profile.

zsh also has a list of files it will execute at shell startup. The list of possible files is even longer, but somewhat more ordered.

all users user login shell interactive shell scripts Terminal.app
/etc/zshenv .zshenv
/etc/zprofile .zprofile x x
/etc/zshrc .zshrc x x
/etc/zlogin .zlogin x x
/etc/zlogout .zlogout x x

The files in /etc/ will be launched (when present) for all users. The .z* files only for the individual user.

By default, zsh will look in the root of the home directory for the user .z* files, but this behavior can be changed by setting the ZDOTDIR environment variable to another directory (e.g. ~/.zsh/) where you can then group all user zsh configuration in one place.

On macOS you could set the ZDOTDIR to ~/Documents/zsh/ and then use iCloud syncing (or a different file sync service) to have the same files on all your Macs. (I prefer to use git.)

bash will either use .bash_profile for login shells, or .bashrc for interactive shells. That means, when you want to centralize configuration for all use cases, you need to source your .bashrc from .bash_profile or vice versa.

zsh behaves differently. zsh will run all of these files in the appropriate context (login shell, interactive shell) when they exist.

zsh will start with /etc/zshenv, then the user’s .zshenv. The zshenv files are always used when they exist, even for scripts with the #!/bin/zsh shebang. Since changes applied in the zshenv will affect zsh behavior in all contexts, you should you should be very cautious about changes applied here.

Next, when the shell is a login shell, zsh will run /etc/zprofile and .zprofile. Then for interactive shells /etc/zshrc and .zshrc. Then, again, for login shells /etc/zlogin and .zlogin. Why are there two files for login shells? The zprofile exists as an analog for bash’s and sh’s profile files, and zlogin as an analog for ksh login files.

Finally, there are zlogout files that can be used for cleanup, when a login shell exits. In this case, the user level .zlogout is read first, then the central /etc/zlogout. If the shell is terminated by an external process, these files might not be run.

Apple Provided Configuration Files

macOS Mojave (and earlier versions) includes /etc/zprofile and /etc/zshrc files. Both are very basic.

/etc/zprofile uses /usr/libexec/path_helper to set the default PATH. Then /etc/zshrc enables UTF–8 with setopt combiningchars.

Like /etc/bashrc there is a line in /etc/zshrc that would load /etc/zshrc_Apple_Terminal if it existed. This is interesting as /etc/bashrc_Apple_Terminal contains quite a lot of code to help bash to communicate with the Terminal application. In particular bash will send a signal to the Terminal on every new prompt to update the path and icon displayed in the Terminal window title bar, and provides other code relevant for saving and restoring Terminal sessions between application restarts.

However, there is no /etc/zshrc_Apple_Terminal and we will have to provide some of this functionality ourselves.

Note: As of this writing, /etc/zshrc in the macOS Catalina beta is different from the Mojave /etc/zshrc and provides more configuration. However, since Catalina is still beta, I will focus these articles on Mojave and earlier. Once Catalina is released, I may update these articles or write a new one for Catalina, if necessary.

Which File to use?

When you want to use the ZDOTDIR variable to change the location of the other zsh configuration files, setting that variable in ~/.zshenv seems like a good choice. Other than that, you probably want to avoid using the zshenv files, since it will change settings for all invocations of zsh, including scripts.

macOS Terminal considers every new shell to be a login shell and an interactive shell. So, in Terminal a new zsh will potentially run all configuration files.

For simplicity’s sake, you should use just one file. The common choice is .zshrc.

Most tools you can download to configure zsh, such as ‘prezto’ or ‘oh-my-zsh’, will override or re-configure your .zshrc. You could consider moving your code to .zlogin instead. Since .zlogin is sourced after .zshrc it can override settings from .zshrc. However, .zlogin is only called for login shells.

The most common situation where you do not get a login shell with macOS Terminal, is when you switch to zsh from another shell by typing the zsh command.

I would recommend to put your configuration in your .zshrc file and if you want to use any of the theme projects, read and follow their instructions closely as to how you can preserve your configurations together with theirs.

Managing the shell for Administrators

MacAdmins may have the need to manage certain shell settings for their users, usually environment variables to configure certain command line tool’s behaviors.

The most common need is to expand the PATH environment variable for third party tools. Often the third party tools in question will have elaborate postinstall scripts that attempt to modify the current user’s .bash_profile or .bashrc. Sometimes, these tools even consider that a user might have changed the default shell to something other than bash.

On macOS, system wide changes to the PATH should be done by adding files to /etc/paths.d.

As an administrator you should be on the lookout for scripts and installers that attempt to modify configuration files on the user level, disable the scripts during deployment, and manage the required changes centrally. This will allow you to keep control of the settings even as tools change, are added or removed from the system, while preserving the user’s custom configurations.

To manage environment variables other than PATH centrally, administrators should consider /etc/zshenv or adding to the existing /etc/zshrc. In these cases you should always monitor whether updates to macOS overwrite or change these files with new, modified files of their own.

Summary

There are many possible files where the zsh can load user configuration. You should use ~/.zshrc for your personal configurations.

There are many tools and projects out there that will configure zsh for you. This is fine, but might keep you from really understanding how things work.

MacAdmins who need to manage these settings centrally, should use /etc/paths.d and similar technologies or consider /etc/zshenv or /etc/zshrc.

Apple’s built-in support for zsh in Terminal is not as detailed as it is for bash.

Next: Part 3 – Shell Options

Moving to zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

zsh (I believe it is pronounced zee-shell, though zish is fun to say) will succeed bash as the default shell. bash has been the default shell since Mac OS X 10.3 Panther.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

Why?

The bash binary bundled with macOS has been stuck on version 3.2 for a long time now. bash v4 was released in 2009 and bash v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3. bash v3 is still GPL v2.

zsh, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default. zsh has been available as on macOS for a long time. The zsh version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current zsh 5.7.1.

Is bash gone!?

No.

macOS Catalina still has the same /bin/bash (version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.

Many scripts in macOS, management systems, and Apple and third party installers rely on /bin/bash. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.

Users that have /bin/bash as their default shell on Catalina will see a prompt at the start of each Terminal session stating that zsh is now the recommended default shell. If you want to continue using /bin/bash, you can supress this message by setting an environment variable in your .bash_profile or .bashrc.

export BASH_SILENCE_DEPRECATION_WARNING=1

You can also download and install a newer version of bash yourself. Keep in mind that custom bash installations reside in a different directory, usually /usr/local/bin/bash.

Will bash remain indefinitely?

Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to bash, but didn’t really care if you remained on tcsh. In fact, tcsh is still present on macOS.

Apple’s messaging should tell us, that the days of /bin/bash are numbered. Probably not very soon, but eventually keeping a more than ten year old version of bash on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.

Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’ dash has been added to the lineup of shells. dash is designed to be a minimal implementation of the Posix standard shell sh. So far, in macOS (including Catalina),sh invokes bash in sh-compatibility mode.

As Apple’s support article mentions, Catalina also adds a new mechanism for users and admins to change which shell handles sh invocations. MacAdmins or users can change the symbolic link stored in /var/select/sh to point to a shell other than /bin/bash. This changes which shell interprets scripts the #!/bin/sh shebang or scripts invoked with sh -c. Changing the interpreter for sh should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.

All of these changes are indicators that Apple is preparing to remove /bin/bash at some, yet indeterminate, time in the future.

Do I need to wait for Catalina to switch to zsh?

No, zsh is available Mojave and on older macOS versions. You can start testing zsh or even switch your default shell already.

If you want to just see how zsh works, you can just open Terminal and type zsh:

$ zsh
MacBook%

The main change you will see is that the prompt looks different. zsh uses the % character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in bash.

If you want to already switch your default shell to zsh you can use the chsh command:

$ chsh -s /bin/zsh

This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.

How is zsh different?

Like bash (‘Bourne again shell’ ), zshderives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.

The main difference between bash and zsh is configuration. Since zsh ignores the bash configuration files (.bash_profile or .bashrc) you cannot simply copy customized bash settings over to zsh. zsh has much more options and points to change zsh configuration and behavior. There is an entire eco-system of configuration tools and themes called oh-my-zsh which is very popular.

zsh also offers better configuration for auto-completion which is far easier than in bash.

I am planning a separate post, describing how to transfer (and translate) your configurations from bash to zsh.

What about scripting?

Since zsh has been present on macOS for a long time, you could start moving your scripts from bash to zsh right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to #!/bin/zsh.

You will gain some features where zsh is superior to bash v3, such as arrays and associative arrays (dictionaries).

There is one exception where I would now recommend to use /bin/sh for your scripts: the Recovery system does not contain the /bin/zsh shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider older macOS installations where zsh is definitely not present in Recovery.

When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as Twocanoes MDS, installr or bootstrappr, then you cannot rely on /bin/zsh.

Since we now know that bash is eventually going away, the only common choice left is /bin/sh.

When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using /bin/sh as the shebang from now on.

I used to recommend using /bin/bash for everything MacAdmin related. /bin/sh is definitely a step down in functionality, but it seems like the safest choice for continued support.

Summary

Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’ bash with zsh, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!

Next

In the next part we will look at the configuration files for zsh.

Show Exit Code in your bash Prompt

I prefer a minimal bash prompt. Recently, however, I saw an oh-my-zsh prompt, that I thought would be useful.

The intriguing prompt displayed a symbol indicating whether the previous command exited successful (exit code zero) or failed (non-zero exit code).

You can always get the exit code of the previous command from the $? variable, but seeing it right there and in color, is more direct.

While I find fish andzsh quite intriguing, I am still unwilling to move my setup for just a single feature. “This has to be possible in bash,” I thought… And it is, though the implementation was a bit more complex than I expected. But I learned a lot more about how the bash prompt worked.

The trick for changeable or dynamic prompts in bash is to create a bash function that assembles the PS1 variable, on every prompt. You can enable that function by setting the PROMPT_COMMAND environment variable to your custom function.

Obviously, you should not overload your function with time intensive processes, but with modern processing power, a lot can be done in a short time.

After a lot of experimentation, I settled on this setup:

Update: there was an error in the code that would prevent the prompt from ever showing a red exit code. I fixed it now, the change is in the last line. (Thanks to co-worker Mattias for pointing that out.)

You can add this code to your .bash_profile or .bashrc. (If you do not know what that means, read this post.)

I experimented with special characters and even Emoji to signify the exit code, but then settled on colors and the square root symbol (option-V on the US and international keyboard, looks like a checkmark) for success and the question mark ? with the exit code for errors.

Obviously, you can use a modified prompt command to show all kinds of other statuses as well. Enjoy!

On Smart Quotes and Terminal

Typography is a wonderful art and has a long history. When humans turned from manual typesetting to machines, type writers and then computers, some compromises had to be made. One of these compromises was to use simple straight quote symbols for opening and closing the quote, rather than different quote symbols for opening and closing.

Note: which kind of quotes are used for opening and closing dependent on the language or and some convention. English uses upper quotes “…”, German opens with a lower quote: „…“, French uses ‘guillemets:’ «…», and Japanese uses hooks: 「…」

Quotation Marks have funny names in many languages. Germans call them “Gänsefüßchen,” or “little goose feet.”

See this Wikipedia article for more details.

macOS, iOS and other modern operating systems have a feature which replaces the simple or straight quote symbols with the typographic quotes. So, you type "Hello!" and the quotes are automatically replaced with the proper (depending on localization) typographic quotes. This is called “smart quotes.”

This is pretty nice, but can be troublesome when dealing with Terminal and text editors. Scripting languages and shells always use straight quotes, and cannot deal with typographic quotes.

Now, if someone sends you a command or a script that uses quotes, and it goes through an app that replaces them with smart quotes, then bash and Terminal will fail miserably.

There is not much you can do, other than be aware of this and check pasted code carefully. There is something you can do to make this easier, though.

The default monospace font used in Terminal on macOS are ‘Menlo’ or ‘SF Mono,’ depending on the macOS version. Now these are fine typefaces, but their typographic quotes are not very curly at all, making them very hard to distinguish from the ‘dumb’ straight quotes that Terminal expects. The classic ‘Monaco’ typeface on the other hand has beautiful curly typographic quotes, making them very distinct from the the straight quote.

My favorite mono space typeface ‘Source Code Pro’ also has nice curly typographic quotes. I have built this table with many common monospace typefaces and their quotes.

Quotation Mark Comparison
Quotation Mark Comparison

Now this shouldn’t be your only criteria in choosing your Terminal font, but it may be something that helps avoid quote errors.

Swift 5 for MacAdmins

macOS 10.14.4 also includes Swift 5. The main new feature of Swift 5 is that Swift is ABI stable.

Simply said, ABI stability allows swift binaries to use a Swift library on the system instead of having to bundle the libraries with them. This will, of course reduce the size of the binaries.

For example, my desktoppr tool compiles to 6.5MB with Swift 4.2 and 56KB with Swift 5.

However, command line tools built with Xcode 10.2 now rely on the Swift library to be available on the system. macOS 10.14.4 and future versions will include the libraries, but older macOS versions did not. There is no option in Xcode 10.2 to keep the old behavior of bundling the libraries in the tool.

This means that when you re-build a tool in Xcode 10.2 with Swift it will not run on older macOS version:

$ sw_vers -productVersion
10.13.6
$ ./desktoppr 
dyld: Library not loaded: @rpath/libswiftAppKit.dylib
  Referenced from: /Users/armin/Desktop/desktoppr
  Reason: image not found
Abort trap: 6

Apple provides an installer for the Swift libraries for “earlier versions of macOS.” (The package installer declares a minimum OS version of 10.9.)

The libraries are installed in /usr/lib/swift, where your binaries can find them:

$ ./desktoppr 
/Library/Desktop Pictures/BoringBlueDesktop.png

Note: these libraries are used by compiled binaries. Installing the runtime libriaries will not allow you to run swift script files, i.e. text files starting with the #!/usr/bin/swift shebang. You still need to install Xcode or the Developer Command Line tools for that.

So far, only command line tools written and compiled in Xcode 10.2 will require the libraries. Application bundles will continue to include their own libraries.

As Swift gets updated, you will need to update the installed libraries as well. You can get the installed version of the Runtime with pkgutil:

$ pkgutil --info com.apple.pkg.SwiftRuntimeForCommandLineTools
package-id: com.apple.pkg.SwiftRuntimeForCommandLineTools
version: 10.2.0.0.1.1552586384
volume: /
location: /
install-time: 1553789052
groups: com.apple.FindSystemFiles.pkg-group 

However, on a Mac with 10.14.4 the swift libraries are present but not installed by the same installer package, they are part of the entire system:

$ pkgutil --file-info /usr/lib/swift/libswiftFoundation.dylib
volume: /
path: /usr/lib/swift/libswiftFoundation.dylib

pkgid: com.apple.pkg.update.os.10.14.4.18E226
pkg-version: 1.0.0.0.1.1553145155
install-time: 1553765876
uid: 0
gid: 0
mode: 755

If/When Apple updates the runtime libraries, this might be a challenge to track and update properly.

Note: While I used my tool desktoppr as an example, I have not updated the version available for download to Swift 5 yet. So, that is still built with the “old” Swift and Xcode and should work everywhere without the runtime libraries.

Nevertheless, it should be prudent for MacAdmins to install the Swift 5 runtime libraries on their fleet. At the very least be aware that these errors can occur and how to fix them.

Parsing dscl Output in Scripts

On macOS dscl is a very useful to access data in the local user directory or another directory the Mac is bound to. For example you can read a user’s UID with:

$ dscl /Search read /Users/armin UniqueID
UniqueID: 501

This output looks easy enough to parse, you can just use cut or awk:

$ dscl /Search read /Users/armin UniqueID | cut -d ' ' -f 2
501
$ dscl /Search read /Users/armin UniqueID | awk '{print $2;}'
501

However, dscl is a treacherous. Its output format changes, depending on the contents of an attribute. When an attribute value contains whitespace, the format of the output has two lines:

$ dscl /Search read /Users/armin RealName
RealName:
 Armin Briegel

With attributes like the UID, it is fairly safe safe to assume that there will be no whitespace in the value. With other attributes, such as RealName or NFSHomeDirectory, you cannot make that prediction with certainty. Real names may or may not have been entered with a space. A user (or management script) may have changed their home directory to something starting with /Volumes/User HD/... and your script may fail.

To remove this output ambiguity, dscl has a -plist option which will print the output as a property list:

 $ dscl -plist . read /Users/armin RealName
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>dsAttrTypeStandard:RealName</key>
    <array>
        <string>Armin Briegel</string>
    </array>
</dict>
</plist>

The resulting property list is a dict containing a key with the native attribute name and an array containing the values, even when there is only one value.

Having a property list is nice, but parsing property lists in a shell script is challenging. I have found two solutions

Xpath

You can use the xpath tool extract data from the XML output:

$ dscl -plist . read /Users/armin RealName | xpath "//string[1]/text()" 2>/dev/null
Armin Briegel

Note that the xpath output does not include a final new line character, which makes it look a bit strange.

The xpath argument in detail means:

  • //string[1]: the first of any string element
  • /text() the text contents of that stringobject

This syntax makes a lot of assumptions about the property list input. I believe they are safe with the dscl output. (Please test)

If you want to play around with xpath syntax, I recommend using an interactive tool. I used this one from Code Beautify which worked well enough, but frankly I just randomly chose one from the list of search results for ‘xpath tester’. (If you can recommend a great one, let us know in the comments.)

PlistBuddy

As I said, the xpath solution makes a lot of assumptions about the layout of the property list. A safer way of parsing property lists would be a dedicated tool, such as PlistBuddy. However, PlistBuddy does not read from stdin. At least not voluntarily.

A few weeks ago Erik Berglund shared this trick on Mac Admins Slack which makes PlistBuddy read the output from another command. We can adapt this for our use case:

$ /usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/armin RealName)
Armin Briegel

Note that you have to escape the : in the attribute name, since PlistBuddy uses the colon as a path separator.

You can use this in scripts to assign the value to a variable with

realName=$(/usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/$username RealName))

This uses nested command substitution with the $(... $(...) ...) syntax which is not possible using backticks.

Either way, you can get a safe value from dscl in shell script, whether it contains whitespace or not.

Demystifying `root` on macOS, Part 4 —The Authorization Database

Beyond the Shell

sudo allows you to gain super user privileges from the interactive shell. launchd, installer Packages, management systems and other tools will run scripts as root when required. This covers many cases where administrators need to influence the system on macOS.

However, for most users, macOS is exclusively the graphical interface. Users can authorize to perform certain tasks when they are administrators, like unlocking a Preference Pane or running an installer package.

These tasks are not controlled by sudo but by a separate mechanism. The data for that mechanism is stored in the authorization database.

Here there be Dragons!

As mentioned in earlier parts of this series, many pieces of macOS assume users have administrator accounts. This might not be a possible or useful configuration in your environment. But you still need to provide access to some privileges without giving a user full administrator accounts and access.

However, if you find yourself frequently editing the authorization database, you should re-evaluate your approach. Future macOS updates might change privileges or rules and then your heaviliy modified setup will need to be adpated. Small modifications or granting users admin privileges will be the the least fragile configuration going forward with future macOS releases.

What is it?

The active authorization database lives in sqlite3 database files in /var/db/auth.db.

You can use the sqlite3 command or another tool to read the database directly.

$ sudo sqlite3 /var/db/auth.db .dump

You can use this database access to change the settings directly. However, this is not recommended. The only way sanctioned by Apple to access and change the authorization database is through the security authorization command.

This layer of abstraction allows Apple to change the underlying data store, while keeping tools and frameworks that access the data same.

The authorization database is intialized from the property list file /System/Library/Security/authorization.plist. In older versions of macOS administrators could replace this file with a modified version and delete the database files to have the modified property list initialize the database with the new settings. However, in current versions of macOS this file is protected by SIP, so this strategy is no longer useful.

However, the authorization.plist file is still useful to look at the default values and get an idea of how the authorization database is configured and works. Since there is quite a lot of data in this file, it is best to open it in a graphical property list editor such as Xcode or PlistEdit Pro.

The authorization property list consists of two main dictionaries: rights and rules. There are also comment fields distributed all through the file, which provide some context to what the individual elements are for.

rights designate a certain context that permits a certain action, group of actions or access to configure some part of the os. The names of the rights follow a hierarchy and are denoted in reverse DNS notation.

This website has an overview of all the rights and rules from the authorization.plist. More importantly it shows which rights and rules are available in which version of macOS.

How it works

Within the dictionary for a given right you can find the requirements that a user needs to fulfill to gain the right. There are two main classes: user and rule.

(There is also a third possible value for the class: evaluate-mechanisms will test multiple mechanisms in order. This is used for more complex processes such as the login window.)

The user class will verify whether the user asking to gain the right is in a particular user group. Usually this is the admin group. for example you can inspect the configuration of the system.preferences.datetime right with:

$ security authorizationdb read system.preferences.datetime
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>allow-root</key>
    <true/>
    <key>authenticate-user</key>
    <true/>
    <key>class</key>
    <string>user</string>
    <key>comment</key>
    <string>Checked by the Admin framework when making changes to the Date &amp; Time preference pane.</string>
    <key>created</key>
    <real>522083506.22018802</real>
    <key>group</key>
    <string>admin</string>
    <key>modified</key>
    <real>522083506.22018802</real>
    <key>session-owner</key>
    <false/>
    <key>shared</key>
    <false/>
    <key>timeout</key>
    <integer>2147483647</integer>
    <key>tries</key>
    <integer>10000</integer>
    <key>version</key>
    <integer>1</integer>
</dict>
</plist>
YES (0)

(The YES (0) at the end of the command’s output means that retrieving the data was successful.)

You can also look for the system.preferences.datetime entry in the authorization.plist. This tells us that when a user clicks on the lock in the ‘Date & Time’ preference pane the system will ask for authentication (authenticate-user is true) to check if the user a member of the admin group.

Read the com.apple.configurationprofiles.userprofile.trustcert right to see an example for a rule based right:

$ security authorizationdb read com.apple.configurationprofiles.userprofile.trustcert
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>class</key>
    <string>rule</string>
    <key>comment</key>
    <string>Install user configuration profile with certificate requiring trust change.</string>
    <key>created</key>
    <real>529239529.937994</real>
    <key>modified</key>
    <real>529239529.937994</real>
    <key>rule</key>
    <array>
        <string>authenticate-session-owner-or-admin</string>
    </array>
    <key>version</key>
    <integer>0</integer>
</dict>
</plist>
YES (0)

Rather than defining the approving criteria in the right itself, this right references the authenticate-session-owner-or-admin rule. This name is already quite self-explanatory. However, we can also read the definition of the rule:

$ security authorizationdb read authenticate-session-owner-or-admin
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>allow-root</key>
    <false/>
    <key>authenticate-user</key>
    <true/>
    <key>class</key>
    <string>user</string>
    <key>comment</key>
    <string>Authenticate either as the owner or as an administrator.</string>
    <key>created</key>
    <real>522083506.22018802</real>
    <key>group</key>
    <string>admin</string>
    <key>modified</key>
    <real>522083506.22018802</real>
    <key>session-owner</key>
    <true/>
    <key>shared</key>
    <false/>
    <key>timeout</key>
    <integer>2147483647</integer>
    <key>tries</key>
    <integer>10000</integer>
    <key>version</key>
    <integer>0</integer>
</dict>
</plist>
YES (0)

Requests to grant a right are logged. When you are unsure what the name of right is you can perform/unlock the right or task in the interface and then read the log with:

$ sudo log show --style syslog --predicate 'subsystem == "com.apple.Authorization" && eventMessage CONTAINS[c] "validating credential"' --last 1h

This will list the rights (and rules) that were requested in the last hour.

(Thanks to Erik Berglund on the MacAdmins Slack for showing me how to do this.)

Changing Behavior

You can use the security authorizationdb command to change rights and rules in the authorization database.

WARNING: you can really mess up your system and make it unusable with the wrong settings in the authorization database. You should not test these commands on your (or anybody else’s) work machine. You should do your experimentation and testing on a virtual machine, where you can quickly revert the system to a known working state. You should only use these commands on production Macs when they have been thoroughly tested.

You need super user privileges to change the authorization db. I will use sudo for the interactive examples.

The easiest way to change behavior is to change the right to a different preset rule. The most permissive rule is allow which allow any user the right and not even prompt for authorization.

When you change the ‘Date & Time’ right with:

$ sudo security authorizationdb write system.preferences.datetime allow
Password:
YES (0)

And then open the ‘Date & Time’ preference pane with any user. You will see the pane is unlocked by default.

This may be just a bit too permissive. A more useful rule to use is the authenticate-session-owner-or-admin rule, which will prompt for authentication, and accept the currently logged in user (session-owner) or any admin. This provides for some security so that other people cannot walk up and change the current user’s settings.

$ sudo security authorizationdb write system.preferences.datetime authenticate-session-owner-or-admin
YES (0)

This will allow the any user to unlock the preference pane in their own session. Note that the dialog to unlock does not prefill the user’s name and the text in the dialog still asks for an ‘administrator’s name and password’. So this may be confusing for users. Proper documentation for the user’s affected will help.

Another use case for this rule is to allow administrator account to unlock a locked screen in another user’s session (particularly useful in lab or classroom settings).

$ sudo security authorizationdb write system.login.screensaver authenticate-session-owner-or-admin
Password:
YES (0)

Now, any admin user can unlock any other user’s locked screen. This will of course drop them in that user’s login session so the admin’s have to be responsible with this privilege.

When you take a look through the rules, you will notice that some start with authenticate- and other with is-. The difference is that the authenticate- rules will prompt for user name and password, while the is- rules will grant the privilege when the user satisfies the rule with out a prompt.

There are also some rules whose names end in -nonshared. These have the shared key set to false. This means that these rules will not share credentials with other requests for authentication. Shared credentials will not prompt multiple times for the same credentials in a certain time (the timeout, usually 300 seconds/5 minutes). More sensitive settings are usually not shared.

Exporting and importing privileges

Changing the rule is a straightforward and fairly safe way to change a right’s behavior. When a rule fits your requirements, you should probably use a rule.

However, sometimes you need more fine-grained control of a right. For this you can export a right to a property list file, modify this and re-import it.

For example, say you do not want to grant every user access to a certain right but just a certain group of users even though they are not administrators. Common examples for this would be developers, teachers or lab techs. So, in our example, we create a group named techs:

$ sudo dseditgroup -o create -n . -r "Lab Techs" techs

to be safe we will nest the admin group in the techs group so that members of admin also gain all the rights of techs:

$ sudo dseditgroup -o edit -a admin -t group techs

and add the example user beth

$ sudo dseditgroup -o edit -a beth techs

You can verify that everything worked with:

$ sudo dseditgroup -o read techs
dsAttrTypeStandard:GeneratedUID -
        F9EBB33D-8A71-45EE-A65D-4DBBFF421B49
dsAttrTypeStandard:RecordName -
        techs
dsAttrTypeStandard:AppleMetaNodeLocation -
        /Local/Default
dsAttrTypeStandard:GroupMembers -
        87EE7744-5872-47C1-9574-D247D3DD4D5C
dsAttrTypeStandard:PrimaryGroupID -
        501
dsAttrTypeStandard:NestedGroups -
        ABCDEFAB-CDEF-ABCD-EFAB-CDEF00000050
dsAttrTypeStandard:RealName -
        Lab Techs
dsAttrTypeStandard:GroupMembership -
        beth
dsAttrTypeStandard:RecordType -
        dsRecTypeStandard:Groups

Now we can assign this group to a right. Our example will be the system.preferences.energysaver right. First export it to a property list file:

$ security authorizationdb read system.preferences.energysaver > energysaver.plist
YES (0)

Then change the value of the group key in the property list file to techs:

$ /usr/libexec/PlistBuddy -c "set group techs" energysaver.plist

If you want to learn more about working with property list files, I have written a book on it: ‘Property Lists, Preferences and Profiles for Apple Administrators’

And re-import the modified property list to the authorization database:

$ sudo security authorizationdb write system.preferences.energysaver < energysaver.plist

This alone is not sufficient to unlock the Energy Saver preference pane. When you look at the log with command from above, you can see that unlocking the energy saver pane requires the system.preferences right as well. You can modify this the same way:

$ security authorizationdb read system.preferences > system.preferences.plist
YES (0)
$ /usr/libexec/PlistBuddy -c "set group techs" system.preferences.plist
$ security authorizationdb write system.preferences < system.preferences.plist

Scripting

When scripting this setup you do not have to go through the export/import cycle but can directly import a prepared property list file or here doc:

Summary

There are many levels of access privileges in macOS. Most of them are controlled by different users and groups.

The super userroot exists on macOS like any other unix, but logging in as root is disabled by default as a security measure. The sudo command allows for interactive super user privileges in a shell. There are other ways administration scripts can be launched with super user privileges, such as LaunchDaemons, management systems etc.

The authorization database controls access to elevated rights in the macOS UI. You can modify it as an admin, but should do so with care.

Demystifying root on macOS: Conclusion

This ends my series on the super user in macOS. I hope it clarifies some confusing terms and configurations. As admins we have to use these privileges regularly. But, as the saying goes: “With great power comes great responsibility.”

Understanding how things work and how they affect the system will help you understand what to do when and how to avoid unexpected consequences.