Moving to zsh, part 5: Completions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

I am preparing a book on this topic, reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can pre-order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

What are Completions?

Man shells use the tab key (⇥) for completion. When you press that key, the shell tries to guess what you are typing and will complete it, or if the beginning of what you typed is ambiguous, suggest from a list of possible completions.

For example when you want to cd to your Documents folder, you can save typing:

% cd ~/Doc⇥

% cd ~/Documents/


When you hit the tab key, the system will complete the path to the Documents folder.

When the completion is ambiguous, the shell will list possible completions:

% cd ~/D⇥

Desktop/    Documents/  Downloads/


At this point, you can add a character or two to get to a unique completion, and hit the tab key again. In zsh you can also hit the tab key repeatedly to cycle through the suggested completions. In this example, the first tab keystroke will show the list, the second will complete ~/Desktop/, the third completes ~/Documents, and so on.

You can use tab completion commands as well:

% system⇥

system_profiler     systemkeychain      systemsetup         systemsoundserverd  systemstats

% system_⇥

% system_profiler 


Not having to type path and file names saves time and avoids errors, especially with complex paths with spaces and other special characters:

% cd ~/Li⇥
% cd ~/Library/Appl⇥

% cd ~/Library/Application S⇥

Application Scripts/  Application Support/

% cd ~/Library/Application Su⇥

% cd ~/Library/Application Support/


Using tab completion is a huge productivity boost when using a shell.

Turning It On

In the default configuration, tab completion in zsh is very basic. It will complete commands and paths, but not much else. But you can enable a very powerful, and useful completion system.

zsh comes with a tool you can use to setup this completion system. When you run the compinstall command it will lead you through a complex and hard to understand list of menus which explains the options and will generate the code necessary to set this configuration up and add it to your .zshrc file or another configuration file of your choice.

Since the commands to configure the completion are quite arcane and hard to understand, this is a good way to get something to start out with. I will explain some of these options and commands in detail.

Whether you use compinstall or not, to turn on the more powerful completion system, you need to add at least this command to your zsh configuration file:

autoload -Uz compinit && compinit


This will initialize the zsh completion system. The details of this system are documented here.

If you want to configure the system, the configuration commands (usually zstyle commands) should be added to the zsh configuration file before you enable the system. (This only matters for a few configurations, but as a general rule it is safer.)

All of these completion rules need to be loaded and prepared. zsh’s completion system creates a cache in the file ~/.zcompdump. The first time you run compinit it might take a noticeable time, but subsequent runs should use this cache and be much faster.

Sometimes, especially when building and debugging your own completion files, you may need to delete this file to force a rebuild:

% rm -f ~/.zcompdump
% compinit


Case Insensitive Completion

Since the macOS file systems are usually case-insensitive, I prefer my tab-completion to be case-insensitive as well. For bash you configure that in the ~/.inputrc. In zsh you modify the completion systems behavior with this (monstrous) command:

# case insensitive path-completion

zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*'


I have seen many varieties for this configuration in different websites, but this is what compinstall adds when I select case-insensitive completion, so I am going with that.

Partial Completion

This is a particularly nice feature. You can type fragments of each path segment and the completion will try to complete them all at once:

% cd /u/lo/b⇥

% cd /usr/local/bin

% cd ~/L/P/B⇥

% ~/Library/Preferences/ByHost/


If the fragments are ambiguous, there are different strategies to what the completion system suggests. I have configured these like this:

# partial completion suggestions

zstyle ':completion:*' list-suffixes
zstyle ':completion:*' expand prefix suffix


Commands with built-in completion

zsh comes with several completion definitions for many commands. For example, when you type cp and then hit tab, the system will correctly assume you want to complete a file path and show the suggestions from the current working directory.

However, when you type cp -⇥ the completion can tell from the - that you want to add an option to the command and suggest a list of options for cp, with short descriptions.

% cp -⇥
 
-H  -- follow symlinks on the command line in recursive mode
 
-L  -- follow all symlinks in recursive mode

 -P  -- do not follow symlinks in recursive mode (default)
 
-R  -- copy directories recursively

 -X  -- don't copy extended attributes or resource forks
 
-a  -- archive mode, same as -RpP

 -f  -- force overwriting existing file
 
-i  -- confirm before overwriting existing file

 -n  -- don't overwrite existing file
 
-p  -- preserve timestamps, mode, owner, flags, ACLs, and extended attributes
 -v  -- show file names as they are copied


As the context of command prompt you are assembling changes, you may get different completion suggestions. For example, the completion for ssh will suggest host names:

% ssh armin@⇥


zsh comes with completion definitions for many common commands. Nevertheless, it can be helpful to just hit tab, especially when wondering about options.

On macOS completions are stored in /usr/share/zsh/5.3/functions (replace the5.3 with 5.7.1 in Catalina). This directory stores many functions used with zsh and is in the default fpath. All the files in that directory that start with an underscore _ contain the completion definitions command. So, the file _cp contains the definition for the cp command. (Some of the definition files contain the definitions for multiple commands.)

Completions for macOS Commands

There are even a few macOS specific command that come with the default zsh installation.

% system_profiler ⇥⇥


macOS High Sierra and macOS Mojave come with zsh 5.3, which is now nearly two years old. zsh 5.3 contains less macOS specific completion definitions than the current zsh 5.7.1 which will is the pre-installed zsh in macOS Catalina. Some of the completions in 5.3 have also been updated in 5.7.1.

Tool zsh 5.3 zsh 5.7.1
caffeinate
defaults
fink
fs_usage
hdiutil
mdfind
mdls
mdutil
networksetup
nvram
open
osascript
otool
pbcopy/pbpaste
plutil
say
sc_usage
scselect
scutil
softwareupdate
sw_vers
swift
system_profiler
xcode_select

Load bash completions

Since the default shell on macOS has been bash for so long, there are quite a few bash completion definitions for macOS commands and third party tools available. For example Tony Williams’ bash completion for autopkg (post, Github).

You do not have to rewrite these completions, since the zsh completion system can use bash completion scripts as well: (add this to your zsh configuration file)

# load bashcompinit for some old bash completions

autoload bashcompinit && bashcompinit


[[ -r ~/Projects/autopkg_complete/autopkg ]] && source ~/Projects/autopkg_complete/autopkg


When you have multiple bash completion scripts you want to load, you only need to load bashcompinit once.

Build your own completions

Once you start using completions, you will want to have them everywhere. While many built-in completions exists, there are still many commands that lack a good definition.

Some commands, like the swift command line tool, have a built-in option to generate the completion syntax. You can then store that in a file and put it in your fpath:

% swift package completion-tool generate-zsh-script >_swift


Note: in the case of swift, its definition will conflict with the _openstack definition in zsh 5.3. You can fix this with the command compdef _swift swift after loading the completion system.

Some commands provide a list of options and arguments with the -h/--help option. If this list follows a certain syntax, you can get a decent completion working with

% compdef _gnu_generic <command>


One example on macOS, where this has decent results is the xed command which opens a file or folder in Xcode.

But for best results, you will often have to build the description yourself. Unfortunately this is not a simple task. The syntax is meticulously, but also quite abstractly documented in the zsh documentation for the Completion System. I also found the ‘howto’ documentation in the zsh-completions repository very useful, as well as the ‘zsh Completion Style Guide.’

To avoid everyone re-inventing the wheel, I have started a repository on Github for macOS specific completion files. The page has the instructions on how to install them and I will welcome pull requests with contributions. Since I am just starting to learn this as well, I am sure there are improvements that can be made on the completions I have built so far and there are several commands where you can test your skills and build a new one.

I suggest the #zsh channel on the MacAdmins Slack for discussion.

Next

In the next post in this series, we will discuss how to configure zsh’s command line prompt.

Install Bash 5 on macOS with Patches

I recently posted an article on how to download, install, and build a macOS installer pkg for bash 5. In that first version of this post I ignored patches, minor updates to the bash source code and binary. But as the patches to bash 5 are accumulating, I cannot ignore them much longer.

This post will extend the instructions in the original post.

After downloading and expanding the bash-5.0.tar.gz, create a patches folder:

$ cd path/to/bash-5.0
$ mkdir patches
$ cd patches

You can download the patches for bash-5.0 here. As of this writing, there are seven patches for bash-5.0 labelled bash50-001 through bash50-007. You can download all at once with:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' -O

(Adapt the numbers when there are more patches in the future.)

Then move up one directory level to the bash-5.0 root directory and apply the patches using the patch command.

$ cd ..
$ patch -p0 -i patches/bash50-001
$ patch -p0 -i patches/bash50-002

(etc.)

You can download and patch with a single step. Make sure your working directory is the bash-5.0 with all the code and run:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' | patch -p0

From here, you can continue with the remaining build steps from the original post. The next step will be running ./configure.

The script to build the pkg installer has also been updated in the repository to download and apply the patches before building.

Moving to zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

zsh (I believe it is pronounced zee-shell, though zish is fun to say) will succeed bash as the default shell. bash has been the default shell since Mac OS X 10.3 Panther.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

Why?

The bash binary bundled with macOS has been stuck on version 3.2 for a long time now. bash v4 was released in 2009 and bash v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3. bash v3 is still GPL v2.

zsh, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default. zsh has been available as on macOS for a long time. The zsh version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current zsh 5.7.1.

Is bash gone!?

No.

macOS Catalina still has the same /bin/bash (version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.

Many scripts in macOS, management systems, and Apple and third party installers rely on /bin/bash. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.

Users that have /bin/bash as their default shell on Catalina will see a prompt at the start of each Terminal session stating that zsh is now the recommended default shell. If you want to continue using /bin/bash, you can supress this message by setting an environment variable in your .bash_profile or .bashrc.

export BASH_SILENCE_DEPRECATION_WARNING=1

You can also download and install a newer version of bash yourself. Keep in mind that custom bash installations reside in a different directory, usually /usr/local/bin/bash.

Will bash remain indefinitely?

Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to bash, but didn’t really care if you remained on tcsh. In fact, tcsh is still present on macOS.

Apple’s messaging should tell us, that the days of /bin/bash are numbered. Probably not very soon, but eventually keeping a more than ten year old version of bash on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.

Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’ dash has been added to the lineup of shells. dash is designed to be a minimal implementation of the Posix standard shell sh. So far, in macOS (including Catalina),sh invokes bash in sh-compatibility mode.

As Apple’s support article mentions, Catalina also adds a new mechanism for users and admins to change which shell handles sh invocations. MacAdmins or users can change the symbolic link stored in /var/select/sh to point to a shell other than /bin/bash. This changes which shell interprets scripts the #!/bin/sh shebang or scripts invoked with sh -c. Changing the interpreter for sh should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.

All of these changes are indicators that Apple is preparing to remove /bin/bash at some, yet indeterminate, time in the future.

Do I need to wait for Catalina to switch to zsh?

No, zsh is available Mojave and on older macOS versions. You can start testing zsh or even switch your default shell already.

If you want to just see how zsh works, you can just open Terminal and type zsh:

$ zsh
MacBook%

The main change you will see is that the prompt looks different. zsh uses the % character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in bash.

If you want to already switch your default shell to zsh you can use the chsh command:

$ chsh -s /bin/zsh

This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.

How is zsh different?

Like bash (‘Bourne again shell’ ), zshderives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.

The main difference between bash and zsh is configuration. Since zsh ignores the bash configuration files (.bash_profile or .bashrc) you cannot simply copy customized bash settings over to zsh. zsh has much more options and points to change zsh configuration and behavior. There is an entire eco-system of configuration tools and themes called oh-my-zsh which is very popular.

zsh also offers better configuration for auto-completion which is far easier than in bash.

I am planning a separate post, describing how to transfer (and translate) your configurations from bash to zsh.

What about scripting?

Since zsh has been present on macOS for a long time, you could start moving your scripts from bash to zsh right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to #!/bin/zsh.

You will gain some features where zsh is superior to bash v3, such as arrays and associative arrays (dictionaries).

There is one exception where I would now recommend to use /bin/sh for your scripts: the Recovery system does not contain the /bin/zsh shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider older macOS installations where zsh is definitely not present in Recovery.

When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as Twocanoes MDS, installr or bootstrappr, then you cannot rely on /bin/zsh.

Since we now know that bash is eventually going away, the only common choice left is /bin/sh.

When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using /bin/sh as the shebang from now on.

I used to recommend using /bin/bash for everything MacAdmin related. /bin/sh is definitely a step down in functionality, but it seems like the safest choice for continued support.

Summary

Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’ bash with zsh, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!

Next

In the next part we will look at the configuration files for zsh.

Open Apps with custom Shortcuts in macOS

Someone on the MacAdmins Slack recently asked how you could assign a global keyboard short cut to open Terminal on macOS.

Note: alternative terminal applications such as iTerm2 may have this built-in.

macOS has an option to assign custom global keystrokes to pretty much anything, but it is not obvious how to get there.

  • First, open the Automator application. In the chooser for a new Workflow, choose ‘Quick Action’ (on Mojave) or ‘Service’ on earlier versions of macOS.
The new Workflow chooser in Mojave
The new Workflow chooser in Mojave
  • In the new workflow configure the input to be ‘no input’ and the application to be ‘any application.’
  • Then search for ‘Launch Application’ action in the library pane on the left and add it to your workflow by double-clicking or dragging.
  • The popup menu where you can slect an application in the action will only show applications from the /Applications folder. Choose ‘Other…’ and select Terminal in the ’/Applications/Utilities` folder.
Configure your workflow
Configure your workflow
  • Save the workflow. Give it a meaningful name such as ‘Open Terminal.’ Since you chose Quick Action or Service, this workflow will be saved in ~/Library/Services.
  • Open System Preferences > Keyboard. Click the ‘Shortcuts’ tab and select ‘Services’ from the list on the left side. (Even on Mojave, it is still called ‘Services’.)
  • Scroll all the way down the list of services under the ‘General’ heading, you should find the service you just created. Select it and click ‘Add Shortcut’ to assign a global shortcut.
Keyboard Shortcut Preferences
Keyboard Shortcut Preferences
  • You are done!

When the active application uses the same keystroke, the application’s definition will precede your global shortcut.

Of course, you don’t have stop at launching applications. You can assign a global keyboard shortcut to any Automator workflow this way. Since Automator workflows can include AppleScript, Python or shell scripts, you can do pretty much anything this way!

However, most Apple users don’t bother with shortcuts to launch apps. Just invoke Spotlight with command-space and start typing term and hit return.

On Smart Quotes and Terminal

Typography is a wonderful art and has a long history. When humans turned from manual typesetting to machines, type writers and then computers, some compromises had to be made. One of these compromises was to use simple straight quote symbols for opening and closing the quote, rather than different quote symbols for opening and closing.

Note: which kind of quotes are used for opening and closing dependent on the language or and some convention. English uses upper quotes “…”, German opens with a lower quote: „…“, French uses ‘guillemets:’ «…», and Japanese uses hooks: 「…」

Quotation Marks have funny names in many languages. Germans call them “Gänsefüßchen,” or “little goose feet.”

See this Wikipedia article for more details.

macOS, iOS and other modern operating systems have a feature which replaces the simple or straight quote symbols with the typographic quotes. So, you type "Hello!" and the quotes are automatically replaced with the proper (depending on localization) typographic quotes. This is called “smart quotes.”

This is pretty nice, but can be troublesome when dealing with Terminal and text editors. Scripting languages and shells always use straight quotes, and cannot deal with typographic quotes.

Now, if someone sends you a command or a script that uses quotes, and it goes through an app that replaces them with smart quotes, then bash and Terminal will fail miserably.

There is not much you can do, other than be aware of this and check pasted code carefully. There is something you can do to make this easier, though.

The default monospace font used in Terminal on macOS are ‘Menlo’ or ‘SF Mono,’ depending on the macOS version. Now these are fine typefaces, but their typographic quotes are not very curly at all, making them very hard to distinguish from the ‘dumb’ straight quotes that Terminal expects. The classic ‘Monaco’ typeface on the other hand has beautiful curly typographic quotes, making them very distinct from the the straight quote.

My favorite mono space typeface ‘Source Code Pro’ also has nice curly typographic quotes. I have built this table with many common monospace typefaces and their quotes.

Quotation Mark Comparison
Quotation Mark Comparison

Now this shouldn’t be your only criteria in choosing your Terminal font, but it may be something that helps avoid quote errors.

Install Bash 5 on macOS

The default bash on macOS is still bash v3:

$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.

Just recently, bash v5 was released. The discrepancy comes from the fact that bash has been licensed as GPL v3 since version 4. Apple does not include GPL v3 licensed tools with macOS.

However, nothing is keeping you from downloading and installing the latest bash version.

New features include, among many other things, associated arrays (i.e. dictionaries) and better auto-completion setup.

While you would think this is a common desire, most pages I have found will simply point to Homebrew to download and install a newer bash version.

The main challenge with using brew is that it does not work on the scale that MacAdmins require. brew is designed for single user installation, where the user has administrator privileges. brew’s workflows do not scale to large deployments controlled with a management system.

Ideally, there would be package installer for the latest bash version. Unfortunately, the bash project does not provide one.

In this post, I will show how you can install the latest bash version without brew and how to build an installer package for deployment.

Manual Installation

This requires Xcode or the Developer Command Line Tools to be installed.

First, download the source for the latest bash version from this page. As of this writing the latest version is bash-5.0 and the file you want is bash-5.0.tar.gz. Once downloaded, you can expand the archive in Finder by double-clicking.

Update: I have a post with some updated instructions to include the patches to bash 5.0.

Open a Terminal window and change directory to the newly expanded bash-5.0 directory. Then run the configure script there.

$ cd ~/Downloads/bash-5.0
$ ./configure

The configure process will take a while, there will be plenty of messages showing progress.

Once the configure process is complete. You can build bash with the make command.

$ make

This will build the bash binary and the supporting files in the current directory. That’s not where we want it in the end, but it is probably a good idea see if the build process works. This will (again) take a while. There will be some odd looking warnings, but you can ignore those.

When make succeeds, you can actually install bash v5 with

$ sudo make install

This will build and install the bash binary and supporting files in /usr/local/bin and /usr/local. sudo is required to modify /usr/local.

If you were just looking for a way to install bash v5 without brew, you are done!

There is more useful information in the rest of the post, though, so keep reading!

How the new and the old bash interact

By default, the bash v5 binary is called bash and will be installed in /usr/local/bin. The macOS default PATH lists /usr/local/bin before /bin where the default bash v3 binary, also called bash, is located.

This means, that when a user types bash in to a shell, the version in /usr/local/bin will be preferred over the pre-installed bash v3.

You can test this behavior in Terminal. Since the default shell has not yet been changed from /bin/bash the Terminal still opens to bash v3. You can test this by showing the BASH_VERSION environment variable:

$ echo $BASH_VERSION
3.2.57(1)-release

But when you then run bash it will invoke /usr/local/bin/bash, so it will run the new bash v5. It will show this in the prompt, but you can also verify the BASH_VERSION.

$ bash
bash-5.0$ echo $BASH_VERSION
5.0.0(2)-release

This might be the setup you want, when you want to use bash v5 always. It might lead to some unexpected behavior for some users, though.

One option to avoid this ambiguity is to rename the binary in /usr/local/bin to bash5. But then other tools such as env (mentioned below) will not find the binary any more.

Note: the PATH in other contexts will likely not contain /usr/local/bin and further confuse matters.

bash v5 and Scripting

Scripts using bash, should have the full path to the binary in the shebang. This way, the script author can control whether a script is executed by the default bash v3 (/bin/bash) or the newer bash v5 (/usr/local/bin/bash or /usr/local/bin/bash5).

It is often recommended to use the env command in the shebang:

#!/usr/bin/env bash

The env command will determine the path to the bash binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the bash binary is unknown, in other words across multiple Unix and Unix-like platforms. However, this renders the actual version of bash that will interpret the script unpredictable.

For example, assume you have bash v5 installed in the default configuration (as /usr/local/bin/bash. A script with the shebang #!/usr/bin/env bash launched in the user environment (i.e. from Terminal) will use the newer bash, as /usr/local/bin comes before /bin in the search order.

When you launch the same script in a different context, e.g. as an installation script, an AppleScript, or a management system, /usr/local/bin will likely not be part of the PATH in that environment. Then the env shebang will choose /bin/bash (v3). The script will be interpreted and might behave differently.

Administrators prefer certainty in their managed environments. Administrators should know the location and versions of the binaries on their systems. For management scripts, you should avoid env and use the proper full path to the desired interpreter binary.

The solutions to resolve the ambiguity are

  • use the full path to the binary in the shebang
  • manage and update the additional custom version of bash with a management system
  • (optional) rename the newer bash binary to bash5 or bash4 (this also allows you to have bash v4 and bash v5 available on the same system)
  • Scripting OS X: On the Shebang
  • Scripting OS X: Setting the PATH in Scripts

Changing a user’s default Shell to bash v5

Even though we have installed bash v5, the default shell of a new Terminal window will still use the built-in bash v3.

The path to the default shell is stored in the user record. You can directly change the UserShell attribute with dscl, in the ‘Advanced Options’ of the ‘Users & Groups’ preference pane, or in Directory Utility.

There is also a command to set the default shell:

$ chsh -s /usr/local/bin/bash
Changing shell for armin.
Password for armin: 
chsh: /usr/local/bin/bash: non-standard shell

The chsh (change shell) command will check for allowed shells in the /etc/shells file. You can easily append a line with /usr/local/bin/bash to this file, and then chsh will work fine.

$ chsh -s /usr/local/bin/bash
Changing shell for armin.
Password for armin: 

Note: if you choose to rename the bash binary, you have to use the changed name in /etc/shells and with chsh.

Remember that just running chsh will not change the shell in the current Terminal window. It is best to close the old Terminal window and open a new one to get the new shell.

Packaging bash v5 for mass deployment

While these steps to install and configure bash v5 on a single Mac are simple enough, they would not work well with a management system for hundreds or thousands of Macs. We want to wrap all the files that make install creates into a package installer payload.

The --help option of the configure script yields this useful information:

By default, make install' will install all the files in/usr/local/bin,/usr/local/libetc. You can specify an installation prefix other than/usr/localusing–prefix, for instance–prefix=$HOME`.

When we run the configure script with the --prefix option it creates a folder suitable as a payload for a package installer. We can then use pkgbuild to build to create an installer pkg:

$ cd ~/Downloads/bash-5.0
$ mkdir payload
$ ./configure --prefix=/Users/armin/Downloads/bash-5.0/payload
$ make install
$ pkgbuild --root payload --install-location /usr/local --identifier org.gnu.bash --version 5.0 bash-5.0.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to bash-5.0.pkg

(Note: the --prefix argument requires an absolute path.)

Automate the package creation

So, we have our workflow for building an installer package to distribute and configure bash v5:

  • download the archive
  • extract the archive
  • run configure with the --prefix argument
  • run make install to create the files in a payload folder
  • optional: rename the resulting bash binary to bash5 to avoid conflicts
  • add a postinstall script that adds /usr/local/bin/bash[5] to /etc/shells if not yet present
  • build the installer with pkgbuild

This sounds like a workflow ripe for automation. You can get the script from this repository.

You can pass a different (valid) bash version number as an argument to the script, e.g. 4.4.18. (I did not test anything significantly older.) The script does not autodetect the latest version and defaults to version 5.0 when no argument is given. When an update to bash v5 is published, you will have to modify the version line or run the script with an argument.

I have not (yet) figured out how to detect the latest version from the download web page. An autopkg recipe will have to wait for that. (If someone else wants to tackle that, please do!)

Apple Two-Factor Authentication for a Secondary Apple ID

Apple sent an email to developers, stating that later this months, two-factor authentication will be required for Apple IDs used for developer accounts.

If you, like me, use separate Apple IDs for your personal iCloud and your developer accounts, this will pose some kind of challenge. There is a solution, however Apple does not document it very well.

Update: Apple now has a very detailed support page for this topic.

Two-factor authentication for the primary account

Assumption: you have two-factor authentication (2FA) enabled on your primary, personal Apple ID, and are logged in to that account on your Mac(s) and iOS devices. If you haven’t done that yet, do it now. 2FA does increase your account security significantly.

You can enable 2FA on any device logged in to the account in the iCloud Settings or preference pane. As part of the setup you can provide one or more phone numbers as a fall back mechanism. If no devices can be prompted through Apple’s built-in 2FA, it will send an SMS to the trusted phone numbers. You can use the same phone number for multiple Apple IDs, but there seems to be some limit on how often you can do that.

Enable 2FA for the secondary account

Assumption: The secondary account is your developer Apple ID, you don’t use it for iCloud storage, device backups, mail etc. You use it to log in to developer.apple.com and iTunes Connect, and to get all the certificates and other resources you need as a developer.

The challenge here is that you can only enable 2FA on the first account logged in to iCloud on a device. You could log out of your primary iCloud account, and the log in with the secondary, but this will disrupt a lot of things on your device. I’d rather avoid that.

On a Mac, you can have a separate iCloud account for each local user. So, it is easiest to create a second user account, log out of your first account, log in to the new second account and set up iCloud and 2FA for the developer Apple ID on this second local account.

You can sign in to the secondary Apple ID enable 2FA in System Preferences -> iCloud -> Account Details -> Security as described in Apple’s Support Article.

Follow the prompts to set up 2FA, you can re-use the same phone number as a trusted number. (There seem to be limits to how often you use the same phone number, but two accounts works fine for me.)

Once 2FA is set up, we don’t need the second user account on the Mac any more. Sign out of iCloud, log out of the second account and back in to your normal user account.

If you are ok with using SMS authentication (Apple calls this ‘two-step authentication’, rather than ‘two-factor authentication’) then you are done. However, many will argue codes over SMS are not good enough for secondary authentication, so we want go to ‘full’ 2FA.

Use the secondary Apple ID

As it turns out, you can be logged in to multiple iCloud accounts on the same device or account. Certain services, such as iCloud storage, or the Photo Library, will only work with the primary iCloud account, but other services, including 2FA, will work for all iCloud accounts.

On your iOS device go to Settings > Passwords & Accounts > Add Account, and choose to add another iCloud account. You probably want to turn off all services, like Mail, Calendar, etc. secondary account.

Second iCloud Account on iPhone
Second iCloud Account on iPhone

On the Mac you can do the same in System Preferences > Internet Accounts. You can use both your Mac and iOS devices for 2FA.

Second iCloud Account on macOS
Second iCloud Account on macOS

Now the secondary Apple ID will prompt the devices you are logged in as for 2FA.

2FA prompt on the iPhone
2FA prompt on the iPhone

PLIST Editor can Open Signed Mobileconfigs

Last year I posted about a useful app to read and edit property list files: PLIST Editor.

In its last update (1.15) the app gained a feature which is tremendously useful to MacAdmins: it can now open signed property lists.

MacAdmins regularly encounter signed property lists when they download configuration profiles (.mobileconfig) from a management server. Since the cryptographic signature is binary data wrapping the property list, most property list editors, such as Xcode or PlistEdit Pro, choke on the signature.

The new PLIST Editor (1.15) now detects a signed mobileconfig and will automatically unwrap the plist data from the signature and display a notification.

PLIST Editor Message

When you click on the notification, you will get more detailed information on the signature itself.

Once you edit and save the mobileconfig, the signature will obviously be removed. You can re-sign the edited mobileconfig file with a tool like Hancock or in the Terminal with:

$ security cms -S -N "Identity Name" -i profile.mobileconfig -o signed.mobileconfig

(Learn more on in my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’)

This addition will be very useful for my workflow and think for other MacAdmins as well.

PLIST Editor is available in the Mac App Store for US$3.99 (price will vary depending on region).

On macOS User Groups

User groups are easy, right? A user is either a member or they are not.

Once you start thinking about the details and want or need to automate some of the aspects of user and group management on macOS, there is a lot of devil in those details.

User Membership

You can easily list all groups a given user is a member of. The id command will show all the groups the current user is a member of. id -Gn will list just the groups. Add a username to the id command to see the information for a different user. The groups command does the same as id -Gn.

You can also run a command to check if a given user is a member of a group:

$ dseditgroup -o checkmember -m user staff
yes user is a member of staff
$ dseditgroup -o checkmember -m user wheel
no user is NOT a member of wheel

Group Membership

So far, so good.

A user is a member of a group when one of these applies:

  • the user’s PrimaryGroupID attribute matches the PrimaryGroupID of the group
  • the user’s UUID is listed in the group’s GroupMembers attribute and the user’s shortname is listed in the group’s GroupMembership
  • the user is a member of a group nested in the group

Note: you should not attempt to manipulate the GroupMembers or GroupMembership attributes directly. Use the dseditgroup -o edit command to manage group membership instead. dseditgroup syntax is weird, but it is a really useful tool. Study its man page.

Listing Group Members

Sometimes (mainly for security audits) you need to list all the members of a group. With the above information, it is easy enough to build a script that checks the PrimaryGroupID, the GroupMembership attribute and the recursively loops through the NestedGroups.

This is confused by the fact that PrimaryGroupID stores the numeric User ID, GroupMembership uses the shortname and NestedGroups uses UUIDs. Nevertheless, you can sort through it.

I have written exactly such a script here:

In most cases this script will work fine. But, (and you knew there would be a “but”) macOS has a very nasty wrench to throw in our wheels.

Calculated Groups

There are a few groups on macOS, that have neither GroupMembers, GroupMembership, nor NestedGroups, but still have members. This is because the system calculates membership dynamically. This is similar to Smart Playlists in iTunes, Smart Folders in Finder, or Smart Groups in Jamf Pro.

You can list all calculated groups on macOS with:

$ dscl . list /Groups Comment | grep "calc"

The most interesting calculated groups are everyone, localaccounts, and netaccounts.

These groups can be very useful in certain environments. For example in a DEP setup you could add localaccounts or everyone to the _lpadmin and _developer groups, before the user has even created their standard account. That way any user created on that Mac will can manage printers and use the developer tools.

However, since these groups are calculated magically, a script cannot list all the members of any of these groups. (My script above will show a warning, when it encounters one of these groups.)

While it would probably not be wise to nest the everybody group in the admin group, a malicious user could do that and hide from detection with the above script (or similar methods).

Other Solution

Instead of recursively listing all users, we can loop through all user accounts and check their member status with dseditgroup -checkmember. This script is actually much simpler and dseditgroup can deal with calculated groups.

This works well enough when run against all local users.

I strongly recommend against running this for all users in a large directory infrastructure. It’ll be very slow and generate a lot of requests to the directory server. Because of this the script above runs only on the local directory node by default.

Summary

  • on macOS users can be assigned to groups thorugh different means
  • you can check membership with dseditgroup -o checkmember
  • you can edit group membership with dseditgroup -o edit
  • macOS has a few groups which are dynamically calculated and difficult to process in scripts

Mojave Quick Action to Package Apps

One of the new macOS features in Mojave are “Finder Quick Actions.”

They show as action buttons in the Finder in Column View and the new Gallery View. You can also access Quick Actions in any view from an item’s context menu.

You can configure Quick Actions in the ‘Finder’ section of the ‘Extensions’ Preference Pane. The four sample actions that come with Mojave are ‘Rotate,’ ‘Markup,’ ‘Create PDF,’ and ‘Trim’. While these are certainly useful, they are oriented towards media.

You can, however, build your own Quick Actions with Automator!

Custom Quick Action

When you look at it closely, Quick Actions are re-branded Automator Service Workflows. They are even stored in ~/Library/Services.

Let’s build a useful Quick Action for admins.

Quick Packaging

Recap: You can quickly build an installer package from an application bundle with pkgbuild:

$ pkgbuild --component /Applications/Firefox.app Firefox-63.0.pkg

This even works when the application is not installed on the current system, but then you have to add the --install-location argument:

$ pkgbuild --component /Volumes/Firefox/Firefox.app/ --install-location /Applications Firefox-63.0.pkg

This allows you to build an installer package from a disk image without having to install it.

Note: this method works well with ‘drag to install’ type applications. For many other applications, re-packaging is more elaborate. Learn all the details of packaging in my book: “Packaging for Apple Adminstrators

To make this simple process even simpler, I wrote a small script a while back called quickpkg, which simplifies this even further:

$ quickpkg "~/Downloads/Firefox 63.0.dmg"

(For help with downloading and installing scripts like quickpkg see this post.)

This seems like a good candidate for a Quick Action.

Bring in the Robot

Open the Automator application and create a new Workflow. Choose “Quick Action” as the type of Workflow.

New Workflow Window for a Quick Action in Finder

This will present an empty workflow window with a list of categories and actions on the left and the area where you assemble the workflow on the right. Above the workflow area is a panel where you define the input for our Quick Action. Change the popups to match this image:

Input for our QuickAction
Input for our QuickAction

Then add a ‘Run Shell Script’ action from the list on the left. The easiest way to locate the action is with the search box, or you can find the ‘Run Shell Script’ action in the ‘Utilities’ category. Drag it to the workflow area, labelled ‘Drag actions or file here to build your Workflow.’

Make sure that the shell popup is set to /bin/bash and the ‘Pass input’ popup is set to ‘to stdin’. With these settings, the selected file(s) will be passed as a list of paths, one per line to the stdin stream of the bash code we enter in the text area.

Add the following code:

destination="$HOME/Documents/"

while read -r file; do
    /usr/local/bin/quickpkg --output "$destination" "$file"
done

Your action should look like this:

First, we set a bash variable for the destination folder. You can change this to another path if you want to, but the destination folder has to exist before the workflow runs, otherwise you’ll get an error.

Then we use a while loop with the read command to read the stdin input line by line. Then we run the quickpkg tool once for each line.

You can now save the Workflow (it will be saved in ~/Library/Services/) and then ‘QuickPkg’ (or whatever name you chose) will appear in Finder, for any selected item. Unfortunately, the Automator input controls don’t allow to filter for file types other than the few given ones.

Select a dmg with an application in it, such as the dmg downloaded from the Firefox website and wait a moment. Then check the ~/Documents folder for the result.

(A rotating gear wheel will appear in the menu bar while the action is running. This is all the feedback you can get.)

Revealing the Result

It is annoying that we have to manually open the destination folder to see our result. But the nice thing is that we can let the workflow take care of that. In the action list on the left, search for ‘Reveal Finder Items’ or locate this action in the ‘Files & Folders’ category. You can drag it to the end of your workflow, below the ‘Run Shell Script’ action or just double-click in the list to add to the end of your workflow.

The extended Workflow

Save and run again from the Finder. It should now reveal the pkg file automatically.

You can add more actions to the workflow. For example, you can add actions to

  • open with Pacifist or Suspicious Package
  • tag the pkg file
  • add a comment
  • append the date to the file name
  • copy the pkg to a file share

Improving the Workflow

You may have noticed during testing that in its current form the workflow doesn’t really react well when something goes wrong.

quickpkg can work with .app, .dmg, and .zip files. Unfortunatly, Automator does not let us filter for just those file types in the input setup. The script will report an error when you try to run it against a different file type, but the error is not displayed in Finder.

It is not that difficult to extend our short script to make it a bit more resilient. Change the code in the ‘Run Shell Script’ action to this:

destination="$HOME/Documents/"

while read -r file; do
    result=$(/usr/local/bin/quickpkg --output "$destination" "$file")

    if [[ $? != 0 ]]; then
        osascript -e "display alert \"QuickPkg: An error occured: $result\""
    else
        echo "$result"
    fi
done

With this code we check result code $? of the quickpkg command for an error. If the code is non-zero we display an alert with osascript. If all went well, we echo the result (the path to the new pkg) to stdout, so that Automator can pass that into the following actions.

This is still a fairly crude error handling, but much better than nothing.

Summary

It is quite easy to turn simple shell commands into Automator workflows and Finder Quick Actions. Keep this in mind, when you find yourself performing repetetive tasks in Finder or in Terminal.