Build a shellcheck installer for macOS

Shellcheck is an invaluable tool for anyone who writes, well, shell scripts. The tool will point out common and less common errors, such as forgetting to quote a variable substitution, not putting that space before the closing ]], or not testing if a variable substitution used with rm -Rf might be empty.

You can use the tool’s website to check your code, but it is unwieldy, not to mention you or your organization might have concerns uploading all your code to some website, useful as it may be. You can install a shellcheck command line tool, but the instructions available for macOS will only work with Homebrew.

If you already have Homebrew installed, or don’t mind installing it and only need to install shellcheck on a single Mac, then you can use brew install shellcheck and skip ahead to the end, where I share how to use the shellcheck binary with BBEdit.

Relevant for MacAdmins

Homebrew may be a fine tool for individual users, but it is near-impossible to centrally manage a deployment of brew let alone a piece software installed with brew.

My attempts to build the shellcheck binary without homebrew lead to several dead-ends. Shellcheck is written in Haskell Cabal, which can be easily installed with… brew. There is also an official ghcup script, which is the ‘recommended way to install Haskell’, but this script fails on macOS because it requires a tool named xz. This is a (de-)compression tool, which is part of xz-utils, whose website does not explain how to install on macOS, but can be easily installed with… guess… brew

Even without brew, you have to install a number of tools that you probably don’t want hanging around on your production machine. I am sure cabal is a fine environment to build other tools as well, but I don’t need it on my production system.

I would recommend running these steps on a virtual machine. That will simplify ‘cleanup’ as well as the ‘start over’ process, should you need it.

How to build the shellcheck binary for macOS

Commands shown are correct at the time this article was written and may change in the future. Links to original sources are given, so you can check for changes.

Developer Tools

  • install Xcode or command line developer tools (CLI dev tools have the advantage of not being >6GB in size)
$ xcode-select --install

XZ Utils

Note: the macOS Archive Utility can deal with .xz archives just fine. There is really no need on macOS to have the xz-utils installed, but the GHC Cabal installer requires them.

$ cd ~/Downloads/xz-5.2.4
$ ./configure
$ sudo make install
  • check to see of xz was installed correctly
$ xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.4

Install Pandoc

We will require pandoc to build the man page for shellcheck.

Haskell Cabal

$ curl https://get-ghcup.haskell.org -sSf | sh
  • make sure the cabal environment is setup
$ source "$HOME/.ghcup/env"
$ cabal update

Shellcheck

$ git clone "https://github.com/koalaman/shellcheck.git"
  • build the shellcheck binary (this will take a while)
$ cd shellcheck
$ cabal install
  • this will build and install the shellcheck binary into ~/.cabal/bin, you can test if this worked with
$ ~/.cabal/bin/shellcheck --version
ShellCheck - shell script analysis tool
version: 0.6.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net

man page

  • there is also a man page in the repo, written in markdown format. The ReadMe says you can convert it to man format with pandoc:
$ pandoc -s -f markdown-smart -t man shellcheck.1.md -o shellcheck.1

Pkg it up

  • create a payload directory for the binary and man file and move the files there:
$ mkdir -p payload/local/bin
$ cp ~/.cabal/bin/shellcheck payload/local/bin/
$ mkdir -p payload/share/man/man1
$ cp shellcheck.1 payload/share/man/man1/
  • build the pkg installer from the payload with pkgbuild
$ pkgbuild --root payload --install-location /usr --identifier com.scriptingosx.pkg.shellcheck --version 0.6.0 shellcheck-0.6.0.pkg

Automation!

Of course, you can do this once and then forget about until the next update of shellcheck. While I wrote this post, I had to go through these steps more than once and got bored with it halfway through the second cycle. So, there is a script, which you can get at this Github repo.

Since this script will install xz-utils, cabal, and pandoc, I do not recommend to run this on your work machine. I use it with clean virtual machine. I suggest this workflow:

  • boot to clean vm
  • snapshot the vm
  • open Terminal
  • install developer command line tools: xcode-select --install
  • clone the script: git clone https://github.com/scriptingosx/BuildShellcheckPkg
  • run the script: cd BuildShellcheckPkg; ./buildShellCheckPkg.sh
  • confirm the prompts (once the cabal installation starts for good, the prompts are over)
  • wait…
  • wait…
  • start doing other work such as answering emails, MacAdmins Slack, Twitter,…
  • much later, remember there was something important, go to the vm, copy the brand new shellcheck pkg to your work machine
  • revert the vm to snapshot
  • buy ‘Packaging for Apple Administrators’ to show your gratitude and learn how to wield pkgbuild yourself

Using shellcheck

Using the shellcheck command line tool is very straightforward. Just pass one (or more) shell script files as arguments. There are some flags and options that you can read about in the --help option or the man page.

$ shellcheck serial.sh

In serial.sh line 11:
serial_number=$(serial) # stores output of function serial in variable
^-----------^ SC2034: serial_number appears unused. Verify use (or export if used externally).

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- serial_number appears unused. Ver...

Use shellcheck with BBEdit

My favored text editor – BBEdit – has a tool that will display the output of tools like shellcheck. This allows you to click on an error or warning and see and edit the code right there in a BBEdit window.

To get this, pipe the output of shellcheck into bbresults:

$ shellcheck -f gcc serial.sh | bbresults

I have created a function in my .bash_profile and .zshrc, so I don’t have to remember the details:

function bbshellcheck {
    shellcheck -f gcc "$@" | bbresults
}

If you do not know what the .bash_profile or functions are, then you can read about them here:

Note: you cannot use shellcheck to check zsh scripts. As of now it only works for sh or bash scripts. You can tell shellcheck to ignore the #!/bin/zsh shebang and interpret the script as using bash syntax, which should work in most cases, but will not always have good results.

$ shellcheck -s bash script.zsh

Moving to zsh, part 5: Completions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class in Amsterdam on September 6! Visit our website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

What are Completions?

Man shells use the tab key (⇥) for completion. When you press that key, the shell tries to guess what you are typing and will complete it, or if the beginning of what you typed is ambiguous, suggest from a list of possible completions.

For example when you want to cd to your Documents folder, you can save typing:

% cd ~/Doc⇥
% cd ~/Documents/

When you hit the tab key, the system will complete the path to the Documents folder.

When the completion is ambiguous, the shell will list possible completions:

% cd ~/D⇥
Desktop/    Documents/  Downloads/

At this point, you can add a character or two to get to a unique completion, and hit the tab key again. In zsh you can also hit the tab key repeatedly to cycle through the suggested completions. In this example, the first tab keystroke will show the list, the second will complete ~/Desktop/, the third completes ~/Documents, and so on.

You can use tab completion commands as well:

% system⇥
system_profiler     systemkeychain      systemsetup         systemsoundserverd  systemstats
% system_⇥
% system_profiler 

Not having to type path and file names saves time and avoids errors, especially with complex paths with spaces and other special characters:

% cd ~/Li⇥
% cd ~/Library/Appl⇥
% cd ~/Library/Application S⇥
Application Scripts/  Application Support/
% cd ~/Library/Application Su⇥
% cd ~/Library/Application Support/

Using tab completion is a huge productivity boost when using a shell.

Turning It On

In the default configuration, tab completion in zsh is very basic. It will complete commands and paths, but not much else. But you can enable a very powerful, and useful completion system.

zsh comes with a tool you can use to setup this completion system. When you run the compinstall command it will lead you through a complex and hard to understand list of menus which explains the options and will generate the code necessary to set this configuration up and add it to your .zshrc file or another configuration file of your choice.

Since the commands to configure the completion are quite arcane and hard to understand, this is a good way to get something to start out with. I will explain some of these options and commands in detail.

Whether you use compinstall or not, to turn on the more powerful completion system, you need to add at least this command to yourzsh` configuration file:

autoload -Uz compinit && compinit

This will initialize the zsh completion system. The details of this system are documented here.

If you want to configure the system, the configuration commands (usually zstyle commands) should be added to the zsh configuration file before you enable the system. (This only matters for a few configurations, but as a general rule it is safer.)

All of these completion rules need to be loaded and prepared. zsh’s completion system creates a cache in the file ~/.zcompdump. The first time you run compinit it might take a noticeable time, but subsequent runs should use this cache and be much faster.

Sometimes, especially when building and debugging your own completion files, you may need to delete this file to force a rebuild:

% rm -f ~/.zcompdump
% compinit

Case Insensitive Completion

Since the macOS file systems are usually case-insensitive, I prefer my tab-completion to be case-insensitive as well. For bash you configure that in the ~/.inputrc. In zsh you modify the completion systems behavior with this (monstrous) command:

# case insensitive path-completion
zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*'

I have seen many varieties for this configuration in different websites, but this is what compinstall adds when I select case-insensitive completion, so I am going with that.

Partial Completion

This is a particularly nice feature. You can type fragments of each path segment and the completion will try to complete them all at once:

% cd /u/lo/b⇥
% cd /usr/local/bin
% cd ~/L/P/B⇥
% ~/Library/Preferences/ByHost/

If the fragments are ambiguous, there are different strategies to what the completion system suggests. I have configured these like this:

# partial completion suggestions
zstyle ':completion:*' list-suffixes
zstyle ':completion:*' expand prefix suffix

Commands with built-in completion

zsh comes with several completion definitions for many commands. For example, when you type cp and then hit tab, the system will correctly assume you want to complete a file path and show the suggestions from the current working directory.

However, when you type cp -⇥ the completion can tell from the - that you want to add an option to the command and suggest a list of options for cp, with short descriptions.

% cp -⇥ 
-H  -- follow symlinks on the command line in recursive mode
-L  -- follow all symlinks in recursive mode
-P  -- do not follow symlinks in recursive mode (default)
-R  -- copy directories recursively
-X  -- don't copy extended attributes or resource forks
-a  -- archive mode, same as -RpP
-f  -- force overwriting existing file
-i  -- confirm before overwriting existing file
-n  -- don't overwrite existing file
-p  -- preserve timestamps, mode, owner, flags, ACLs, and extended attributes
-v  -- show file names as they are copied

As the context of command prompt you are assembling changes, you may get different completion suggestions. For example, the completion for ssh will suggest host names:

% ssh armin@⇥

zsh comes with completion definitions for many common commands. Nevertheless, it can be helpful to just hit tab, especially when wondering about options.

On macOS completions are stored in /usr/share/zsh/5.3/functions (replace the5.3 with 5.7.1 in Catalina). This directory stores many functions used with zsh and is in the default fpath. All the files in that directory that start with an underscore _ contain the completion definitions command. So, the file _cp contains the definition for the cp command. (Some of the definition files contain the definitions for multiple commands.)

Completions for macOS Commands

There are even a few macOS specific command that come with the default zsh installation.

% system_profiler ⇥⇥

macOS High Sierra and macOS Mojave come with zsh 5.3, which is now nearly two years old. zsh 5.3 contains less macOS specific completion definitions than the current zsh 5.7.1 which will is the pre-installed zsh in macOS Catalina. Some of the completions in 5.3 have also been updated in 5.7.1.

Tool zsh 5.3 zsh 5.7.1
caffeinate
defaults
fink
fs_usage
hdiutil
mdfind
mdls
mdutil
networksetup
nvram
open
osascript
otool
pbcopy/pbpaste
plutil
say
sc_usage
scselect
scutil
softwareupdate
sw_vers
swift
system_profiler
xcode_select

Load bash completions

Since the default shell on macOS has been bash for so long, there are quite a few bash completion definitions for macOS commands and third party tools available. For example Tony Williams’ bash completion for autopkg (post, Github).

You do not have to rewrite these completions, since the zsh completion system can use bash completion scripts as well: (add this to your zsh configuration file)

# load bashcompinit for some old bash completions
autoload bashcompinit && bashcompinit

[[ -r ~/Projects/autopkg_complete/autopkg ]] && source ~/Projects/autopkg_complete/autopkg

When you have multiple bash completion scripts you want to load, you only need to load bashcompinit once.

Build your own completions

Once you start using completions, you will want to have them everywhere. While many built-in completions exists, there are still many commands that lack a good definition.

Some commands, like the swift command line tool, have a built-in option to generate the completion syntax. You can then store that in a file and put it in your fpath:

% swift package completion-tool generate-zsh-script >_swift

Note: in the case of swift, its definition will conflict with the _openstack definition in zsh 5.3. You can fix this with the command compdef _swift swift after loading the completion system.

Some commands provide a list of options and arguments with the -h/--help option. If this list follows a certain syntax, you can get a decent completion working with

% compdef _gnu_generic <command>

One example on macOS, where this has decent results is the xed command which opens a file or folder in Xcode.

But for best results, you will often have to build the description yourself. Unfortunately this is not a simple task. The syntax is meticulously, but also quite abstractly documented in the zsh documentation for the Completion System. I also found the ‘howto’ documentation in the zsh-completions repository very useful, as well as the ‘zsh Completion Style Guide.’

To avoid everyone re-inventing the wheel, I have started a repository on Github for macOS specific completion files. The page has the instructions on how to install them and I will welcome pull requests with contributions. Since I am just starting to learn this as well, I am sure there are improvements that can be made on the completions I have built so far and there are several commands where you can test your skills and build a new one.

I suggest the #zsh channel on the MacAdmins Slack for discussion.

Next

In the next post in this series, we will discuss how to configure zsh’s command line prompt.

Weekly News Summary for Admins — 2019-07-05

For my US readers: belated Happy 4th of July!

This is where you celebrate Jeff Goldblum hacking an alien mainframe with a PowerBook 5300, right?

Because of the US holiday week-end and, I presume, several MacAdmins ‘finishing’ their PSU conference presentations, news was slow this week.

Safe travels and lots of fun, networking, and learning to those going to PSU.

Everyone else, I hope you get to enjoy a wonderful summer week-end!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

macOS Catalina

News and Opinion

MacAdmins on Twitter

  • The Macalope: “It’s going to be fun hearing the people who said design was dead at Apple bemoan the loss of the person chiefly responsible for designing all the things they said were so poorly designed.”

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Listen

Just for Fun

  • ‎Dice by PCalc (I am actually off to GM my campaign right after posting this newsletter!)

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Install Bash 5 on macOS with Patches

I recently posted an article on how to download, install, and build a macOS installer pkg for bash 5. In that first version of this post I ignored patches, minor updates to the bash source code and binary. But as the patches to bash 5 are accumulating, I cannot ignore them much longer.

This post will extend the instructions in the original post.

After downloading and expanding the bash-5.0.tar.gz, create a patches folder:

$ cd path/to/bash-5.0
$ mkdir patches
$ cd patches

You can download the patches for bash-5.0 here. As of this writing, there are seven patches for bash-5.0 labelled bash50-001 through bash50-007. You can download all at once with:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' -O

(Adapt the numbers when there are more patches in the future.)

Then move up one directory level to the bash-5.0 root directory and apply the patches using the patch command.

$ cd ..
$ patch -p0 -i patches/bash50-001
$ patch -p0 -i patches/bash50-002

(etc.)

You can download and patch with a single step. Make sure your working directory is the bash-5.0 with all the code and run:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' | patch -p0

From here, you can continue with the remaining build steps from the original post. The next step will be running ./configure.

The script to build the pkg installer has also been updated in the repository to download and apply the patches before building.

Moving to zsh, part 4: Aliases and Functions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

Aliases

Aliases in zsh work just like aliases in bash. You declare an alias with the alias (built-in) command and it will work as a text replacement at the beginning of the command prompt:

alias ll='ls -al'

You can just copy your alias declarations from your .bash_profile or .bashrc to your .zshrc. I had aliases for .. and cd.. which are now handled by Auto CD and shell correction respectively, so I didn’t bother to move those. (part 3: ‘Shell Options’)

After the alias is declared, you can use it at the beginning of a command. When you try to use the alias anywhere else in the command, the alias will not work:

% sudo ll
sudo: ll: command not found

Global Aliases

This is where zsh has an advantage. You can declare an alias as a ‘global’ alias, and then will be replaced anywhere in the command line:

% alias -g badge='tput bel'
% sudo badge        #<beeps> with privilege

Identifying Aliases

There is one more feature of zsh that is useful with aliases. The which command will show if a command stems from an alias substitution:

% which ll
ll: aliased to ls -l

However, when you try this with global aliases, the substitution occurs before the which command can evaluate the alias, which leads to an unexpected result:

% which badge
/usr/bin/tput
bel not found

You can suppress the alias substitution by escaping the first character or by quoting the entire alias name:

% which \badge
badge: globally aliased to tput bel
% which 'badge'
badge: globally aliased to tput bel

Functions

As with aliases, functions in your zsh configuration will work just as they did in bash.

function vnc() {
    open vnc://"$USER"@"$1"
}

This code in your zsh configuration file will define the vnc function and make it available in the shell.

Autoload Functions

However, zsh has some features which make using functions more flexible. There is (once again) a bit of configuration required to get this working.

Instead of declaring the function directly the configuration file, you can put the function in a separate file. zsh has a built-in variable called fpath which is an array of paths where zsh will look for files defining a function. You can add your own directory to this search path:

fpath+=~/Projects/dotfiles/zshfunctions

Just having a file in the directory is not enough. You still have to tell zsh that you want to use this particular function:

autoload vnc

This command tells zsh: “’Declare a function named vnc. To execute it, load a file named vnc, it is somewhere in the fpath.”

Note: you often see the -U or -Uz option added to the autoload command. These options help avoid conflicts with your personal settings. They suppress alias substitution and ksh-style loading of functions, respectively.

The vnc file in my zshfunctions directory can look like this:

# uses the arguments as hostnames for `open vnc://` (Screen Sharing)
# uses the $USER username as default account name

for x in $@; do 
    open vnc://"$USER"@"$x"
done

The vnc function will open a Screen Sharing session with the current user name pre-filled in.

Initializing Autoload Functions

You could also put the code in the function file into a function block:

function vnc() {
    for x in $@; do 
        open vnc://"$vnc_user"@"$x"
    done
}

# initialization code
vnc_user="remote_admin"
alias screen_sharing='vnc'

The function name should match the function name declared with autoload.

When you have additional code outside the function, the autoload behavior changes. When the function is called for the first time, the function will be defined and the code outside the function will be run. The function itself will not be executed on the first run. On subsequent calls, the function will be executed and the code outside the function is ignored.

You can use this to provide setup and initialization code for the function. You can even have more functions defined in the function file. The above example declares and sets a variable to use for account name and an alias for the vnc command.

Since you have to run the function once for the initialization, you often see this syntax in the zsh configuration file:

autoload vnc && vnc

Which means ‘declare the function and if that succeeds run it.’

In some functions, the initialization code will already launch the function itself:

function vnc() {
    ...
}
# initialization
vnc_user="remote_admin"
vnc()

Since the behavior will vary from each autoloaded function to the next, be sure to study any documentation or the function’s code.

Identifying Functions

Finally, the which command will show the function code:

 % which vnc
vnc () {
    for x in $@
    do
        open vnc://"$USER"@"$x"
    done
}

The functions command without any parameters, will print all functions (there will be a lot of them). Use functions + to just list the function names.

Debugging Functions

When you are working on complex autoloaded functions, you will at some point have to do some debugging. You can enable tracing for functions with

% functions -t vnc
% vnc Client.local
+vnc:1> x=Client.local
+vnc:2> open vnc://armin@Client.local

You can disable tracing for this function with functions +t vnc.

Next

In the next part we will enable, use and configure tab completions.

Weekly News Summary for Admins — 2019-06-28

As expected, the public betas of iOS 13, iPadOS 13 and macOS 10.15 Catalina were released this week, keeping MacAdmins busy with both preventing users from installing them and starting their own testing procedures.

Together with the public betas, we get a bunch of early Catalina reviews and opinions.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Headlines

On Scripting OS X

News and Opinion

macOS Catalina and iOS 13

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh, part 3: Shell Options

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

Now that we have chosen a file to configure our zsh, we need to decide on ‘what’ to configure and ‘how.’ In this post, I want to talk about zsh’s shell options.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

In the previous post, I listed some features that I would like to transfer from my bash configuration. While researching how to implement these options in zsh, I found a few, new and interesting options in zsh.

The settings from bash which I want in zsh were:

  • case-insensitive globbing
  • command history, shared across windows and sessions

Note: bash in this series of posts specifically refers to the version of bash that comes with macOS as /bin/bash (v3.2.57).

Note 2: Mono-typed lines starting with a % show commands and results from zsh. Mono-typed lines starting with $ show commands and results in bash

What are Shell Options?

Shell options are preferences for the shell’s behavior. You are using shell options in bash, when you enable ‘trace mode’ for scripts with the set -x command or the bash -x option. (Note: this also works with zsh scripts.)

zsh has a lot of shell options. Many of these options serve the purpose of enabling (or disabling) compatibility with other shells. There are also many options which are specific to zsh.

You can set an option with the setopt command. For compatibility with other shells the setopt command and set -o have the same effect (set an option by name). The following commands set the same option:

set -o AUTO_CD
setopt AUTO_CD

The names or labels of the options are commonly written in all capitals in the documentation but in lowercase when listed with the setopt tool. The labels of the options are case insensitive and any underscores in the label are ignored. So, these commands set the same option:

setopt AUTO_CD
setopt autocd
setopt auto_cd
setopt autoCD

There are quite a few ways to negate or unset an option. First you can use unsetopt or set +o. Alternatively, you can prefix with NO or no to negate an option. The following commands all have the same effect of turning off the previously set option AUTO_CD

unsetopt AUTO_CD
set +o AUTO_CD
unsetopt autocd
setopt NO_AUTO_CD
setopt noautocd

Any options you change will only take effect in the current instance of zsh. When you want to change the settings for all new shells, you have to put the commands in one of the configuration files (usually .zshrc).

Showing the current Options

You can list the existing shell options with the setopt command:

% setopt
combiningchars
interactive
login
monitor
shinstdin
zle

This list only shows options are changed from the default set of options for zsh. These options are marked with <D> (default for all shell emulations) or <Z> (default for zsh) in the documentation or the zshoptions man page.

You can also get a list of all default zsh options with the command:

% emulate -lLR zsh

Some zsh Options I use

As I have mentioned before in my posts on bash configuration, I prefer minimal configuration changes, so I do not feel all awkward and lost when I have to work on an ‘un-configured’ Mac.

These configurations are a personal choice and you should pick and choose your own. You can find a full list of zsh options in the zsh Manual or with man zshoptions.

On the other hand, exploring the options allows us to explore a few useful zsh features.

Case Insensitive Globbing

Note: ‘Globbing’ is a unix/shell term that refers to the expansion of wildcard characters, such as * and ? into full file paths and names. I.e. ~/D* is expanded into /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Since the file system on macOS is (usually) case-insensitive, I prefer globbing and tab-completion to be case-insensitive as well.

The zsh option which controls this is CASE_GLOB. Since we want globbing to be case-insensitive, we want to turn the option off, so:

setopt NO_CASE_GLOB

You can test this in the shell:

% ls ~/d*<tab>

In zsh tab completion will replace the wildcard with the actual result. So after the tab you will see:

% ls /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Using tab completion this way to see and possibly edit the actual replacement for wildcards is a useful safety net.

In bash hit the tab key will list possible completions, but not substitute them in the command prompt.

If you do not like this behavior in zsh then you can change to behavior similar to bash with:

setopt GLOB_COMPLETE

Automatic CD

Sometimes you enter the path to a directory, but forget the leading cd:

$ Library/Preferences/
bash: Library/Preferences/: is a directory

% Library/Preferences
zsh: permission denied: Library/Preferences

With AUTO_CD enabled in zsh, the shell will automatically change directory:

% Library/Preferences
% pwd
/Users/armin/Library/Preferences

This works with relative and absolute paths, including the ..:

% ..
% pwd
/Users/armin/Library
% ../Desktop 
% pwd
/Users/armin/Desktop

I have an alias in my .bash_profile that sets the .. command to cd ... Auto CD replaces that functionality and more.

Enable Auto CD with:

setopt AUTO_CD

Shell History

Shells commonly remember previously executed commands and allows you to recall them with the up and down arrow keys, search or special history commands.

Most of those keys work the same in zsh. However, there are a few things you need to configure for zsh history to work as you are used to with bash on macOS.

By default, zsh does not save its history when the shell exits. The history is ‘forgotten’ when you close a Terminal window or tab. To make zsh save its history to a file when it exits, you need to set a variable in the shell:

HISTFILE=${ZDOTDIR:-$HOME}/.zsh_history

Note: this is not a shell option but shell variable or parameter. I will cover some more of those later, You can find a list of variables used by zsh in the documentation.

The HISTFILE variable tells zsh where to store the history data. The syntax ${ZDOTDIR:-$HOME} means it will use the value of ZDOTDIR when it is set or default to the value of HOME otherwise. When a user has set the ZDOTDIR variable to group their configurations files in a specific directory, the history will be stored there as well.

By default zsh simply writes each command in its own line in the history file. You can view the file’s contents with any text editor or list the last few commands:

% tail -n 10 ~/.zsh_history

You can make zsh add a bit more data (timestamp in unix epoch time and elapsed time of the command) by setting the EXTENDED_HISTORY shell option.

setopt EXTENDED_HISTORY

You can set limits on how many commands the shell should remember in the session and in the history file with the HISTSIZE and SAVEHIST variables:

SAVEHIST=5000
HISTSIZE=2000

When the shell reaches this limit the oldest commands will be removed from memory or the history file.

By default, when you exit zsh (for example, by closing the window or tab) this particular instance of zsh will overwrite an existing history file with its history. So when you have multiple Terminal windows or tabs open, they will all overwrite each others’ histories eventually.

You can tell zsh to use a single, shared history file across the sessions and append to it rather than overwrite:

# share history across multiple zsh sessions
setopt SHARE_HISTORY
# append to history
setopt APPEND_HISTORY

Furthermore, you can tell zsh to update the history file after every command, rather than waiting for the shell to exit:

# adds commands as they are typed, not at shell exit
setopt INC_APPEND_HISTORY

When you use a shared history file, it will grow very quickly, and you may want to use some options to clean out duplicates and blanks:

# expire duplicates first
setopt HIST_EXPIRE_DUPS_FIRST 
# do not store duplications
setopt HIST_IGNORE_DUPS
#ignore duplicates when searching
setopt HIST_FIND_NO_DUPS
# removes blank lines from history
setopt HIST_REDUCE_BLANKS

(some of these are redundant)

Most of the time you will access the history with the up arrow key to recall the last command, or maybe a few more steps. You can search through the history with ctrl-R

In zsh, you can also use the !! history substitution, which will be replaced with the entire last command. This is most commonly used in combination with sudo:

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

By default, the shell will show the command it is substituting before it is run. But at that point, it is too late to make any changes. When you set the HIST_VERIFY option, zsh will show the substituted command in the prompt instead, giving you a chance to edit or cancel it, or just confirm it.

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
% sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

This works for other history substitutions such as !$ or !*, as well. You can find all of zsh’s history expansions in the documentation.

Correction

When you mistype a command or path, the shell is usually unforgiving. In zsh you can enable correction. Then, the shell will make a guess of what you meant to type and ask whether you want do that instead:

% systemprofiler 
zsh: correct 'systemprofiler' to 'system_profiler' [nyae]?

Your options are to

  • n: execute as typed
  • y: accept and execute the suggested correction
  • a: abort and do nothing
  • e: return to the prompt to continue editing

I have found this far less annoying and far more useful than I expected. Especially, since it works together with AUTO_CD:

% Dekstop
zsh: correct 'Dekstop' to 'Desktop' [nyae]?

You enable zsh correction with these options:

setopt CORRECT
setopt CORRECT_ALL

Reverting to defaults

Most of the changes mentioned here affect the interactive shell and will have little impact on zsh scripts. However, there are some options that do affect the behavior of things like variable substitutions which will affect scripts.

You can revert the options for the current shell to the default settings with the following command:

emulate -LR zsh

We encountered this command earlier when we listed the default settings. The -l option will list the settings rather than apply them.

If in doubt, it may be useful to add this at the beginning of your zsh scripts.

Next

In the next part we will take a look at aliases and functions.

Weekly News Summary for Admins — 2019-06-21

Happy Solstice, everyone!

Things are calming down after the WWDC storm. This week we got some more WWDC reactions, a major Firefox exploit with fix, and some more information on Catalina notarization.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • Victor (groob): “Working on the next release of MicroMDM and added an important documentation section I felt was missing for too long. MicroMDM is not a product! Link

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Watch

To Listen

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh, part 2: Configuration Files

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

In part one I talked about Apple’s motivation to switch the default shell and urge existing users to change to zsh.

Since I am new to zsh as well, I am planning to document my process of transferring my personal bash setup and learning the odds and ends of zsh.

Many websites and tutorials leap straight to projects like oh-my-zsh or prezto where you can choose from hundreds of pre-customized and pre-configured themes.

While these projects are very impressive and certainly show off the flexibility and power of zsh customization, I feel this will actually prevent an understanding of how zsh works and how it differs from bash. So, I am planning to build my own configuration ‘by hand’ first.

At first, I actually took a look at my current bash_profile and cleaned it up. There were many aliases and functions which I do not use or broke in some macOS update. I the end, this is what I want to re-create in zsh:

Most of these should be fairly easy to transfer. Some might be… interesting.

But first, where do we put our custom zsh configuration?

zsh Configuration Files

bash has a list of possible files that it tries in predefined order. I have the description in my post on the bash_profile.

zsh also has a list of files it will execute at shell startup. The list of possible files is even longer, but somewhat more ordered.

all users user login shell interactive shell scripts Terminal.app
/etc/zshenv .zshenv
/etc/zprofile .zprofile x x
/etc/zshrc .zshrc x
/etc/zlogin .zlogin x x
/etc/zlogout .zlogout x x

The files in /etc/ will be launched (when present) for all users. The .z* files only for the individual user.

By default, zsh will look in the root of the home directory for the user .z* files, but this behavior can be changed by setting the ZDOTDIR environment variable to another directory (e.g. ~/.zsh/) where you can then group all user zsh configuration in one place.

On macOS you could set the ZDOTDIR to ~/Documents/zsh/ and then use iCloud syncing (or a different file sync service) to have the same files on all your Macs. (I prefer to use git.)

bash will either use .bash_profile for login shells, or .bashrc for interactive shells. That means, when you want to centralize configuration for all use cases, you need to source your .bashrc from .bash_profile or vice versa.

zsh behaves differently. zsh will run all of these files in the appropriate context (login shell, interactive shell) when they exist.

zsh will start with /etc/zshenv, then the user’s .zshenv. The zshenv files are always used when they exist, even for scripts with the #!/bin/zsh shebang. Since changes applied in the zshenv will affect zsh behavior in all contexts, you should you should be very cautious about changes applied here.

Next, when the shell is a login shell, zsh will run /etc/zprofile and .zprofile. Then for interactive shells (and login shells) /etc/zshrc and .zshrc. Then, again, for login shells /etc/zlogin and .zlogin. Why are there two files for login shells? The zprofile exists as an analog for bash’s and sh’s profile files, and zlogin as an analog for ksh login files.

Finally, there are zlogout files that can be used for cleanup, when a login shell exits. In this case, the user level .zlogout is read first, then the central /etc/zlogout. If the shell is terminated by an external process, these files might not be run.

Apple Provided Configuration Files

macOS Mojave (and earlier versions) includes /etc/zprofile and /etc/zshrc files. Both are very basic.

/etc/zprofile uses /usr/libexec/path_helper to set the default PATH. Then /etc/zshrc enables UTF–8 with setopt combiningchars.

Like /etc/bashrc there is a line in /etc/zshrc that would load /etc/zshrc_Apple_Terminal if it existed. This is interesting as /etc/bashrc_Apple_Terminal contains quite a lot of code to help bash to communicate with the Terminal application. In particular bash will send a signal to the Terminal on every new prompt to update the path and icon displayed in the Terminal window title bar, and provides other code relevant for saving and restoring Terminal sessions between application restarts.

However, there is no /etc/zshrc_Apple_Terminal and we will have to provide some of this functionality ourselves.

Note: As of this writing, /etc/zshrc in the macOS Catalina beta is different from the Mojave /etc/zshrc and provides more configuration. However, since Catalina is still beta, I will focus these articles on Mojave and earlier. Once Catalina is released, I may update these articles or write a new one for Catalina, if necessary.

Which File to use?

When you want to use the ZDOTDIR variable to change the location of the other zsh configuration files, setting that variable in ~/.zshenv seems like a good choice. Other than that, you probably want to avoid using the zshenv files, since it will change settings for all invocations of zsh, including scripts.

macOS Terminal considers every new shell to be a login shell and an interactive shell. So, in Terminal a new zsh will potentially run all configuration files.

For simplicity’s sake, you should use just one file. The common choice is .zshrc.

Most tools you can download to configure zsh, such as ‘prezto’ or ‘oh-my-zsh’, will override or re-configure your .zshrc. You could consider moving your code to .zlogin instead. Since .zlogin is sourced after .zshrc it can override settings from .zshrc. However, .zlogin is only called for login shells.

The most common situation where you do not get a login shell with macOS Terminal, is when you switch to zsh from another shell by typing the zsh command.

I would recommend to put your configuration in your .zshrc file and if you want to use any of the theme projects, read and follow their instructions closely as to how you can preserve your configurations together with theirs.

Managing the shell for Administrators

MacAdmins may have the need to manage certain shell settings for their users, usually environment variables to configure certain command line tool’s behaviors.

The most common need is to expand the PATH environment variable for third party tools. Often the third party tools in question will have elaborate postinstall scripts that attempt to modify the current user’s .bash_profile or .bashrc. Sometimes, these tools even consider that a user might have changed the default shell to something other than bash.

On macOS, system wide changes to the PATH should be done by adding files to /etc/paths.d.

As an administrator you should be on the lookout for scripts and installers that attempt to modify configuration files on the user level, disable the scripts during deployment, and manage the required changes centrally. This will allow you to keep control of the settings even as tools change, are added or removed from the system, while preserving the user’s custom configurations.

To manage environment variables other than PATH centrally, administrators should consider /etc/zshenv or adding to the existing /etc/zshrc. In these cases you should always monitor whether updates to macOS overwrite or change these files with new, modified files of their own.

Summary

There are many possible files where the zsh can load user configuration. You should use ~/zshrc for your personal configurations.

There are many tools and projects out there that will configure zsh for you. This is fine, but might keep you from really understanding how things work.

MacAdmins who need to manage these settings centrally, should use /etc/paths.d and similar technologies or consider /etc/zshenv or /etc/zshrc.

Apple’s built-in support for zsh in Terminal is not as detailed as it is for bash.

Next: Part 3 – Shell Options

Weekly News Summary for Admins — 2019-06-14

First week post-WWDC and people are slowly catching up with the firehose of information. Lot’s of opinion pieces and podcasts to catch up with.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • macshome: “With so many people running macOS in a VM right now here is a ProTip for Fusion on a TouchID Mac. Set: board-id.reflectHost = "FALSE" Now your auth dialog spins are gone!”
  • Craig Hockenberry: “I know a lot of developers who have been working with Apple’s products for decades. The overwhelming consensus is that we’re seeing something that will change our lives for decades to come. 1976 -> 1984 -> 1996 -> 2008 -> 2019”
  • Daniel Jalkut: “Apple has been doing hardware penance lately. I hope they realize that exiting the personal networking market (Airport) was as as misguided as exiting pro Mac, pro displays, etc. We’re ready for the big comeback.”
  • Derek Fulmer: “With the impending changes to macOS in 10.15, I’m giving zsh a go. Really digging its customizability. Feels way more modern. But, I’m still sentimental about bash.”

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

  • Erik Schwiebert: “Microsoft Office, macOS 10.15 Catalina, and You: @mrexchange explains our support plans for the beta and final OS release.”

To Listen

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!