Moving to zsh, part 4: Aliases and Functions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

Aliases

Aliases in zsh work just like aliases in bash. You declare an alias with the alias (built-in) command and it will work as a text replacement at the beginning of the command prompt:

alias ll='ls -al'

You can just copy your alias declarations from your .bash_profile or .bashrc to your .zshrc. I had aliases for .. and cd.. which are now handled by Auto CD and shell correction respectively, so I didn’t bother to move those. (part 3: ‘Shell Options’)

After the alias is declared, you can use it at the beginning of a command. When you try to use the alias anywhere else in the command, the alias will not work:

% sudo ll
sudo: ll: command not found

Global Aliases

This is where zsh has an advantage. You can declare an alias as a ‘global’ alias, and then will be replaced anywhere in the command line:

% alias -g badge='tput bel'
% sudo badge        #<beeps> with privilege

Identifying Aliases

There is one more feature of zsh that is useful with aliases. The which command will show if a command stems from an alias substitution:

% which ll
ll: aliased to ls -l

However, when you try this with global aliases, the substitution occurs before the which command can evaluate the alias, which leads to an unexpected result:

% which badge
/usr/bin/tput
bel not found

You can suppress the alias substitution by escaping the first character or by quoting the entire alias name:

% which \badge
badge: globally aliased to tput bel
% which 'badge'
badge: globally aliased to tput bel

Functions

As with aliases, functions in your zsh configuration will work just as they did in bash.

function vnc() {
    open vnc://"$USER"@"$1"
}

This code in your zsh configuration file will define the vnc function and make it available in the shell.

Autoload Functions

However, zsh has some features which make using functions more flexible. There is (once again) a bit of configuration required to get this working.

Instead of declaring the function directly the configuration file, you can put the function in a separate file. zsh has a built-in variable called fpath which is an array of paths where zsh will look for files defining a function. You can add your own directory to this search path:

fpath+=~/Projects/dotfiles/zshfunctions

Just having a file in the directory is not enough. You still have to tell zsh that you want to use this particular function:

autoload vnc

This command tells zsh: “’Declare a function named vnc. To execute it, load a file named vnc, it is somewhere in the fpath.”

Note: you often see the -U or -Uz option added to the autoload command. These options help avoid conflicts with your personal settings. They suppress alias substitution and ksh-style loading of functions, respectively.

The vnc file in my zshfunctions directory can look like this:

# uses the arguments as hostnames for `open vnc://` (Screen Sharing)
# uses the $USER username as default account name

for x in $@; do 
    open vnc://"$USER"@"$x"
done

The vnc function will open a Screen Sharing session with the current user name pre-filled in.

Initializing Autoload Functions

You could also put the code in the function file into a function block:

function vnc {
    for x in $@; do 
        open vnc://"$vnc_user"@"$x"
    done
}

# initialization code
vnc_user="remote_admin"
alias screen_sharing='vnc'

The function name should match the function name declared with autoload.

When you have additional code outside the function, the autoload behavior changes. When the function is called for the first time, the function will be defined and the code outside the function will be run. The function itself will not be executed on the first run. On subsequent calls, the function will be executed and the code outside the function is ignored.

You can use this to provide setup and initialization code for the function. You can even have more functions defined in the function file. The above example declares and sets a variable to use for account name and an alias for the vnc command.

Since you have to run the function once for the initialization, you often see this syntax in the zsh configuration file:

autoload vnc && vnc

Which means ‘declare the function and if that succeeds run it.’

In some functions, the initialization code will already launch the function itself:

function vnc {
    ...
}
# initialization
vnc_user="remote_admin"
vnc()

Since the behavior will vary from each autoloaded function to the next, be sure to study any documentation or the function’s code.

Identifying Functions

Finally, the which command will show the function code:

 % which vnc
vnc () {
    for x in $@
    do
        open vnc://"$USER"@"$x"
    done
}

The functions command without any parameters, will print all functions (there will be a lot of them). Use functions + to just list the function names.

Debugging Functions

When you are working on complex autoloaded functions, you will at some point have to do some debugging. You can enable tracing for functions with

% functions -t vnc
% vnc Client.local
+vnc:1> x=Client.local
+vnc:2> open vnc://armin@Client.local

You can disable tracing for this function with functions +t vnc.

Next

In the next part we will enable, use and configure tab completions.

Weekly News Summary for Admins — 2019-06-28

As expected, the public betas of iOS 13, iPadOS 13 and macOS 10.15 Catalina were released this week, keeping MacAdmins busy with both preventing users from installing them and starting their own testing procedures.

Together with the public betas, we get a bunch of early Catalina reviews and opinions.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Headlines

On Scripting OS X

News and Opinion

macOS Catalina and iOS 13

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh, part 3: Shell Options

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

Now that we have chosen a file to configure our zsh, we need to decide on ‘what’ to configure and ‘how.’ In this post, I want to talk about zsh’s shell options.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

In the previous post, I listed some features that I would like to transfer from my bash configuration. While researching how to implement these options in zsh, I found a few, new and interesting options in zsh.

The settings from bash which I want in zsh were:

  • case-insensitive globbing
  • command history, shared across windows and sessions

Note: bash in this series of posts specifically refers to the version of bash that comes with macOS as /bin/bash (v3.2.57).

Note 2: Mono-typed lines starting with a % show commands and results from zsh. Mono-typed lines starting with $ show commands and results in bash

What are Shell Options?

Shell options are preferences for the shell’s behavior. You are using shell options in bash, when you enable ‘trace mode’ for scripts with the set -x command or the bash -x option. (Note: this also works with zsh scripts.)

zsh has a lot of shell options. Many of these options serve the purpose of enabling (or disabling) compatibility with other shells. There are also many options which are specific to zsh.

You can set an option with the setopt command. For compatibility with other shells the setopt command and set -o have the same effect (set an option by name). The following commands set the same option:

set -o AUTO_CD
setopt AUTO_CD

The names or labels of the options are commonly written in all capitals in the documentation but in lowercase when listed with the setopt tool. The labels of the options are case insensitive and any underscores in the label are ignored. So, these commands set the same option:

setopt AUTO_CD
setopt autocd
setopt auto_cd
setopt autoCD

There are quite a few ways to negate or unset an option. First you can use unsetopt or set +o. Alternatively, you can prefix with NO or no to negate an option. The following commands all have the same effect of turning off the previously set option AUTO_CD

unsetopt AUTO_CD
set +o AUTO_CD
unsetopt autocd
setopt NO_AUTO_CD
setopt noautocd

Any options you change will only take effect in the current instance of zsh. When you want to change the settings for all new shells, you have to put the commands in one of the configuration files (usually .zshrc).

Showing the current Options

You can list the existing shell options with the setopt command:

% setopt
combiningchars
interactive
login
monitor
shinstdin
zle

This list only shows options are changed from the default set of options for zsh. These options are marked with <D> (default for all shell emulations) or <Z> (default for zsh) in the documentation or the zshoptions man page.

You can also get a list of all default zsh options with the command:

% emulate -lLR zsh

Some zsh Options I use

As I have mentioned before in my posts on bash configuration, I prefer minimal configuration changes, so I do not feel all awkward and lost when I have to work on an ‘un-configured’ Mac.

These configurations are a personal choice and you should pick and choose your own. You can find a full list of zsh options in the zsh Manual or with man zshoptions.

On the other hand, exploring the options allows us to explore a few useful zsh features.

Case Insensitive Globbing

Note: ‘Globbing’ is a unix/shell term that refers to the expansion of wildcard characters, such as * and ? into full file paths and names. I.e. ~/D* is expanded into /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Since the file system on macOS is (usually) case-insensitive, I prefer globbing and tab-completion to be case-insensitive as well.

The zsh option which controls this is CASE_GLOB. Since we want globbing to be case-insensitive, we want to turn the option off, so:

setopt NO_CASE_GLOB

You can test this in the shell:

% ls ~/d*<tab>

In zsh tab completion will replace the wildcard with the actual result. So after the tab you will see:

% ls /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Using tab completion this way to see and possibly edit the actual replacement for wildcards is a useful safety net.

In bash hit the tab key will list possible completions, but not substitute them in the command prompt.

If you do not like this behavior in zsh then you can change to behavior similar to bash with:

setopt GLOB_COMPLETE

Automatic CD

Sometimes you enter the path to a directory, but forget the leading cd:

$ Library/Preferences/
bash: Library/Preferences/: is a directory

% Library/Preferences
zsh: permission denied: Library/Preferences

With AUTO_CD enabled in zsh, the shell will automatically change directory:

% Library/Preferences
% pwd
/Users/armin/Library/Preferences

This works with relative and absolute paths, including the ..:

% ..
% pwd
/Users/armin/Library
% ../Desktop 
% pwd
/Users/armin/Desktop

I have an alias in my .bash_profile that sets the .. command to cd ... Auto CD replaces that functionality and more.

Enable Auto CD with:

setopt AUTO_CD

Shell History

Shells commonly remember previously executed commands and allows you to recall them with the up and down arrow keys, search or special history commands.

Most of those keys work the same in zsh. However, there are a few things you need to configure for zsh history to work as you are used to with bash on macOS.

By default, zsh does not save its history when the shell exits. The history is ‘forgotten’ when you close a Terminal window or tab. To make zsh save its history to a file when it exits, you need to set a variable in the shell:

HISTFILE=${ZDOTDIR:-$HOME}/.zsh_history

Note: this is not a shell option but shell variable or parameter. I will cover some more of those later, You can find a list of variables used by zsh in the documentation.

The HISTFILE variable tells zsh where to store the history data. The syntax ${ZDOTDIR:-$HOME} means it will use the value of ZDOTDIR when it is set or default to the value of HOME otherwise. When a user has set the ZDOTDIR variable to group their configurations files in a specific directory, the history will be stored there as well.

By default zsh simply writes each command in its own line in the history file. You can view the file’s contents with any text editor or list the last few commands:

% tail -n 10 ~/.zsh_history

You can make zsh add a bit more data (timestamp in unix epoch time and elapsed time of the command) by setting the EXTENDED_HISTORY shell option.

setopt EXTENDED_HISTORY

You can set limits on how many commands the shell should remember in the session and in the history file with the HISTSIZE and SAVEHIST variables:

SAVEHIST=5000
HISTSIZE=2000

When the shell reaches this limit the oldest commands will be removed from memory or the history file.

By default, when you exit zsh (for example, by closing the window or tab) this particular instance of zsh will overwrite an existing history file with its history. So when you have multiple Terminal windows or tabs open, they will all overwrite each others’ histories eventually.

You can tell zsh to use a single, shared history file across the sessions and append to it rather than overwrite:

# share history across multiple zsh sessions
setopt SHARE_HISTORY
# append to history
setopt APPEND_HISTORY

Furthermore, you can tell zsh to update the history file after every command, rather than waiting for the shell to exit:

# adds commands as they are typed, not at shell exit
setopt INC_APPEND_HISTORY

When you use a shared history file, it will grow very quickly, and you may want to use some options to clean out duplicates and blanks:

# expire duplicates first
setopt HIST_EXPIRE_DUPS_FIRST 
# do not store duplications
setopt HIST_IGNORE_DUPS
#ignore duplicates when searching
setopt HIST_FIND_NO_DUPS
# removes blank lines from history
setopt HIST_REDUCE_BLANKS

(some of these are redundant)

Most of the time you will access the history with the up arrow key to recall the last command, or maybe a few more steps. You can search through the history with ctrl-R

In zsh, you can also use the !! history substitution, which will be replaced with the entire last command. This is most commonly used in combination with sudo:

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

By default, the shell will show the command it is substituting before it is run. But at that point, it is too late to make any changes. When you set the HIST_VERIFY option, zsh will show the substituted command in the prompt instead, giving you a chance to edit or cancel it, or just confirm it.

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
% sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

This works for other history substitutions such as !$ or !*, as well. You can find all of zsh’s history expansions in the documentation.

Correction

When you mistype a command or path, the shell is usually unforgiving. In zsh you can enable correction. Then, the shell will make a guess of what you meant to type and ask whether you want do that instead:

% systemprofiler 
zsh: correct 'systemprofiler' to 'system_profiler' [nyae]?

Your options are to

  • n: execute as typed
  • y: accept and execute the suggested correction
  • a: abort and do nothing
  • e: return to the prompt to continue editing

I have found this far less annoying and far more useful than I expected. Especially, since it works together with AUTO_CD:

% Dekstop
zsh: correct 'Dekstop' to 'Desktop' [nyae]?

You enable zsh correction with these options:

setopt CORRECT
setopt CORRECT_ALL

Reverting to defaults

Most of the changes mentioned here affect the interactive shell and will have little impact on zsh scripts. However, there are some options that do affect the behavior of things like variable substitutions which will affect scripts.

You can revert the options for the current shell to the default settings with the following command:

emulate -LR zsh

We encountered this command earlier when we listed the default settings. The -l option will list the settings rather than apply them.

If in doubt, it may be useful to add this at the beginning of your zsh scripts.

Next

In the next part we will take a look at aliases and functions.

Weekly News Summary for Admins — 2019-06-21

Happy Solstice, everyone!

Things are calming down after the WWDC storm. This week we got some more WWDC reactions, a major Firefox exploit with fix, and some more information on Catalina notarization.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • Victor (groob): “Working on the next release of MicroMDM and added an important documentation section I felt was missing for too long. MicroMDM is not a product! Link

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Watch

To Listen

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh, part 2: Configuration Files

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

In part one I talked about Apple’s motivation to switch the default shell and urge existing users to change to zsh.

Since I am new to zsh as well, I am planning to document my process of transferring my personal bash setup and learning the odds and ends of zsh.

Many websites and tutorials leap straight to projects like oh-my-zsh or prezto where you can choose from hundreds of pre-customized and pre-configured themes.

While these projects are very impressive and certainly show off the flexibility and power of zsh customization, I feel this will actually prevent an understanding of how zsh works and how it differs from bash. So, I am planning to build my own configuration ‘by hand’ first.

At first, I actually took a look at my current bash_profile and cleaned it up. There were many aliases and functions which I do not use or broke in some macOS update. I the end, this is what I want to re-create in zsh:

Most of these should be fairly easy to transfer. Some might be… interesting.

But first, where do we put our custom zsh configuration?

zsh Configuration Files

bash has a list of possible files that it tries in predefined order. I have the description in my post on the bash_profile.

zsh also has a list of files it will execute at shell startup. The list of possible files is even longer, but somewhat more ordered.

all users user login shell interactive shell scripts Terminal.app
/etc/zshenv .zshenv
/etc/zprofile .zprofile x x
/etc/zshrc .zshrc x x
/etc/zlogin .zlogin x x
/etc/zlogout .zlogout x x

The files in /etc/ will be launched (when present) for all users. The .z* files only for the individual user.

By default, zsh will look in the root of the home directory for the user .z* files, but this behavior can be changed by setting the ZDOTDIR environment variable to another directory (e.g. ~/.zsh/) where you can then group all user zsh configuration in one place.

On macOS you could set the ZDOTDIR to ~/Documents/zsh/ and then use iCloud syncing (or a different file sync service) to have the same files on all your Macs. (I prefer to use git.)

bash will either use .bash_profile for login shells, or .bashrc for interactive shells. That means, when you want to centralize configuration for all use cases, you need to source your .bashrc from .bash_profile or vice versa.

zsh behaves differently. zsh will run all of these files in the appropriate context (login shell, interactive shell) when they exist.

zsh will start with /etc/zshenv, then the user’s .zshenv. The zshenv files are always used when they exist, even for scripts with the #!/bin/zsh shebang. Since changes applied in the zshenv will affect zsh behavior in all contexts, you should you should be very cautious about changes applied here.

Next, when the shell is a login shell, zsh will run /etc/zprofile and .zprofile. Then for interactive shells /etc/zshrc and .zshrc. Then, again, for login shells /etc/zlogin and .zlogin. Why are there two files for login shells? The zprofile exists as an analog for bash’s and sh’s profile files, and zlogin as an analog for ksh login files.

Finally, there are zlogout files that can be used for cleanup, when a login shell exits. In this case, the user level .zlogout is read first, then the central /etc/zlogout. If the shell is terminated by an external process, these files might not be run.

Apple Provided Configuration Files

macOS Mojave (and earlier versions) includes /etc/zprofile and /etc/zshrc files. Both are very basic.

/etc/zprofile uses /usr/libexec/path_helper to set the default PATH. Then /etc/zshrc enables UTF–8 with setopt combiningchars.

Like /etc/bashrc there is a line in /etc/zshrc that would load /etc/zshrc_Apple_Terminal if it existed. This is interesting as /etc/bashrc_Apple_Terminal contains quite a lot of code to help bash to communicate with the Terminal application. In particular bash will send a signal to the Terminal on every new prompt to update the path and icon displayed in the Terminal window title bar, and provides other code relevant for saving and restoring Terminal sessions between application restarts.

However, there is no /etc/zshrc_Apple_Terminal and we will have to provide some of this functionality ourselves.

Note: As of this writing, /etc/zshrc in the macOS Catalina beta is different from the Mojave /etc/zshrc and provides more configuration. However, since Catalina is still beta, I will focus these articles on Mojave and earlier. Once Catalina is released, I may update these articles or write a new one for Catalina, if necessary.

Which File to use?

When you want to use the ZDOTDIR variable to change the location of the other zsh configuration files, setting that variable in ~/.zshenv seems like a good choice. Other than that, you probably want to avoid using the zshenv files, since it will change settings for all invocations of zsh, including scripts.

macOS Terminal considers every new shell to be a login shell and an interactive shell. So, in Terminal a new zsh will potentially run all configuration files.

For simplicity’s sake, you should use just one file. The common choice is .zshrc.

Most tools you can download to configure zsh, such as ‘prezto’ or ‘oh-my-zsh’, will override or re-configure your .zshrc. You could consider moving your code to .zlogin instead. Since .zlogin is sourced after .zshrc it can override settings from .zshrc. However, .zlogin is only called for login shells.

The most common situation where you do not get a login shell with macOS Terminal, is when you switch to zsh from another shell by typing the zsh command.

I would recommend to put your configuration in your .zshrc file and if you want to use any of the theme projects, read and follow their instructions closely as to how you can preserve your configurations together with theirs.

Managing the shell for Administrators

MacAdmins may have the need to manage certain shell settings for their users, usually environment variables to configure certain command line tool’s behaviors.

The most common need is to expand the PATH environment variable for third party tools. Often the third party tools in question will have elaborate postinstall scripts that attempt to modify the current user’s .bash_profile or .bashrc. Sometimes, these tools even consider that a user might have changed the default shell to something other than bash.

On macOS, system wide changes to the PATH should be done by adding files to /etc/paths.d.

As an administrator you should be on the lookout for scripts and installers that attempt to modify configuration files on the user level, disable the scripts during deployment, and manage the required changes centrally. This will allow you to keep control of the settings even as tools change, are added or removed from the system, while preserving the user’s custom configurations.

To manage environment variables other than PATH centrally, administrators should consider /etc/zshenv or adding to the existing /etc/zshrc. In these cases you should always monitor whether updates to macOS overwrite or change these files with new, modified files of their own.

Summary

There are many possible files where the zsh can load user configuration. You should use ~/.zshrc for your personal configurations.

There are many tools and projects out there that will configure zsh for you. This is fine, but might keep you from really understanding how things work.

MacAdmins who need to manage these settings centrally, should use /etc/paths.d and similar technologies or consider /etc/zshenv or /etc/zshrc.

Apple’s built-in support for zsh in Terminal is not as detailed as it is for bash.

Next: Part 3 – Shell Options

Weekly News Summary for Admins — 2019-06-14

First week post-WWDC and people are slowly catching up with the firehose of information. Lot’s of opinion pieces and podcasts to catch up with.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • macshome: “With so many people running macOS in a VM right now here is a ProTip for Fusion on a TouchID Mac. Set: board-id.reflectHost = "FALSE" Now your auth dialog spins are gone!”
  • Craig Hockenberry: “I know a lot of developers who have been working with Apple’s products for decades. The overwhelming consensus is that we’re seeing something that will change our lives for decades to come. 1976 -> 1984 -> 1996 -> 2008 -> 2019”
  • Daniel Jalkut: “Apple has been doing hardware penance lately. I hope they realize that exiting the personal networking market (Airport) was as as misguided as exiting pro Mac, pro displays, etc. We’re ready for the big comeback.”
  • Derek Fulmer: “With the impending changes to macOS in 10.15, I’m giving zsh a go. Really digging its customizability. Feels way more modern. But, I’m still sentimental about bash.”

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

  • Erik Schwiebert: “Microsoft Office, macOS 10.15 Catalina, and You: @mrexchange explains our support plans for the beta and final OS release.”

To Listen

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Imaging is still Dead

At WWDC last week, there was a very interesting session on “Apple File Systems” (APFS). It covered the new split system layout in macOS Catalina with a read-only system volume, volume replication with APFS, and how external USB drives and SMB works on iPadOS.

The entire session is very interesting and well worth watching. Go ahead, I’ll wait…

Around the 13 minute mark, during the ‘Volume Replication’ segment, the engineer on stage talks about using asr (Apple Software Restore) tool to ‘replicate’ a system volume to several computers at once and gives the example of a computer lab. He then proceeds to explain the new options in asr regarding APFS volumes and snapshots.

Slide from WWDC 2019, Session 710

The new features are hugely interesting and I think they will be very useful for backup solutions. There will probably be some applications for MacAdmins, but I disagree with the engineer on stage and some MacAdmins on Twitter and Slack:

Catalina will not bring a revival of imaging.

Note: I wrote a book on this: “macOS Installation for Apple Administrators

What killed imaging?

Back in the Sierra days, there was this idea that the introduction of APFS would ‘kill’ imaging. The asr tool relied on many HFS+ behaviors and it was questionable that Apple could or would maintain that for APFS. But while there were some changes to asr in the High Sierra and Mojave upgrades, it still worked.

What killed imaging as a process for MacAdmins was the T2 system controller, first introduced with the iMac Pro. There are two main aspects:

  • NetBoot and external boot are defunct
  • Firmware needs to be updated with the system

Netboot and external boot are defunct

To re-image or re-install a system, you have to boot it off a different system volume (NetBoot, Recovery, external drive). Alternatively, you can put the system into target disk mode and image or install the system directly on the internal drive.

On Macs with the T2 system controller, NetBoot is explicitly defunct. External boot is disabled by default. It can be re-enabled, but the process is convuluted, requires at least one full setup process, and cannot be automated.

This leaves Recovery as the system to use to replace the system volume and, not surprisingly, there are a few tools that have focussed on using Recovery in the new T2 Mac world:

Firmware needs to be updated with the system

You could also put the target Mac in target disk mode and image its system. This will work, as long as the system on the image is the same version as the system that was installed before. We have been warned about this in the infamous HT208020 support article:

Apple doesn’t recommend or support monolithic system imaging as an installation method. The system image might not include model-specific information such as firmware updates.

Modern Macs don’t just require a few files on disk to make a bootable system. Inside your Mac are several subsystems that require their own systems (i.e. firmware) to run. Most prominent are the T1 or T2 system controllers which are actually independent custom ARM-based processers running a system called ‘iBridge’ which is an iOS derivate.

If you just exchange the ‘normal’ system files on the hard drive over TDM, without also updating the various firmwares in the system, you may get your Mac into state where it cannot boot.

This was most obvious with the macOS High Sierra upgrade. After re-imaging a 10.12 Sierra Mac to High Sierra running on APFS, would lead to a Mac that could not read the new system volume. The firmware update that came with High Sierra is needed, so the firmware can mount, read and start the APFS system volume.

How can I update or upgrade?

For security, only Apple’s ‘Install macOS’ application and the intermediate software and security update packages have the necessary entitlements to change the built-in firmware(s).

Firmware updates can be in system updates (minor version updates, i.e. 10.14.4 to 10.14.5), security updates, and major system upgrades (i.e. 10.13 to 10.15).

There are three options to apply a system update (e.g. from 10.14.4 to 10.14.5) or security update:

  • ‘Install macOS *’ application, either manually or with the startosinstall tool
  • Software Update, either manually or through the command line tool
  • system or security update pkg installer downloaded from support.apple.com

When you want to upgrade a Mac through a major version change (e.g. 10.13 to 10.14 or 10.15), there is only one option:

  • ‘Install macOS *’ application, either manually or with the startosinstall tool

The one remaining use case for imaging

Given the above limitations, there is one use case left for imaging. When you have full control over the macOS version installed on the Mac and its firmware and the image matches that version, then you can image.

However, since NetBoot and external boot are defunct, you will have to image either over target disk mode (fast, but only a few Macs at a time) or using the Recovery (hard to automate, comparatively slow).

The remaining strength of the imaging workflow is the raw speed. Some application suites measure several Gigabytes, if not tens of Gigabytes. With installation workflows, these have to be downloaded, decompressed (pkg installers are compressed archives) and copied to the system drive, a process that takes a lot of time. With imaging, these can be layed down with fast block copies.

For example, the re-installation of a MacBook Pro I tested recently took about 25 minutes. This time includes downloading the 6GB ‘Install macOS’ application and the entire re-installation process. (I could probably have sped this up with a caching server or by pre-installing the full ‘Install macOS’ applications.) If I could have used imaging this would take 2–3 minutes.

If you are in a situation where you have to restore Macs to a pre-defined state frequently and quickly, then imaging might still be a useful workflow. One use case may be MacBooks that get frequently handed out as loan units, where the users get administrative privileges, so they can install extra software and configure the loan units.

You will have to invest extra effort during updates or upgrades to apply them first on the devices, to ensure the firmware gets updated, and then to update the image, as well. In some use cases this extra effort can be worthwhile.

MDM (and DEP) is required

With modern macOS there are other considerations for deployment that make classic imaging workflows less practical. Before macOS 10.13 High Sierra, MacAdmins could manage their Mac fleet without an MDM server. In High Sierra 10.13.4 Apple added two things to the MDM protocol:

  • ‘user-approved’ MDM
  • Kernel extension white listing via configuration profile

The second feature (white listing Kernel extensions) requires the first (user-approved MDM). You cannot manage Kernel Extensions or Privacy Preferences Control settings in Mojave, with out a user-approved MDM. In mosts organizations, these are not limitations you can work around. An MDM is now a requirement to manage Macs in an organization.

From what we can glean from the WWDC sessions, the (UA)MDM controls will be increased even further with Catalina. It will be driven even further: DEP or ‘Automated Device Enrollment’ with Apple Business Manager or Apple School Manager will be required for some new management features, such as ‘bootstrap tokens’ for FileVault.

Each Mac client needs to be enrolled in the MDM individually. The MDM enrollment cannot be part of an image. The easiest way to get a Mac enrolled is with Automated Device Enrollment (formerly known as DEP), which happens at first boot after installation.

Third party software

It is not just the macOS system that needs to individually enroll with the MDM server. Many third party solutions now also require subscriptions or licenses to be activated on each device individually. All these additional configurations that need to happen after installation or imaging, decrease the usefulness of including all software and configuration in an image.

Patching and software updates

Most imaging deployments, used a workflow where the image was kept ‘static’ or ‘frozen’ for longer periods of time, usually six or twelve months. This will minimize the effort to update the image, system and software.

However, modern operating systems and third party software have update frequencies of 4–10 weeks. Modern security requirements will require these updates to be applied in a timely matter. Critical security problems can strike at any time, requiring fast updates from the vendors and the Mac Admins.

As with the MDM above, having a system in place that allows the MacAdmin to easily and quickly deploy and, when necessary, enforce an update or patch to the entire fleet of devices is an important requirement.

Software and patch management of non-App Store applications is not part of the MDM protocol. Nevertheless, many MDM solutions also include additional functionality for software management, with varying degree of usefulness.

Some MacAdmins prefer to combine their MDM solution with the open source solution Munki instead. Munki is considered to be the best software management solution for macOS, but does not include MDM functionality itself.

Whichever software management solution you use, once you have that in place, it will be easier to manage (i.e. install and enforce) software through the management system, than to keep an image up-to-date and re-applying it.

You will end up with a ‘thin’ base image and everything else deployed and managed by the management system. At that point you might as well switch to an installation based workflow.

But, the engineer on stage said…

Here are all the limitations on imaging, summarized:

  • NetBoot and external boot are defunct
  • system firmware needs to be updated with the system
  • MDM and DEP are required
  • frequent security updates and patches require continuous software management

None of these limitations are addressed by the changes to the asr tool in Catalina. Changes in other areas of the system in Catalina will actually re-inforce some of these limitations.

Imaging is still dead.

But why even have asr, then?

The asr tool exists because Apple needs a tool to image the operating system to new Macs in the factory. Obviously, Apple has absolute control over the versions of macOS and firmwares deployed to the systems, so they ensure they all match. Speed is a priority, so Apple needs and maintains asr.

Other uses of asr, including the use as an imaging tool for administrators have always been secondary.

As mentioned earlier, when your environment has similar requirements (fast re-deployment) and can provide tight control over the macOS and firmware versions, then imaging might still be a useful workflow for you.

You can already do this with High Sierra or Mojave. You do not have to wait for the new Catalina features for this.

In general, a simpler (albeit slower) installation-based workflow is less complex to deploy and maintain. (Imaging might seem less complex, because it is more familiar.)

So, the new features in the presentation are pointless?

The other use case for asr in the presentation, backups, are very exciting. They will allow the system to take a snapshot and then copy the data of the snapshot to a backup while the system keeps running and changing files. You may also be able to restore a system from a snapshot stored elsewhere.

The split of system volume and user data volume in Catalina is also very intriguing for Mac Admins. This may of course, break some third party software. (Start testing now.) But it may also open up new options for management. One of these (user enrollment) is introduced in the “Managing Apple Devices” WWDC video.

One possible workflow could be to snapshot and/or image the data volume and leave the system volume intact (you have to, it is read-only and SIP protected). It is still questionable how well this might work, since the firmlink connections between the system and the data volume might not survive the replacement of their targets. You can start testing this now, but keep in mind that the details of the new file system layout will still change during the beta phase.

Summary

  • The changes introduced to the file system in macOS Catalina at WWDC are major and will enable new workflows for MacAdmins.
  • Start testing Catalina now.
  • The limitations that ‘killed’ imaging, still apply or might be re-inforced. Imaging is still dead

Weekly News Summary for Admins — 2019-06-07

WWDC! And what a firework it is this year!

We got new previews of all of Apple’s operating systems, including a new, stand-alone, more powerful iPadOS. We got a look at the new Mac Pro with a high-end new 6K display. We got Marzipan… er Project Catalyst. And we got a quick peak at a new Swift-native user interface framework.

There were tons of interesting pieces for MacAdmins already. Apple is switching the default shell to zsh, will stop bundling scripting frameworks with the system, has released new management documentation, is moving to a read-only system partition, and more… And there will be more today when ‘What’s new in Managing Apple Devices’ is presented.

It will take weeks and months to sort through all the changes. Make sure to subscribe to the developer program for the earliest betas, AppleSeed for IT (ask your Apple representative), or the public beta, so you can start testing early and often! And keep reading this news summary, so you know what to look out for.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

WWDC and macOS 10.15 Catalina

Apple Newsroom

Apple Preview Pages

Release Notes

More Notes

MacAdmin reactions and posts

On Scripting OS X

  • Moving to zsh
  • If you are using the Exit Code prompt setup from my post last week: co-worker Mattias found an embarrassing error that may have resulted in all exit codes being a happy green checkmark. The code in the post is now fixed (only the last line changed). On the other hand I will be posting how to do that with zsh soon.
  • After finishing the first ‘Scripting macOS’ class at Pro Academy last week, we have new dates on October 23 and 24 for the next class here in our training rooms in Amsterdam. I will be busy updating the class for macOS Catalina. There will also be a ‘Supporting macOS’ class on September 18 and 19. If you are interested, then please contact us through the form on the website. You can also use the contact form if you are interested but the dates do not fit your schedule. We will consider your preferences for further scheduling.

News and Opinion

MacAdmins on Twitter

  • Michael Palermiti: “Pssst. Hey, I’ve got some exciting news for our enterprise customers… Shared Mailbox support is now in TestFlight for @Outlook for iOS! We need your help to try it out and give us feedback. And before anyone asks, yes, Android support is not far behind!”
  • Nick Takayama: “Shouldn’t it be called the WWDC Beer Zsh Now?”

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

To Watch

To Listen

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

zsh (I believe it is pronounced zee-shell, though zish is fun to say) will succeed bash as the default shell. bash has been the default shell since Mac OS X 10.3 Panther.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

Why?

The bash binary bundled with macOS has been stuck on version 3.2 for a long time now. bash v4 was released in 2009 and bash v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3. bash v3 is still GPL v2.

zsh, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default. zsh has been available as on macOS for a long time. The zsh version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current zsh 5.7.1.

Is bash gone!?

No.

macOS Catalina still has the same /bin/bash (version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.

Many scripts in macOS, management systems, and Apple and third party installers rely on /bin/bash. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.

Users that have /bin/bash as their default shell on Catalina will see a prompt at the start of each Terminal session stating that zsh is now the recommended default shell. If you want to continue using /bin/bash, you can supress this message by setting an environment variable in your .bash_profile or .bashrc.

export BASH_SILENCE_DEPRECATION_WARNING=1

You can also download and install a newer version of bash yourself. Keep in mind that custom bash installations reside in a different directory, usually /usr/local/bin/bash.

Will bash remain indefinitely?

Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to bash, but didn’t really care if you remained on tcsh. In fact, tcsh is still present on macOS.

Apple’s messaging should tell us, that the days of /bin/bash are numbered. Probably not very soon, but eventually keeping a more than ten year old version of bash on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.

Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’ dash has been added to the lineup of shells. dash is designed to be a minimal implementation of the Posix standard shell sh. So far, in macOS (including Catalina),sh invokes bash in sh-compatibility mode.

As Apple’s support article mentions, Catalina also adds a new mechanism for users and admins to change which shell handles sh invocations. MacAdmins or users can change the symbolic link stored in /var/select/sh to point to a shell other than /bin/bash. This changes which shell interprets scripts the #!/bin/sh shebang or scripts invoked with sh -c. Changing the interpreter for sh should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.

All of these changes are indicators that Apple is preparing to remove /bin/bash at some, yet indeterminate, time in the future.

Do I need to wait for Catalina to switch to zsh?

No, zsh is available Mojave and on older macOS versions. You can start testing zsh or even switch your default shell already.

If you want to just see how zsh works, you can just open Terminal and type zsh:

$ zsh
MacBook%

The main change you will see is that the prompt looks different. zsh uses the % character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in bash.

If you want to already switch your default shell to zsh you can use the chsh command:

$ chsh -s /bin/zsh

This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.

How is zsh different?

Like bash (‘Bourne again shell’ ), zshderives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.

The main difference between bash and zsh is configuration. Since zsh ignores the bash configuration files (.bash_profile or .bashrc) you cannot simply copy customized bash settings over to zsh. zsh has much more options and points to change zsh configuration and behavior. There is an entire eco-system of configuration tools and themes called oh-my-zsh which is very popular.

zsh also offers better configuration for auto-completion which is far easier than in bash.

I am planning a separate post, describing how to transfer (and translate) your configurations from bash to zsh.

What about scripting?

Since zsh has been present on macOS for a long time, you could start moving your scripts from bash to zsh right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to #!/bin/zsh.

You will gain some features where zsh is superior to bash v3, such as arrays and associative arrays (dictionaries).

There is one exception where I would now recommend to use /bin/sh for your scripts: the Recovery system does not contain the /bin/zsh shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider older macOS installations where zsh is definitely not present in Recovery.

When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as Twocanoes MDS, installr or bootstrappr, then you cannot rely on /bin/zsh.

Since we now know that bash is eventually going away, the only common choice left is /bin/sh.

When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using /bin/sh as the shebang from now on.

I used to recommend using /bin/bash for everything MacAdmin related. /bin/sh is definitely a step down in functionality, but it seems like the safest choice for continued support.

Summary

Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’ bash with zsh, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!

Next

In the next part we will look at the configuration files for zsh.

Weekly News Summary for Admins — 2019-05-31

Between Memorial Day in the US, Ascension Day in parts of Europe, and WWDC looming next week, this was a quiet news week.

Apple did have one more thing to get out before WWDC: the iPod touch was updated with the A10 processor.

Now, only the Mac Pro remains as a device that has not been updated in the last two years. (MacBook barely makes the two years limit with its last update in June 2017.)

In other news, the first “Scripting macOS” class took place this week. The attendees (and I) believe it went really well! You can still sign up for the next class here. If the next date doesn’t suite you, please use the contact form and let us know when you would like a class. We will be scheduling additional classes soon and your input will be considered.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Headlines

On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • Graham Pugh: “IBM SPSSStatistics 26 still needs Java installed in order to get installed on Mac, but installs a JRE as part of the installation. How hard would it be to put the JRE in the installer itself?”
  • Timo Perfitt: “Right Click->Open to Install?… ” (Click for image)
  • Tom Bridge: “So, I’ve spent a little time with Mosyle + Google SSO + DEP tonight, and I gotta hand it to the team at @mosyle_biz : That’s a helluva beta. I can see that being HIGHLY useful.”

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Listen

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!