Weekly News Summary for Admins — 2019-07-19

Still a lot of fallout from the Zoom invulnerability. It took Apple three updates to MRT (so far) to eradicate all the differently branded varieties of the Zoom client web server. Makes me wonder how many Macs there were or are with multiple of these clients installed.

We also got new betas for 10.14.5 and 10.15 and the respective iOS versions. Some of the worst data-destroying bugs seem to be fixed or at least mitigated but I am still not comfortable moving my production devices to the betas. That’s what test devices are there for. I am getting quite excited about some of the features I have seen, both for end-users and administrators.

One of those features is that zsh will become the default shell for macOS. If you want to know what that means, how to transfer your shell configuration, workflows, and scripts from bash to zsh, and increase your Terminal productivity, we are doing a half-day training in Amsterdam on September 6. You can get more details and sign up on our webpage!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

Many,many thanks to Erik for his contributions to the Mac Admin community and I wish him all the best for the future!

MacAdmins on Twitter

  • Timo Perfitt: “So it begins. We just signed up to be an MDM vendor. MDS DEP deployment coming soon.”

Bugs and Security

macOS 10.15 Catalina and iOS 13

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh, part 6 – Customizing the zsh Prompt

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

The default bash prompt on macOS is quite elaborate. It shows the username, the hostname, and the current directory.

Calypso:~ armin$

On the other hand, the default bash prompt doesn’t show the previous command’s exit code, a piece of information I find very useful. I have written before how I re-configured my bash prompt to have the information I want:

Of course, I wanted to recreate the same experience in zsh.

Minimal Zsh Prompt

The only (visual) difference to my bash prompt is the % instead of the $.

Note: creating a file ~/.hushlogin will suppress the status message at the start of each Terminal session in zsh as well as in bash (or any other shell).

Basic Prompt Configuration

The basic zsh prompt configuration works similar to bash, even though it uses a different syntax. The different placeholders are described in detail in the zsh manual.

zsh uses the same shell variable PS1 to store the default prompt. However, the variable names PROMPT and prompt are synonyms for PS1 and you will see either of those three being used in various examples. I am going to use PROMPT.

The default prompt in zsh is %m%#. The %m shows the first element of the hostname, the %# shows a # when the current prompt has super-user privileges (e.g. after a sudo -s) and otherwise the % symbol (the default zsh prompt symbol).

The zsh default prompt is far shorter than the bash default, but even less useful. Since I work on the local system most of the time, the hostname bears no useful information, and repeating it every line is superfluous.

Note: you can argue that the hostname in the prompt is useful when you frequently have multiple terminal windows open to different hosts. This is true, but then the prompt is defined by the remote shell and its configuration files on the remote host. In your configuration file, you can test if the SSH_CLIENT variable is set and show a different prompt for remote sessions. There are more ways of showing the host in remote shell sessions, for example in the Terminal window title bar or with different window background colors.

In our first iteration, I want to show the current working directory instead of the hostname. When you look through the list of prompt placeholders in the zsh documentation, you find %d, %/, and %~. The first two do exactly the same. The last substitution will display a path that starts with the user’s home directory with the ~, so it will shorten /Users/armin/Projects/ to ~/Projects.

Note: in the end you want to set your PROMPT variable in the .zshrc file, so it will take effect in all your zsh sessions. For testing, however, you can just change the PROMPT variable in the interactive shell. This will give you immediate feedback, how your current setup works.

% PROMPT='%/ %# '
/Users/armin/Projects/dotfiles/zshfunctions % 

% PROMPT='%~ %# '
~/Projects/dotfiles/zshfunctions % 

Note the trailing space in the prompt string, to separate the final % or # from the command entry.

I prefer the shorter output of the %~ option, but it can still be quite long, depending on your working directory. zsh has a trick for this: when you insert a number n between the % and the ~, then only the last n elements of the path will be shown:

% PROMPT='%2~ %# '
dotfiles/zshfunctions %                       

When you do %1~ it will show only the name of the working directory or ~ if it is the home directory. (This also works with %/, e.g. %2/.)

Adding Color

Adding a bit of color or shades of gray to the prompt can make it more readable. In bash you need cryptic escape codes to switch the colors. zsh provides an easier way. To turn the directory in the path blue, you can use:

PROMPT='%F{blue}%1~%f %# '

The F stands for ‘Foreground color.’ zsh understands the colors black, red, green, yellow, blue, magenta, cyan and white. %F or %f resets to the default text color. Furthermore, Terminal.app represents itself as a 256-color terminal to the shell. You can verify this with

% echo $TERM
xterm-256color

You can access the 256 color pallet with %F{0} through %F{255}. There are tables showing which number maps to which color:

So, since I want a dark gray for my current working dir in my prompt, I chose 240, I also set it to bold with the %B code:

PROMPT='%B%F{240}%1~%f%b %# '

You can find a detailed list of the codes for visual effects in the documentation.

Dynamic Prompt

I wrote an entire post on how to get bash to show the color-coded exit code of the last command. As it turns out, this is much easier in zsh.

One of the prompt codes provides a ‘ternary conditional,’ which means it will show one of two expressions, depending on a condition. There are several conditions you can use. Once again the details can be found in the documentation.

There is one condition for the previous commands exit code:

%(?.<success expression>.<failure expression>)

This expression will use the <success expression> when the previous command exited successfully (exit code zero) and <failure expression> when the previous command failed (non-zero exit code). So it is quite easy to build an conditional prompt:

% PROMPT='%(?.√.?%?) %1~ %# ' 
√ ~ % false
?1 ~ % 

You can get the character with option-V on the US or international macOS keyboard layout. The last part of the ternary ?%? looks confusing. The first ? will print a literal question mark, and the second part %? will be replaced with previous command’s exit code.

You can add colors in the ternary expression as well:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# ' 

Another interesting conditional code is ! which returns whether the shell is privileged (i.e. running as root) or not. This allows us to change the default prompt symbol from % to something else, while maintaining the warning functionality when running as root:

% PROMPT='%1~ %(!.#.>) ' 
~ > sudo -s
~ # exit
~ > 

Complete Prompt

Here is the complete prompt we assembled, with all the parts explained:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# '
%(?.√.?%?) if return code ? is 0, show , else show ?%?
%? exit code of previous command
%1~ current working dir, shortening home to ~, show only last 1 element
%# # with root privileges, % otherwise
%B %b start/stop bold
%F{...} text (foreground) color, see table
%f reset to default textcolor

Right Sided Prompt

zsh also offers a right sided prompt. It uses the same placeholders as the ‘normal’ prompt. Use the RPROMPT variable to set the right side prompt:

% RPROMPT='%*'
√ zshfunctions %                    11:02:55

zsh will automatically hide the right prompt when the cursor reaches it when typing a long command. You can use all the other substitutions from the left side prompt, including colors and other visual markers in the right side prompt.

Git Integration

zsh includes some basic integration for version control systems. Once again there is a voluminous, but hard to understand description of it in the documentation.

I found a better, more specific example in the ‘Pro git’ documentation. This example will show the current branch on the right side prompt.

I have changed the example to include the repo name and the branch, and to change the color.

autoload -Uz vcs_info
precmd_vcs_info() { vcs_info }
precmd_functions+=( precmd_vcs_info )
setopt prompt_subst
RPROMPT=\$vcs_info_msg_0_
zstyle ':vcs_info:git:*' formats '%F{240}(%b)%r%f'
zstyle ':vcs_info:*' enable git

In this case %b and %r are placeholders for the VCS (version control system) system for the branch and the repository name.

There are git prompt solutions other than the built-in module, which deliver more information. There is a script in the git repository, and many of the larger zsh theme projects, such as ‘oh-my-zsh’ and ‘prezto’ have all kinds of git status widgets or modules or themes or what ever they call them.

Summary

You can spend (or waste) a lot of time on fine-tuning your prompt. Whether these modifications really improve your productivity is a matter of opinion.

In the next post, we will cover some miscellaneous odds and ends that haven’t yet really fit into any of preceding posts.

Weekly News Summary for Admins — 2019-07-12

Apple released an unexpected update to the entry-level MacBook Air and MacBook Pro 13“ this week. What the new release replaces is interesting, as well: the non-retina entry level MacBook Air, the no-Touchbar 13” MacBook Pro, and the 12″ MacBook are all discontinued.

The iMac and the (cylinder) Mac Pro are the only remaining Mac models without a T2 chip. All current Mac models except the iMac Pro and the Mac Pro now require macOS 10.14 Mojave as their minimum OS.

We also got an interesting vulnerability story with the popular videoconferencing software Zoom. The reaction from Zoom was slow, they did not address the issue until the researcher published the problem, but Apple reacted much faster, adding the problematic local web server, that Zoom installed to bypass some of the system’s user security to macOS’s Malware Removal Tool’s list of, well, malware within 24 hours.

Finally, I am going to give a half-day class based on the ‘Moving to zsh’ article series at our offices in Amsterdam on September 6. If you are wondering why Apple is changing the default shell, how it will affect you, and how you can increase your Terminal productivity with zsh, then sign up here!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

macOS Catalina and iOS 13

MacAdmins on Twitter

  • Sisyphus-J: “MacOS/iOS Entitlement database now updated for MacOS 15 and iOS 13 beta 3. with over 150 more daemons and 500 more entitlements.. including such fine gems as com.apple.rootless.kext-more-securer-management http://newosxbook.com/ent.jl”
  • Mustafa Hussain: “In macOS, when you drop a Calendar event into the Terminal, the Terminal adopts the event’s color. As it should.… ”
  • Steve Troughton-Smith: “Disasters like Zoom are why the Mac gets progressively more locked-down, and why commandline tools and daemons can’t be exempt from signing & security. By default, we should trust no software, and neither should the hardware it runs on…”
  • Alex Wild: “Apparently sailors used to use a log, tied to a measured rope & thrown overboard, to periodically measure the speed of a boat. They’d then write the data in a ”log-book“ Which is why we now use the word ”log“ to record data. Things I learned. https://www.etymonline.com/word/log”
  • Isaac T.: “Confirmed with Apple: sharing with SMB is broken in macOS Mojave 10.14.5. The problem: ”smbd“ deadlocks with multiple users connected to a share. Engineering knows and is working on it. Time to resolution: unknown. Here are the options: 1/” (thread)
  • Rich Mogull: ““Shadow IT” is merely how the fearful describe the reduction of friction and democratization of technology to better enable agile business.”

PSU MacAdmins Talks, Slides and Resources

Bugs and Security

Support and HowTos

To Watch

To Listen

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Build a shellcheck installer for macOS

Shellcheck is an invaluable tool for anyone who writes, well, shell scripts. The tool will point out common and less common errors, such as forgetting to quote a variable substitution, not putting that space before the closing ]], or not testing if a variable substitution used with rm -Rf might be empty.

You can use the tool’s website to check your code, but it is unwieldy, not to mention you or your organization might have concerns uploading all your code to some website, useful as it may be. You can install a shellcheck command line tool, but the instructions available for macOS will only work with Homebrew.

If you already have Homebrew installed, or don’t mind installing it and only need to install shellcheck on a single Mac, then you can use brew install shellcheck and skip ahead to the end, where I share how to use the shellcheck binary with BBEdit.

Relevant for MacAdmins

Homebrew may be a fine tool for individual users, but it is near-impossible to centrally manage a deployment of brew let alone a piece software installed with brew.

My attempts to build the shellcheck binary without homebrew lead to several dead-ends. Shellcheck is written in Haskell Cabal, which can be easily installed with… brew. There is also an official ghcup script, which is the ‘recommended way to install Haskell’, but this script fails on macOS because it requires a tool named xz. This is a (de-)compression tool, which is part of xz-utils, whose website does not explain how to install on macOS, but can be easily installed with… guess… brew

Even without brew, you have to install a number of tools that you probably don’t want hanging around on your production machine. I am sure cabal is a fine environment to build other tools as well, but I don’t need it on my production system.

I would recommend running these steps on a virtual machine. That will simplify ‘cleanup’ as well as the ‘start over’ process, should you need it.

How to build the shellcheck binary for macOS

Commands shown are correct at the time this article was written and may change in the future. Links to original sources are given, so you can check for changes.

Developer Tools

  • install Xcode or command line developer tools (CLI dev tools have the advantage of not being >6GB in size)
$ xcode-select --install

XZ Utils

Note: the macOS Archive Utility can deal with .xz archives just fine. There is really no need on macOS to have the xz-utils installed, but the GHC Cabal installer requires them.

$ cd ~/Downloads/xz-5.2.4
$ ./configure
$ sudo make install
  • check to see of xz was installed correctly
$ xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.4

Install Pandoc

We will require pandoc to build the man page for shellcheck.

Haskell Cabal

$ curl https://get-ghcup.haskell.org -sSf | sh
  • make sure the cabal environment is setup
$ source "$HOME/.ghcup/env"
$ cabal update

Shellcheck

$ git clone "https://github.com/koalaman/shellcheck.git"
  • build the shellcheck binary (this will take a while)
$ cd shellcheck
$ cabal install
  • this will build and install the shellcheck binary into ~/.cabal/bin, you can test if this worked with
$ ~/.cabal/bin/shellcheck --version
ShellCheck - shell script analysis tool
version: 0.6.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net

man page

  • there is also a man page in the repo, written in markdown format. The ReadMe says you can convert it to man format with pandoc:
$ pandoc -s -f markdown-smart -t man shellcheck.1.md -o shellcheck.1

Pkg it up

  • create a payload directory for the binary and man file and move the files there:
$ mkdir -p payload/local/bin
$ cp ~/.cabal/bin/shellcheck payload/local/bin/
$ mkdir -p payload/share/man/man1
$ cp shellcheck.1 payload/share/man/man1/
  • build the pkg installer from the payload with pkgbuild
$ pkgbuild --root payload --installer-location /usr --identifier com.scriptingosx.pkg.shellcheck --version 0.6.0 shellcheck-0.6.0.pkg

Automation!

Of course, you can do this once and then forget about until the next update of shellcheck. While I wrote this post, I had to go through these steps more than once and got bored with it halfway through the second cycle. So, there is a script, which you can get at this Github repo.

Since this script will install xz-utils, cabal, and pandoc, I do not recommend to run this on your work machine. I use it with clean virtual machine. I suggest this workflow:

  • boot to clean vm
  • snapshot the vm
  • open Terminal
  • install developer command line tools: xcode-select --install
  • clone the script: git clone https://github.com/scriptingosx/BuildShellcheckPkg
  • run the script: cd BuildShellcheckPkg; ./buildShellCheckPkg.sh
  • confirm the prompts (once the cabal installation starts for good, the prompts are over)
  • wait…
  • wait…
  • start doing other work such as answering emails, MacAdmins Slack, Twitter,…
  • much later, remember there was something important, go to the vm, copy the brand new shellcheck pkg to your work machine
  • revert the vm to snapshot
  • buy ‘Packaging for Apple Administrators’ to show your gratitude and learn how to wield pkgbuild yourself

Using shellcheck

Using the shellcheck command line tool is very straightforward. Just pass one (or more) shell script files as arguments. There are some flags and options that you can read about in the --help option or the man page.

$ shellcheck serial.sh

In serial.sh line 11:
serial_number=$(serial) # stores output of function serial in variable
^-----------^ SC2034: serial_number appears unused. Verify use (or export if used externally).

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- serial_number appears unused. Ver...

Use shellcheck with BBEdit

My favored text editor – BBEdit – has a tool that will display the output of tools like shellcheck. This allows you to click on an error or warning and see and edit the code right there in a BBEdit window.

To get this, pipe the output of shellcheck into bbresults:

$ shellcheck -f gcc serial.sh | bbresults

I have created a function in my .bash_profile and .zshrc, so I don’t have to remember the details:

function bbshellcheck {
    shellcheck -f gcc "$@" | bbresults
}

If you do not know what the .bash_profile or functions are, then you can read about them here:

Note: you cannot use shellcheck to check zsh scripts. As of now it only works for sh or bash scripts. You can tell shellcheck to ignore the #!/bin/zsh shebang and interpret the script as using bash syntax, which should work in most cases, but will not always have good results.

$ shellcheck -s bash script.zsh

Moving to zsh, part 5: Completions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class in Amsterdam on September 6! Visit our website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

What are Completions?

Man shells use the tab key (⇥) for completion. When you press that key, the shell tries to guess what you are typing and will complete it, or if the beginning of what you typed is ambiguous, suggest from a list of possible completions.

For example when you want to cd to your Documents folder, you can save typing:

% cd ~/Doc⇥
% cd ~/Documents/

When you hit the tab key, the system will complete the path to the Documents folder.

When the completion is ambiguous, the shell will list possible completions:

% cd ~/D⇥
Desktop/    Documents/  Downloads/

At this point, you can add a character or two to get to a unique completion, and hit the tab key again. In zsh you can also hit the tab key repeatedly to cycle through the suggested completions. In this example, the first tab keystroke will show the list, the second will complete ~/Desktop/, the third completes ~/Documents, and so on.

You can use tab completion commands as well:

% system⇥
system_profiler     systemkeychain      systemsetup         systemsoundserverd  systemstats
% system_⇥
% system_profiler 

Not having to type path and file names saves time and avoids errors, especially with complex paths with spaces and other special characters:

% cd ~/Li⇥
% cd ~/Library/Appl⇥
% cd ~/Library/Application S⇥
Application Scripts/  Application Support/
% cd ~/Library/Application Su⇥
% cd ~/Library/Application Support/

Using tab completion is a huge productivity boost when using a shell.

Turning It On

In the default configuration, tab completion in zsh is very basic. It will complete commands and paths, but not much else. But you can enable a very powerful, and useful completion system.

zsh comes with a tool you can use to setup this completion system. When you run the compinstall command it will lead you through a complex and hard to understand list of menus which explains the options and will generate the code necessary to set this configuration up and add it to your .zshrc file or another configuration file of your choice.

Since the commands to configure the completion are quite arcane and hard to understand, this is a good way to get something to start out with. I will explain some of these options and commands in detail.

Whether you use compinstall or not, to turn on the more powerful completion system, you need to add at least this command to yourzsh` configuration file:

autoload -Uz compinit && compinit

This will initialize the zsh completion system. The details of this system are documented here.

If you want to configure the system, the configuration commands (usually zstyle commands) should be added to the zsh configuration file before you enable the system. (This only matters for a few configurations, but as a general rule it is safer.)

All of these completion rules need to be loaded and prepared. zsh’s completion system creates a cache in the file ~/.zcompdump. The first time you run compinit it might take a noticeable time, but subsequent runs should use this cache and be much faster.

Sometimes, especially when building and debugging your own completion files, you may need to delete this file to force a rebuild:

% rm -f ~/.zcompdump
% compinit

Case Insensitive Completion

Since the macOS file systems are usually case-insensitive, I prefer my tab-completion to be case-insensitive as well. For bash you configure that in the ~/.inputrc. In zsh you modify the completion systems behavior with this (monstrous) command:

# case insensitive path-completion
zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} l:|=* r:|=*'

I have seen many varieties for this configuration in different websites, but this is what compinstall adds when I select case-insensitive completion, so I am going with that.

Partial Completion

This is a particularly nice feature. You can type fragments of each path segment and the completion will try to complete them all at once:

% cd /u/lo/b⇥
% cd /usr/local/bin
% cd ~/L/P/B⇥
% ~/Library/Preferences/ByHost/

If the fragments are ambiguous, there are different strategies to what the completion system suggests. I have configured these like this:

# partial completion suggestions
zstyle ':completion:*' list-suffixes
zstyle ':completion:*' expand prefix suffix

Commands with built-in completion

zsh comes with several completion definitions for many commands. For example, when you type cp and then hit tab, the system will correctly assume you want to complete a file path and show the suggestions from the current working directory.

However, when you type cp -⇥ the completion can tell from the - that you want to add an option to the command and suggest a list of options for cp, with short descriptions.

% cp -⇥ 
-H  -- follow symlinks on the command line in recursive mode
-L  -- follow all symlinks in recursive mode
-P  -- do not follow symlinks in recursive mode (default)
-R  -- copy directories recursively
-X  -- don't copy extended attributes or resource forks
-a  -- archive mode, same as -RpP
-f  -- force overwriting existing file
-i  -- confirm before overwriting existing file
-n  -- don't overwrite existing file
-p  -- preserve timestamps, mode, owner, flags, ACLs, and extended attributes
-v  -- show file names as they are copied

As the context of command prompt you are assembling changes, you may get different completion suggestions. For example, the completion for ssh will suggest host names:

% ssh armin@⇥

zsh comes with completion definitions for many common commands. Nevertheless, it can be helpful to just hit tab, especially when wondering about options.

On macOS completions are stored in /usr/share/zsh/5.3/functions (replace the5.3 with 5.7.1 in Catalina). This directory stores many functions used with zsh and is in the default fpath. All the files in that directory that start with an underscore _ contain the completion definitions command. So, the file _cp contains the definition for the cp command. (Some of the definition files contain the definitions for multiple commands.)

Completions for macOS Commands

There are even a few macOS specific command that come with the default zsh installation.

% system_profiler ⇥⇥

macOS High Sierra and macOS Mojave come with zsh 5.3, which is now nearly two years old. zsh 5.3 contains less macOS specific completion definitions than the current zsh 5.7.1 which will is the pre-installed zsh in macOS Catalina. Some of the completions in 5.3 have also been updated in 5.7.1.

Tool zsh 5.3 zsh 5.7.1
caffeinate
defaults
fink
fs_usage
hdiutil
mdfind
mdls
mdutil
networksetup
nvram
open
osascript
otool
pbcopy/pbpaste
plutil
say
sc_usage
scselect
scutil
softwareupdate
sw_vers
swift
system_profiler
xcode_select

Load bash completions

Since the default shell on macOS has been bash for so long, there are quite a few bash completion definitions for macOS commands and third party tools available. For example Tony Williams’ bash completion for autopkg (post, Github).

You do not have to rewrite these completions, since the zsh completion system can use bash completion scripts as well: (add this to your zsh configuration file)

# load bashcompinit for some old bash completions
autoload bashcompinit && bashcompinit

[[ -r ~/Projects/autopkg_complete/autopkg ]] && source ~/Projects/autopkg_complete/autopkg

When you have multiple bash completion scripts you want to load, you only need to load bashcompinit once.

Build your own completions

Once you start using completions, you will want to have them everywhere. While many built-in completions exists, there are still many commands that lack a good definition.

Some commands, like the swift command line tool, have a built-in option to generate the completion syntax. You can then store that in a file and put it in your fpath:

% swift package completion-tool generate-zsh-script >_swift

Note: in the case of swift, its definition will conflict with the _openstack definition in zsh 5.3. You can fix this with the command compdef _swift swift after loading the completion system.

Some commands provide a list of options and arguments with the -h/--help option. If this list follows a certain syntax, you can get a decent completion working with

% compdef _gnu_generic <command>

One example on macOS, where this has decent results is the xed command which opens a file or folder in Xcode.

But for best results, you will often have to build the description yourself. Unfortunately this is not a simple task. The syntax is meticulously, but also quite abstractly documented in the zsh documentation for the Completion System. I also found the ‘howto’ documentation in the zsh-completions repository very useful, as well as the ‘zsh Completion Style Guide.’

To avoid everyone re-inventing the wheel, I have started a repository on Github for macOS specific completion files. The page has the instructions on how to install them and I will welcome pull requests with contributions. Since I am just starting to learn this as well, I am sure there are improvements that can be made on the completions I have built so far and there are several commands where you can test your skills and build a new one.

I suggest the #zsh channel on the MacAdmins Slack for discussion.

Next

In the next post in this series, we will discuss how to configure zsh’s command line prompt.

Weekly News Summary for Admins — 2019-07-05

For my US readers: belated Happy 4th of July!

This is where you celebrate Jeff Goldblum hacking an alien mainframe with a PowerBook 5300, right?

Because of the US holiday week-end and, I presume, several MacAdmins ‘finishing’ their PSU conference presentations, news was slow this week.

Safe travels and lots of fun, networking, and learning to those going to PSU.

Everyone else, I hope you get to enjoy a wonderful summer week-end!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

macOS Catalina

News and Opinion

MacAdmins on Twitter

  • The Macalope: “It’s going to be fun hearing the people who said design was dead at Apple bemoan the loss of the person chiefly responsible for designing all the things they said were so poorly designed.”

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Listen

Just for Fun

  • ‎Dice by PCalc (I am actually off to GM my campaign right after posting this newsletter!)

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Install Bash 5 on macOS with Patches

I recently posted an article on how to download, install, and build a macOS installer pkg for bash 5. In that first version of this post I ignored patches, minor updates to the bash source code and binary. But as the patches to bash 5 are accumulating, I cannot ignore them much longer.

This post will extend the instructions in the original post.

After downloading and expanding the bash-5.0.tar.gz, create a patches folder:

$ cd path/to/bash-5.0
$ mkdir patches
$ cd patches

You can download the patches for bash-5.0 here. As of this writing, there are seven patches for bash-5.0 labelled bash50-001 through bash50-007. You can download all at once with:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' -O

(Adapt the numbers when there are more patches in the future.)

Then move up one directory level to the bash-5.0 root directory and apply the patches using the patch command.

$ cd ..
$ patch -p0 -i patches/bash50-001
$ patch -p0 -i patches/bash50-002

(etc.)

You can download and patch with a single step. Make sure your working directory is the bash-5.0 with all the code and run:

$ curl 'https://ftp.gnu.org/gnu/bash/bash-5.0-patches/bash50-[001-007]' | patch -p0

From here, you can continue with the remaining build steps from the original post. The next step will be running ./configure.

The script to build the pkg installer has also been updated in the repository to download and apply the patches before building.

Moving to zsh, part 4: Aliases and Functions

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

Aliases

Aliases in zsh work just like aliases in bash. You declare an alias with the alias (built-in) command and it will work as a text replacement at the beginning of the command prompt:

alias ll='ls -al'

You can just copy your alias declarations from your .bash_profile or .bashrc to your .zshrc. I had aliases for .. and cd.. which are now handled by Auto CD and shell correction respectively, so I didn’t bother to move those. (part 3: ‘Shell Options’)

After the alias is declared, you can use it at the beginning of a command. When you try to use the alias anywhere else in the command, the alias will not work:

% sudo ll
sudo: ll: command not found

Global Aliases

This is where zsh has an advantage. You can declare an alias as a ‘global’ alias, and then will be replaced anywhere in the command line:

% alias -g badge='tput bel'
% sudo badge        #<beeps> with privilege

Identifying Aliases

There is one more feature of zsh that is useful with aliases. The which command will show if a command stems from an alias substitution:

% which ll
ll: aliased to ls -l

However, when you try this with global aliases, the substitution occurs before the which command can evaluate the alias, which leads to an unexpected result:

% which badge
/usr/bin/tput
bel not found

You can suppress the alias substitution by escaping the first character or by quoting the entire alias name:

% which \badge
badge: globally aliased to tput bel
% which 'badge'
badge: globally aliased to tput bel

Functions

As with aliases, functions in your zsh configuration will work just as they did in bash.

function vnc() {
    open vnc://"$USER"@"$1"
}

This code in your zsh configuration file will define the vnc function and make it available in the shell.

Autoload Functions

However, zsh has some features which make using functions more flexible. There is (once again) a bit of configuration required to get this working.

Instead of declaring the function directly the configuration file, you can put the function in a separate file. zsh has a built-in variable called fpath which is an array of paths where zsh will look for files defining a function. You can add your own directory to this search path:

fpath+=~/Projects/dotfiles/zshfunctions

Just having a file in the directory is not enough. You still have to tell zsh that you want to use this particular function:

autoload vnc

This command tells zsh: “’Declare a function named vnc. To execute it, load a file named vnc, it is somewhere in the fpath.”

Note: you often see the -U or -Uz option added to the autoload command. These options help avoid conflicts with your personal settings. They suppress alias substitution and ksh-style loading of functions, respectively.

The vnc file in my zshfunctions directory can look like this:

# uses the arguments as hostnames for `open vnc://` (Screen Sharing)
# uses the $USER username as default account name

for x in $@; do 
    open vnc://"$USER"@"$x"
done

The vnc function will open a Screen Sharing session with the current user name pre-filled in.

Initializing Autoload Functions

You could also put the code in the function file into a function block:

function vnc() {
    for x in $@; do 
        open vnc://"$vnc_user"@"$x"
    done
}

# initialization code
vnc_user="remote_admin"
alias screen_sharing='vnc'

The function name should match the function name declared with autoload.

When you have additional code outside the function, the autoload behavior changes. When the function is called for the first time, the function will be defined and the code outside the function will be run. The function itself will not be executed on the first run. On subsequent calls, the function will be executed and the code outside the function is ignored.

You can use this to provide setup and initialization code for the function. You can even have more functions defined in the function file. The above example declares and sets a variable to use for account name and an alias for the vnc command.

Since you have to run the function once for the initialization, you often see this syntax in the zsh configuration file:

autoload vnc && vnc

Which means ‘declare the function and if that succeeds run it.’

In some functions, the initialization code will already launch the function itself:

function vnc() {
    ...
}
# initialization
vnc_user="remote_admin"
vnc()

Since the behavior will vary from each autoloaded function to the next, be sure to study any documentation or the function’s code.

Identifying Functions

Finally, the which command will show the function code:

 % which vnc
vnc () {
    for x in $@
    do
        open vnc://"$USER"@"$x"
    done
}

The functions command without any parameters, will print all functions (there will be a lot of them). Use functions + to just list the function names.

Debugging Functions

When you are working on complex autoloaded functions, you will at some point have to do some debugging. You can enable tracing for functions with

% functions -t vnc
% vnc Client.local
+vnc:1> x=Client.local
+vnc:2> open vnc://armin@Client.local

You can disable tracing for this function with functions +t vnc.

Next

In the next part we will enable, use and configure tab completions.

Weekly News Summary for Admins — 2019-06-28

As expected, the public betas of iOS 13, iPadOS 13 and macOS 10.15 Catalina were released this week, keeping MacAdmins busy with both preventing users from installing them and starting their own testing procedures.

Together with the public betas, we get a bunch of early Catalina reviews and opinions.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Headlines

On Scripting OS X

News and Opinion

macOS Catalina and iOS 13

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh, part 3: Shell Options

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

Now that we have chosen a file to configure our zsh, we need to decide on ‘what’ to configure and ‘how.’ In this post, I want to talk about zsh’s shell options.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

In the previous post, I listed some features that I would like to transfer from my bash configuration. While researching how to implement these options in zsh, I found a few, new and interesting options in zsh.

The settings from bash which I want in zsh were:

  • case-insensitive globbing
  • command history, shared across windows and sessions

Note: bash in this series of posts specifically refers to the version of bash that comes with macOS as /bin/bash (v3.2.57).

Note 2: Mono-typed lines starting with a % show commands and results from zsh. Mono-typed lines starting with $ show commands and results in bash

What are Shell Options?

Shell options are preferences for the shell’s behavior. You are using shell options in bash, when you enable ‘trace mode’ for scripts with the set -x command or the bash -x option. (Note: this also works with zsh scripts.)

zsh has a lot of shell options. Many of these options serve the purpose of enabling (or disabling) compatibility with other shells. There are also many options which are specific to zsh.

You can set an option with the setopt command. For compatibility with other shells the setopt command and set -o have the same effect (set an option by name). The following commands set the same option:

set -o AUTO_CD
setopt AUTO_CD

The names or labels of the options are commonly written in all capitals in the documentation but in lowercase when listed with the setopt tool. The labels of the options are case insensitive and any underscores in the label are ignored. So, these commands set the same option:

setopt AUTO_CD
setopt autocd
setopt auto_cd
setopt autoCD

There are quite a few ways to negate or unset an option. First you can use unsetopt or set +o. Alternatively, you can prefix with NO or no to negate an option. The following commands all have the same effect of turning off the previously set option AUTO_CD

unsetopt AUTO_CD
set +o AUTO_CD
unsetopt autocd
setopt NO_AUTO_CD
setopt noautocd

Any options you change will only take effect in the current instance of zsh. When you want to change the settings for all new shells, you have to put the commands in one of the configuration files (usually .zshrc).

Showing the current Options

You can list the existing shell options with the setopt command:

% setopt
combiningchars
interactive
login
monitor
shinstdin
zle

This list only shows options are changed from the default set of options for zsh. These options are marked with <D> (default for all shell emulations) or <Z> (default for zsh) in the documentation or the zshoptions man page.

You can also get a list of all default zsh options with the command:

% emulate -lLR zsh

Some zsh Options I use

As I have mentioned before in my posts on bash configuration, I prefer minimal configuration changes, so I do not feel all awkward and lost when I have to work on an ‘un-configured’ Mac.

These configurations are a personal choice and you should pick and choose your own. You can find a full list of zsh options in the zsh Manual or with man zshoptions.

On the other hand, exploring the options allows us to explore a few useful zsh features.

Case Insensitive Globbing

Note: ‘Globbing’ is a unix/shell term that refers to the expansion of wildcard characters, such as * and ? into full file paths and names. I.e. ~/D* is expanded into /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Since the file system on macOS is (usually) case-insensitive, I prefer globbing and tab-completion to be case-insensitive as well.

The zsh option which controls this is CASE_GLOB. Since we want globbing to be case-insensitive, we want to turn the option off, so:

setopt NO_CASE_GLOB

You can test this in the shell:

% ls ~/d*<tab>

In zsh tab completion will replace the wildcard with the actual result. So after the tab you will see:

% ls /Users/armin/Desktop /Users/armin/Documents /Users/armin/Downloads

Using tab completion this way to see and possibly edit the actual replacement for wildcards is a useful safety net.

In bash hit the tab key will list possible completions, but not substitute them in the command prompt.

If you do not like this behavior in zsh then you can change to behavior similar to bash with:

setopt GLOB_COMPLETE

Automatic CD

Sometimes you enter the path to a directory, but forget the leading cd:

$ Library/Preferences/
bash: Library/Preferences/: is a directory

% Library/Preferences
zsh: permission denied: Library/Preferences

With AUTO_CD enabled in zsh, the shell will automatically change directory:

% Library/Preferences
% pwd
/Users/armin/Library/Preferences

This works with relative and absolute paths, including the ..:

% ..
% pwd
/Users/armin/Library
% ../Desktop 
% pwd
/Users/armin/Desktop

I have an alias in my .bash_profile that sets the .. command to cd ... Auto CD replaces that functionality and more.

Enable Auto CD with:

setopt AUTO_CD

Shell History

Shells commonly remember previously executed commands and allows you to recall them with the up and down arrow keys, search or special history commands.

Most of those keys work the same in zsh. However, there are a few things you need to configure for zsh history to work as you are used to with bash on macOS.

By default, zsh does not save its history when the shell exits. The history is ‘forgotten’ when you close a Terminal window or tab. To make zsh save its history to a file when it exits, you need to set a variable in the shell:

HISTFILE=${ZDOTDIR:-$HOME}/.zsh_history

Note: this is not a shell option but shell variable or parameter. I will cover some more of those later, You can find a list of variables used by zsh in the documentation.

The HISTFILE variable tells zsh where to store the history data. The syntax ${ZDOTDIR:-$HOME} means it will use the value of ZDOTDIR when it is set or default to the value of HOME otherwise. When a user has set the ZDOTDIR variable to group their configurations files in a specific directory, the history will be stored there as well.

By default zsh simply writes each command in its own line in the history file. You can view the file’s contents with any text editor or list the last few commands:

% tail -n 10 ~/.zsh_history

You can make zsh add a bit more data (timestamp in unix epoch time and elapsed time of the command) by setting the EXTENDED_HISTORY shell option.

setopt EXTENDED_HISTORY

You can set limits on how many commands the shell should remember in the session and in the history file with the HISTSIZE and SAVEHIST variables:

SAVEHIST=5000
HISTSIZE=2000

When the shell reaches this limit the oldest commands will be removed from memory or the history file.

By default, when you exit zsh (for example, by closing the window or tab) this particular instance of zsh will overwrite an existing history file with its history. So when you have multiple Terminal windows or tabs open, they will all overwrite each others’ histories eventually.

You can tell zsh to use a single, shared history file across the sessions and append to it rather than overwrite:

# share history across multiple zsh sessions
setopt SHARE_HISTORY
# append to history
setopt APPEND_HISTORY

Furthermore, you can tell zsh to update the history file after every command, rather than waiting for the shell to exit:

# adds commands as they are typed, not at shell exit
setopt INC_APPEND_HISTORY

When you use a shared history file, it will grow very quickly, and you may want to use some options to clean out duplicates and blanks:

# expire duplicates first
setopt HIST_EXPIRE_DUPS_FIRST 
# do not store duplications
setopt HIST_IGNORE_DUPS
#ignore duplicates when searching
setopt HIST_FIND_NO_DUPS
# removes blank lines from history
setopt HIST_REDUCE_BLANKS

(some of these are redundant)

Most of the time you will access the history with the up arrow key to recall the last command, or maybe a few more steps. You can search through the history with ctrl-R

In zsh, you can also use the !! history substitution, which will be replaced with the entire last command. This is most commonly used in combination with sudo:

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

By default, the shell will show the command it is substituting before it is run. But at that point, it is too late to make any changes. When you set the HIST_VERIFY option, zsh will show the substituted command in the prompt instead, giving you a chance to edit or cancel it, or just confirm it.

% systemsetup -getRemoteLogin
You need administrator access to run this tool... exiting!
% sudo !!
% sudo systemsetup -getRemoteLogin
Password:
Remote Login: On

This works for other history substitutions such as !$ or !*, as well. You can find all of zsh’s history expansions in the documentation.

Correction

When you mistype a command or path, the shell is usually unforgiving. In zsh you can enable correction. Then, the shell will make a guess of what you meant to type and ask whether you want do that instead:

% systemprofiler 
zsh: correct 'systemprofiler' to 'system_profiler' [nyae]?

Your options are to

  • n: execute as typed
  • y: accept and execute the suggested correction
  • a: abort and do nothing
  • e: return to the prompt to continue editing

I have found this far less annoying and far more useful than I expected. Especially, since it works together with AUTO_CD:

% Dekstop
zsh: correct 'Dekstop' to 'Desktop' [nyae]?

You enable zsh correction with these options:

setopt CORRECT
setopt CORRECT_ALL

Reverting to defaults

Most of the changes mentioned here affect the interactive shell and will have little impact on zsh scripts. However, there are some options that do affect the behavior of things like variable substitutions which will affect scripts.

You can revert the options for the current shell to the default settings with the following command:

emulate -LR zsh

We encountered this command earlier when we listed the default settings. The -l option will list the settings rather than apply them.

If in doubt, it may be useful to add this at the beginning of your zsh scripts.

Next

In the next part we will take a look at aliases and functions.