Installomator Updated: v0.3

It’s been more than a month since the last update, and while there has been work on the dev branch, I was quite distracted with other things (like this). The good news is, that there have been quite a few contributions from others! A huge thanks to all who helped make this a better script.

All it took was for me to find some time to put all the contributions together, which I finally found some time for.

What’s new in v0.3:

  • added several new labels for total of 98
  • removed the powershell labels, since the installer is not notarized
  • when run without any arguments, the script now lists all labels
  • changed how zips are expanded because this was broken on Mojave
  • improved logging in some statements
  • several more minor improvements

Get the script and find the instructions on the GitHub repo.

Some of the contributions and requests have not yet been addressed. I believe they will require some more thinking and planning. I would like to approach those in the next version.

If you have any feedback or questions, please join us in the #installomator channel on MacAdmins Slack.

Thanks again to all those who contributed!

Use scout to read Property Lists

I have written a few posts in the past about parsing data from property list files in scripts and Terminal. My usual tool for this is PlistBuddy. However, PlistBuddy’s syntax is… well… eccentric.

Recently, Alexis Bridoux, who is also the main developer on Octory, introduced a command line tool called scout which solves many of the issues I have with PlistBuddy.

For example, you can pipe the output of another command into scout, something you can only convince PlistBuddy to do with some major shell syntax hackery.

So instead of this:

> /usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/armin RealName)


With scout I can use this much clearer syntax:

> dscl -plist . read /Users/armin RealName | scout "dsAttrTypeStandard:RealName[0]"


The tool can also modify existing files, by changing, adding or deleting keys.

scout can also parse JSON and (non plist) XML files, so it can also stand in as a replacement for jq and xpath. It will also color-code output for property list, XML and JSON files.

I have been using scout interactively in the Terminal for a while now. So far, I have been refraining from using scout in scripts I use for deployment. To use a non-system tool in deployment scripts, you need to ensure the tool is deployed early in the setup process. Then you also have to write your scripts in a way that they will gracefully fail or fallback to PlistBuddy in the edge case where scout is not installed:

scout="/usr/local/bin/scout"
if [ ! -x "$scout"]; then
    echo "could not find scout, exiting..."
    exit 1
fi

realName=$( dscl -plist . read /Users/armin RealName | scout "dsAttrTypeStandard:RealName[0]" )


All of this overhead, adds extra burden to using a tool. The good news is that scout comes as a signed and notarized package installer, which minimizes deployment effort.

I wills be considering scout for future projects. If anyone at Apple is reading this: please hire Alexis and integrate scout or something like it in macOS.

About bash, zsh, sh, and dash in macOS Catalina and beyond

This is an excerpt from my book: “Moving to zsh.” At the MacAdmins Conference Campfire session I received quite a few questions regarding this, so I thought it would be helpful information. You can get a lot more detailed information on “Moving to zsh” in the book!

Calls to the POSIX sh /bin/sh in macOS are handled by /bin/bash in sh compatibility mode. You can verify this by asking sh for its version:

% sh --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
Copyright (C) 2007 Free Software Foundation, Inc.

If Apple plans to eventually remove the bash binary, they need to have a replacement which handles sh.

Enter dash

Probably not coincidentally, Apple added the dash shell in Catalina. The Debian Almquist Shell (dash) is a minimal implementation of the Posix sh standard and commonly used on other UNIX and Unix-like systems as a stand-in for sh.

Apple also added dash (but not zsh) to the Recovery system in macOS Catalina. While sh is still interpreted by bash in both the Recovery system and the regular system, this is a strong indicator that Apple eventually wants to use dash as the interpreter for sh scripts.

When your scripts which use the #!/bin/sh shebang strictly follow the POSIX sh standard, you should experience no problems when Apple switches to ‘dash-as-sh.’

Tripping over bashisms

However, there are some quirks of the current ‘bash-as-sh’ implementation in macOS that you need to be aware of. When bash stands in as sh, it will nevertheless continue to interpret ’bashisms’—language features available in bash but not sh—without errors.

For example, consider the following script shtest.sh:

#!/bin/sh

if [[ $(true) == $(true) ]]; then
  echo "still good"
else
  echo "nothing is true"
fi

This script declares the #!/bin/sh shebang and it will work fine on macOS with bash-as-sh.

% shtest.sh
still good

However, when you try to run it with zsh-as-sh or dash-as-sh, it will fail.

You can make dash interpret the script instead of bash by switching the shebang to #!/bin/dash. But macOS Catalina has another, new mechanism available. In Catalina, the symbolic link at /var/select/sh determines which shell stands in as sh. By default the link points to /bin/bash:

% readlink /var/select/sh
/bin/bash 

When you change this link to either /bin/zsh or /bin/dash, the respective other shell binary will stand in as sh.

Switch the sh stand-in to dash with:

% sudo ln -sf /bin/dash /var/select/sh

And then run the script again:

% ./shtest.sh
./shtest.sh: 3 ./shtest.sh: [[: not found
nothing is true

When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh.

Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error.

You can switch back to the default bash-as-sh with:

% sudo ln -sf /bin/bash /var/select/sh

Since macOS has been using bash-as-sh for a long time, there may be many such bashisms lurking in your sh scripts. You can change the above symbolic link to test your scripts with dash-as-sh.

Some common ‘bashisms’ are:

  • double square brackets [[ ... ]]
  • here documents and strings (<<< and << operators)
  • double equals operator == for tests

Shellcheck to the rescue

You can also use the shellcheck tool to detect bashisms in your sh scripts:

% shellcheck shtest.sh                                          

In shtest.sh line 3:
if [[ $(true) == $(true) ]]; then
   ^----------------------^ SC2039: In POSIX sh, [[ ]] is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC2039 -- In POSIX sh, [[ ]] is undefined.

When you change the double square brackets for single square brackets, then you get this:

% shellcheck shtest.sh

In shtest.sh line 3:
if [ "$(true)" == "$(true)" ]; then
               ^-- SC2039: In POSIX sh, == in place of = is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC2039 -- In POSIX sh, == in place of = is undefined.

Conclusion

In Catalina Apple started warning us about the eventual demise of bash from macOS. Converting your existing bash scripts and workflows to zsh, sh, or bash v5 is an important first step. But you also need to consider that the behavior of sh scripts will change when Apple replaces the sh interpreter.

Random Desktop Background Color with desktoppr

File this one under: probably useless, but fun.

I recently updated desktoppr with the feature to control the scaling of a custom desktop picture. Because a scaled desktop picture might not cover the entire screen, macOS also allows you to choose a custom color to fill the remaining area. desktoppr v0.3 can be used to control both of these settings.

So, I thought I could use this to just set a random single color as the desktop background, similar to how I did it for Terminal windows.

macOS does not allow you to set no desktop picture. So I created a PNG file that is only a transparent background. This image is basically an invisible desktop picture and all you see on screen is the fill color. Then, you can set any fill color with desktoppr color.

I have added a randombackgroundcolor script to the examples in the desktoppr repository on GitHub.

It’s a bit silly, but also kind of fun. Enjoy!

Advanced Quoting in Shell Scripts

Quoting strings and variable substitutions is a bit of a dark art in shell scripts. It looks simple and straightforward enough, but there are lots of small devils in the details, that can come out and haunt you.

Basics: why we quote strings

In shell scripts (sh, bash, and zsh) you use the equals character = to assign a string value to a variable:

> name=John
> dirpath=/Library

As long as there are no special characters in the literal string, there is no need to quote the string.

When you use the variable, you prefix a $ symbol:

> echo $name
John
> cd $dirpath
> pwd
/Library

When the literal string contains special characters, you need to either escape the special characters with the backslash \ or quote the entire string with either single quotes ' or double quotes ". Space is proverbial ‘killer character’, especially for file paths. (More details in this post.)

name='John Doe'
dirpath="/Library/Application Support"

The difference between single quotes and double quotes is important. Single quotes escape every special character except the single quote itself. A single quoted string of '#$"\!' will represent exactly those characters.

Double quotes escape most characters, except the double quote " the backtick `, the dollar sign $, the backslash \, and the exclamation mark !. (There are slight differences between the shells on this.)

This allows us to use old-style command substitution with backticks and variable substitution (dollar sign) within double quoted strings:

> echo "Hello, $name"
Hello, John Doe
> echo "The Computer Name is `scutil --get ComputerName`"

Though you should be using the $(…) syntax for command substitution instead of backticks `. The parenthesis syntax is more readable and can be nested.

In general, it is a good rule to always quote literal strings. Whether you should use double quotes or single quotes depends on the use case.

Combining literal strings with special characters

Things can start getting complicated when you want special characters with their special functionality. For example, when you want to refer to the path ~/Library/Application Support, you should put it in quotes, because of the space. But when you put the ~ in the quotes, it will not be substituted to the user’s home directory path.

There are a few ways to solve this problem. You could escape the space with a backslash. You could use the $HOME variable instead (but be sure you are in a context where this is set). But the easiest is to move the special character out of the quotes:

dirpath=~"/Library/Application Support"

Quotes in quotes

Sometimes it is necessary to have a set of quotes within quotes. A common situation for MacAdmins is the following osascript:

osascript -e 'display dialog "Hello, World"'

The osascript command can be used to run Apple commands or scripts. Since AppleScript uses double quotes for literal strings, the entire AppleScript command is passed in single quotes. This keep the command string together and the double quotes in single quotes don’t confuse the shell.

This works fine, until you want to do something like this:

computerName=$(scutil --get ComputerName)
newName=$(osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "$computerName")')

Again, we put the AppleScript command in single quotes, so we can use double quotes inside. But now, the single quotes are also blocking the variable substitution and we get the literal $computerName in the dialog.

There are a few solutions out of this, I will demonstrate three:

First, you could close the single quotes before the variable substitution and re-open them after:

osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'$computerName'")'

This will in this form as long as $computerName contains no spaces. This is unlikely as the default computer name is something like Armin's MacBook Pro. The shell will consider that space a separator before a new argument, breaking the AppleScript command into meaningless pieces and failing the osascript command. We can avoid that by putting the substitution itself in double quotes:

osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'"$computerName"'")'

This works and is entirely legal syntax, but not very legible.

Escaping the escape characters

Another solution is to use double quotes for the entire AppleScript command, we can use variable substitution inside. But then we have to deal with the double quotes required for the AppleScript string literal. The good news here is that we can escape those with the backslash:

osascript -e "text returned of (display dialog \"Enter Computer Name\" default answer \"$computerName\")"

This doesn’t win prizes for legibility either, but I consider it an improvement over the previous approach.

Here Docs

The above approaches with work in sh, bash, and zsh. But bash and zsh have another tool available that can work here. The ‘here doc’ syntax can be used to include an entire block of AppleScript code in a bash or zsh script:

#!/bin/bash

computerName=$(scutil --get ComputerName)

newName=$(osascript <<EndOfScript
    text returned of (display dialog "Enter Computer Name" default answer "$computerName")
EndOfScript
)

echo "New name: $newName"

The syntax is a bit weird. The <<EndOfScript says: take all the text until the next appearance of EndOfScript and pipe it into the preceding command, in this case osascript.

The ‘marker’ EndOfScript is entirely arbitrary. Many people choose EOF but I prefer something a little more descriptive. Whatever label you choose the ending marker has to stand alone in its line. This is why the parenthesis ) which closes the command substition $( has to stand alone in the next line.

You can still use variable substitution in a here doc, so the variable $computerName will be substituted before the here doc is piped into osascript.

macOS shell command to create a new Terminal Window

Of course, you can easily create a new Terminal window from the ‘Shell’ menu or by using the ⌘N (or ⌘T) keyboard shortcut. But in some cases, it can be more useful to use a shell command.

New windows created with the keyboard shortcut or from the menu will always have the home directory ~ as the current working directory. What I want, is a new window that defaults to current working directory or a custom directory that I can provide with an argument:

> new           # opens a new terminal window at the current working directory
> new ~/Desktop # opens a new terminal window at ~/Desktop

No luck with AppleScript

After my last success using AppleScript, I thought this would be the best solution again. Unfortunately, this particular piece of the AppleScript dictionary is broken. The make new window or make new tab commands fail with errors and I have tried several combinations.

After some web searching, it looks like this has been broken for a long time. I filed an issue in Feedback Assistant.

You can create a new Terminal window with AppleScript using the do script command in the Terminal dictionary. (Not to be confused with do shell script.) So this AppleScript, sort of does what I want, but seems cumbersome.

tell application "Terminal"
    do script "cd ~/Desktop"
end tell

If you know of a better way to create a new Terminal window or, even better, a Terminal tab with AppleScript, then please let me know. (No UI Scripting solutions – those have their own issues.) I have a few other ideas where this might come in useful.

Enter the open command

During those web searches, I also found suggestions to use the open command, instead:

> open -a Terminal ~/Documents

Will open a new Terminal window with ~/Documents as the working directory. This is already really close to what I wanted.

I created this function in my shell configuration file (bash, zsh):

# creates a new terminal window
function new() {
    if [[ $# -eq 0 ]]; then
        open -a "Terminal" "$PWD"
    else
        open -a "Terminal" "$@"
    fi
}

With this, I can now type

> new Projects/desktoppr

and get a new Terminal window there. This is very useful when combined with the history substitution variable !$ (last argument of previous command):

> mkdir Projects/great_new_tool
> new !$

And an unexpected, but useful side effect is that the new function can also open an ssh session in a new window:

> new ssh://username@computer.example.com

Hope you find this useful, too!

Book Update – Moving to zsh v3

I have pushed an update for the “Moving to zsh” book.

Just a few changes and fixes that have accumulated over the past two weeks. Much of this has been from feedback of readers. Thanks to everyone who sent in their notes.

The update is free if you have already purchased the book. You should get a notification from the Books application to update. (On macOS, I have seen that it can help to delete the local download of the book to force the update.)

If you are enjoying the book, please rate it on the Books store, or (even better) leave a review. These really help, thank you!

Also, please recommend the book to friends, co-workers, and anyone else (not just MacAdmins) who might be facing the zsh transition as they upgrade to Catalina.

The changes in v3 are listed here. This list is also in the ‘Version History’ section in the book. There, you will get links to the relevant section of the book, so you can find the changes quickly.

  • Added a section explaining how to work with upper- or lower-case strings in zsh scripts
  • Added a section explaining the differences in the read built-in command
  • Clarified the section on Connected Variables
  • Fixed file names in the table for Configuration Files and added a note for how to use configuration files with python environments
  • As usual, several typos and clarifications (Thanks to many readers)

Random Terminal Background Colors

In an older post, I showed a trick to get random terminal backgrounds from a selection of colors. Others have used similar approaches.

While I have used this for a long time, the limited number of colors has always annoyed me. And with the introduction of Dark Mode in macOS it seemed just not useful anymore.

Mike’s approach to create the color files with a python script sent me down a rabbit hole to recreate this in Swift. I actually succeeded in creating such a Swift tool, but then, when I worked on connecting the tool with Terminal, I found an even simpler, and arguably better way to do this.

Surprisingly, it involved AppleScript.

Changing the Terminal Background color

Terminal has a powerful AppleScript library, which allows to read and change (among other things) the background color of a Terminal window or tab:

tell application "Terminal"
    get background color of selected tab of window 1
            --> {65535, 65533, 65534}
end tell

The background color is returned as a list of three RGB numbers ranging from 0 to 65535 (216 – 1). You can also set the background color:

tell application "Terminal"
    set background color of selected tab of window 1 to {60000, 45000, 45000}
end tell

This will set the background color of the current window to pastel pink (salmon?).

Armed with this knowledge it is fairly straight forward to write a script that will set the background color to a random color:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    
    set redValue to random number from 0 to maxValue
    set greenValue to random number from 0 to maxValue
    set blueValue to random number from 0 to maxValue
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

You can paste this code in Script Editor and hit the ‘play’ button and watch the Terminal window change. But since we want to use from within Terminal, we will take a different approach: paste the code into your favorite text editor and save it as a text file named randombackground (no extension).

Then open Terminal and change directory to where you save the file and set its executable bit:

> chmod +x randombackground

Now you can run this AppleScript file like any other script file from Terminal:

> ./randombackground

This is fun!

I am not the first to discover and use this, Daniel Jalkut and Erik Barzeski have documented this in 2006.

Enter Dark Mode

Fast forward back to 2018: Along with the rest of macOS, Terminal gained “Dark Mode” in macOS Mojave.

The default “Basic” window profile in Terminal has black text on a white background in light mode and white text on a black background in dark mode. There is some “magic” that happens when the system switches to Dark or Light mode.

However, once we customize the background color (or any other color) the magic does not work any more. When our random backgrounds are too dark in light mode (or vice versa), they don’t really look nice any more, and the text becomes hard to read or completely illegible.

So, we want to change the script to detect dark or light mode and limit the colors accordingly. You can detect dark mode in AppleScript with:

tell application "System Events"
  get dark mode of appearance preferences
end tell

This will return true for dark mode and false for light mode. We modify the script to use just a subrange of all available colors, depending on the mode:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    tell application "System Events"
        set mode to dark mode of appearance preferences
    end tell
    
    if mode then
        set rangestart to 0
        set rangeend to (maxValue * 0.4)
    else
        set rangestart to (maxValue * 0.6)
        set rangeend to maxValue
    end if
    
    set redValue to random number from rangestart to rangeend
    set greenValue to random number from rangestart to rangeend
    set blueValue to random number from rangestart to rangeend
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

When you run this from Terminal for the first time, it may prompt you to allow access to send events to “System Events.” Click ‘OK’ to confirm that:

Automatically setting the background color

Now we can randomize the color by running the command. For simplicity, you may want to put the script file somewhere in your PATH. I put mine in ~/bin/, a folder which a few useful tools and I also added to my PATH(in bash and in zsh).

It is still annoying that it doesn’t happen automatically when we create a new window or tab, but that is exactly what the shell configuration files are for. Add this code to your bash or zsh configuration file.

# random background color
if [[ $TERM_PROGRAM == "Apple_Terminal" ]]; then
    if [[ -x ~/bin/randombackground ]]; then
        ~/bin/randombackground
    fi
fi

Our script will likely fail when the shell is run in any other terminal application or context (such as over ssh). The first if clause checks if the shell is running in Terminal.app. Then the code check to see if the script exists and is executable, and then it executes the script.

This will result in random Terminal colors, matching your choice of dark or light mode.

Note: macOS Catalina added the option to automatically switch the theme depending on the time of day. This script will detect the mode correctly when creating a new window, but Terminal windows and tabs that are already open will retain their color. I am working on a solution…

Upper- or lower-casing strings in bash and zsh

String comparison in most programming languages is case-sensitive. That means that the string 'A' and 'a' are considered different. Humans usually don’t think that way, so there is bound to be trouble and confusion.

If you are looking at single letters, the bracket expansion can be quite useful:

case $input in
    [aA])
        # handle 'a'
        ;;   
    [bB])   
        # handle 'b'
        ;;
    [qQ])
        # handle 'q': quit
        exit 0
        ;;
    *)  
        echo "Option is not available. Please try again"
        ;;
esac

However, for longer strings the bracket expansion gets cumbersome. To cover all case combinations of the word cat you need [cC][aA][tT]. For longer, or unknown strings, it is easier to convert a string to uppercase (all capitals) or lower case before comparing.

sh and bash3

Bash3 and sh have no built-in means to convert case of a string, but you can use the tr tool:

name="John Doe"
# sh
echo $(echo "$name" |  tr '[:upper:]' '[:lower:]' )
john doe

# bash3
echo $(tr '[:upper:]' '[:lower:]' <<< "$name")
john doe

Switch the [:upper:] and [:lower:] arguments to convert to upper case.

There are many other tools available that can provide this functionality, such as awk or sed.

Bash 5

Bash 5 has a special parameter expansion for upper- and lowercasing strings:

name="John Doe"
echo ${name,,}
john doe
echo ${name^^}
JOHN DOE

Zsh

In zsh you can use expansion modifiers:

% name="John Doe"
% echo ${name:l}
john doe
% echo ${name:u}
JOHN DOE

You can also use expansion flags:

% name="John Doe"    
% echo ${(L)name}     
john doe
% echo ${(U)name}
JOHN DOE

In zsh you can even declare a variable as inherently lower case or upper case. This will not affect the contents of the variable, but it will automatically be lower- or uppercased on expansion:

% typeset -l name      
% name="John Doe"
% echo $name        
john doe
% typeset -u name      
% echo $name        
JOHN DOE

Comparing Version strings in zsh

Another excerpt from the book “Moving to zsh.” I found this one so useful, I thought I’d like to share it.

You can get the version of zsh with the ZSH_VERSION variable:

% echo $ZSH_VERSION
5.7.1

And you can get the version of macOS with the sw_vers command:

% sw_vers -productVersion
10.15.1

Comparing version strings is usually fraught with potential errors. Strings are compared by the character code for each character.‘2’ is alphabetically greater than ‘10’ when compared as strings, because the character code for 2 is greater than the character code for 1. So, a string comparison of macOS version numbers will return that 10.9.5 is greater than 10.15.1.

Zsh, however, provides a function is-at-least which helps with version string comparisons.

With a single argument, is-at-least will return if the current zsh version matches or is higher than a given number:

if ! is-at-least 2.6-17; then
  echo "is-at-least is not available"
fi

When you provide two arguments to is-at-least, then the second argument is compared (using version string rules) with the first and needs to match or be higher:

autoload is-at-least
if is-at-least 10.9 $(sw_vers -productVersion); then
  echo "can run Catalina installer"
else
  echo "cannot run Catalina installer"
fi

Note: when used in a script, you will probably have to autoload is-at-least before using it. In an interactive shell, it is often already loaded, because many other autoloader functions will have already loaded it.