I have written a few posts in the past about parsing data from property list files in scripts and Terminal. My usual tool for this is PlistBuddy. However, PlistBuddy’s syntax is… well… eccentric.
Recently, Alexis Bridoux, who is also the main developer on Octory, introduced a command line tool called scout which solves many of the issues I have with PlistBuddy.
For example, you can pipe the output of another command into scout, something you can only convince PlistBuddy to do with some major shell syntax hackery.
The tool can also modify existing files, by changing, adding or deleting keys.
scout can also parse JSON and (non plist) XML files, so it can also stand in as a replacement for jq and xpath. It will also color-code output for property list, XML and JSON files.
I have been using scout interactively in the Terminal for a while now. So far, I have been refraining from using scout in scripts I use for deployment. To use a non-system tool in deployment scripts, you need to ensure the tool is deployed early in the setup process. Then you also have to write your scripts in a way that they will gracefully fail or fallback to PlistBuddy in the edge case where scout is not installed:
scout="/usr/local/bin/scout" if [ ! -x "$scout"]; then echo "could not find scout, exiting..." exit 1 fi realName=$( dscl -plist . read /Users/armin RealName | scout "dsAttrTypeStandard:RealName[0]" )
All of this overhead, adds extra burden to using a tool. The good news is that scout comes as a signed and notarized package installer, which minimizes deployment effort.
I wills be considering scout for future projects. If anyone at Apple is reading this: please hire Alexis and integrate scout or something like it in macOS.
This is an excerpt from my book: “Moving to zsh.” At the MacAdmins Conference Campfire session I received quite a few questions regarding this, so I thought it would be helpful information. You can get a lot more detailed information on “Moving to zsh” in the book!
Calls to the POSIX sh /bin/sh in macOS are handled by /bin/bash in sh compatibility mode. You can verify this by asking sh for its version:
% sh --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
Copyright (C) 2007 Free Software Foundation, Inc.
If Apple plans to eventually remove the bash binary, they need to have a replacement which handles sh.
Enter dash
Probably not coincidentally, Apple added the dash shell in Catalina. The Debian Almquist Shell (dash) is a minimal implementation of the Posix sh standard and commonly used on other UNIX and Unix-like systems as a stand-in for sh.
Apple also added dash (but not zsh) to the Recovery system in macOS Catalina. While sh is still interpreted by bash in both the Recovery system and the regular system, this is a strong indicator that Apple eventually wants to use dash as the interpreter for sh scripts.
When your scripts which use the #!/bin/sh shebang strictly follow the POSIX sh standard, you should experience no problems when Apple switches to ‘dash-as-sh.’
Tripping over bashisms
However, there are some quirks of the current ‘bash-as-sh’ implementation in macOS that you need to be aware of. When bash stands in as sh, it will nevertheless continue to interpret ’bashisms’—language features available in bash but not sh—without errors.
For example, consider the following script shtest.sh:
#!/bin/sh
if [[ $(true) == $(true) ]]; then
echo "still good"
else
echo "nothing is true"
fi
This script declares the #!/bin/sh shebang and it will work fine on macOS with bash-as-sh.
% shtest.sh
still good
However, when you try to run it with zsh-as-sh or dash-as-sh, it will fail.
You can make dash interpret the script instead of bash by switching the shebang to #!/bin/dash. But macOS Catalina has another, new mechanism available. In Catalina, the symbolic link at /var/select/sh determines which shell stands in as sh. By default the link points to /bin/bash:
% readlink /var/select/sh
/bin/bash
When you change this link to either /bin/zsh or /bin/dash, the respective other shell binary will stand in as sh.
Switch the sh stand-in to dash with:
% sudo ln -sf /bin/dash /var/select/sh
And then run the script again:
% ./shtest.sh
./shtest.sh: 3 ./shtest.sh: [[: not found
nothing is true
When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh.
Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error.
You can switch back to the default bash-as-sh with:
% sudo ln -sf /bin/bash /var/select/sh
Since macOS has been using bash-as-sh for a long time, there may be many such bashisms lurking in your sh scripts. You can change the above symbolic link to test your scripts with dash-as-sh.
Some common ‘bashisms’ are:
double square brackets [[ ... ]]
here documents and strings (<<< and << operators)
double equals operator == for tests
Shellcheck to the rescue
You can also use the shellcheck tool to detect bashisms in your sh scripts:
% shellcheck shtest.sh
In shtest.sh line 3:
if [[ $(true) == $(true) ]]; then
^----------------------^ SC2039: In POSIX sh, [[ ]] is undefined.
For more information:
https://www.shellcheck.net/wiki/SC2039 -- In POSIX sh, [[ ]] is undefined.
When you change the double square brackets for single square brackets, then you get this:
% shellcheck shtest.sh
In shtest.sh line 3:
if [ "$(true)" == "$(true)" ]; then
^-- SC2039: In POSIX sh, == in place of = is undefined.
For more information:
https://www.shellcheck.net/wiki/SC2039 -- In POSIX sh, == in place of = is undefined.
In Catalina Apple started warning us about the eventual demise of bash from macOS. Converting your existing bash scripts and workflows to zsh, sh, or bash v5 is an important first step. But you also need to consider that the behavior of sh scripts will change when Apple replaces the sh interpreter.
I recently updated desktoppr with the feature to control the scaling of a custom desktop picture. Because a scaled desktop picture might not cover the entire screen, macOS also allows you to choose a custom color to fill the remaining area. desktoppr v0.3 can be used to control both of these settings.
So, I thought I could use this to just set a random single color as the desktop background, similar to how I did it for Terminal windows.
macOS does not allow you to set no desktop picture. So I created a PNG file that is only a transparent background. This image is basically an invisible desktop picture and all you see on screen is the fill color. Then, you can set any fill color with desktoppr color.
Quoting strings and variable substitutions is a bit of a dark art in shell scripts. It looks simple and straightforward enough, but there are lots of small devils in the details, that can come out and haunt you.
Basics: why we quote strings
In shell scripts (sh, bash, and zsh) you use the equals character = to assign a string value to a variable:
> name=John
> dirpath=/Library
As long as there are no special characters in the literal string, there is no need to quote the string.
When you use the variable, you prefix a $ symbol:
> echo $name
John
> cd $dirpath
> pwd
/Library
When the literal string contains special characters, you need to either escape the special characters with the backslash \ or quote the entire string with either single quotes ' or double quotes ". Space is proverbial ‘killer character’, especially for file paths. (More details in this post.)
The difference between single quotes and double quotes is important. Single quotes escape every special character except the single quote itself. A single quoted string of '#$"\!' will represent exactly those characters.
Double quotes escape most characters, except the double quote " the backtick `, the dollar sign $, the backslash \, and the exclamation mark !. (There are slight differences between the shells on this.)
This allows us to use old-style command substitution with backticks and variable substitution (dollar sign) within double quoted strings:
> echo "Hello, $name"
Hello, John Doe
> echo "The Computer Name is `scutil --get ComputerName`"
Though you should be using the $(…) syntax for command substitution instead of backticks `. The parenthesis syntax is more readable and can be nested.
In general, it is a good rule to always quote literal strings. Whether you should use double quotes or single quotes depends on the use case.
Combining literal strings with special characters
Things can start getting complicated when you want special characters with their special functionality. For example, when you want to refer to the path ~/Library/Application Support, you should put it in quotes, because of the space. But when you put the ~ in the quotes, it will not be substituted to the user’s home directory path.
There are a few ways to solve this problem. You could escape the space with a backslash. You could use the $HOME variable instead (but be sure you are in a context where this is set). But the easiest is to move the special character out of the quotes:
dirpath=~"/Library/Application Support"
Quotes in quotes
Sometimes it is necessary to have a set of quotes within quotes. A common situation for MacAdmins is the following osascript:
osascript -e 'display dialog "Hello, World"'
The osascript command can be used to run Apple commands or scripts. Since AppleScript uses double quotes for literal strings, the entire AppleScript command is passed in single quotes. This keep the command string together and the double quotes in single quotes don’t confuse the shell.
This works fine, until you want to do something like this:
computerName=$(scutil --get ComputerName)
newName=$(osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "$computerName")')
Again, we put the AppleScript command in single quotes, so we can use double quotes inside. But now, the single quotes are also blocking the variable substitution and we get the literal $computerName in the dialog.
There are a few solutions out of this, I will demonstrate three:
First, you could close the single quotes before the variable substitution and re-open them after:
osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'$computerName'")'
This will in this form as long as $computerName contains no spaces. This is unlikely as the default computer name is something like Armin's MacBook Pro. The shell will consider that space a separator before a new argument, breaking the AppleScript command into meaningless pieces and failing the osascript command. We can avoid that by putting the substitution itself in double quotes:
osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'"$computerName"'")'
This works and is entirely legal syntax, but not very legible.
Escaping the escape characters
Another solution is to use double quotes for the entire AppleScript command, we can use variable substitution inside. But then we have to deal with the double quotes required for the AppleScript string literal. The good news here is that we can escape those with the backslash:
osascript -e "text returned of (display dialog \"Enter Computer Name\" default answer \"$computerName\")"
This doesn’t win prizes for legibility either, but I consider it an improvement over the previous approach.
Here Docs
The above approaches with work in sh, bash, and zsh. But bash and zsh have another tool available that can work here. The ‘here doc’ syntax can be used to include an entire block of AppleScript code in a bash or zsh script:
#!/bin/bash
computerName=$(scutil --get ComputerName)
newName=$(osascript <<EndOfScript
text returned of (display dialog "Enter Computer Name" default answer "$computerName")
EndOfScript
)
echo "New name: $newName"
The syntax is a bit weird. The <<EndOfScript says: take all the text until the next appearance of EndOfScript and pipe it into the preceding command, in this case osascript.
The ‘marker’ EndOfScript is entirely arbitrary. Many people choose EOF but I prefer something a little more descriptive. Whatever label you choose the ending marker has to stand alone in its line. This is why the parenthesis ) which closes the command substition $( has to stand alone in the next line.
You can still use variable substitution in a here doc, so the variable $computerName will be substituted before the here doc is piped into osascript.
Of course, you can easily create a new Terminal window from the ‘Shell’ menu or by using the ⌘N (or ⌘T) keyboard shortcut. But in some cases, it can be more useful to use a shell command.
New windows created with the keyboard shortcut or from the menu will always have the home directory ~ as the current working directory. What I want, is a new window that defaults to current working directory or a custom directory that I can provide with an argument:
> new # opens a new terminal window at the current working directory
> new ~/Desktop # opens a new terminal window at ~/Desktop
No luck with AppleScript
After my last success using AppleScript, I thought this would be the best solution again. Unfortunately, this particular piece of the AppleScript dictionary is broken. The make new window or make new tab commands fail with errors and I have tried several combinations.
You can create a new Terminal window with AppleScript using the do script command in the Terminal dictionary. (Not to be confused with do shell script.) So this AppleScript, sort of does what I want, but seems cumbersome.
tell application "Terminal"
do script "cd ~/Desktop"
end tell
If you know of a better way to create a new Terminal window or, even better, a Terminal tab with AppleScript, then please let me know. (No UI Scripting solutions – those have their own issues.) I have a few other ideas where this might come in useful.
Enter the open command
During those web searches, I also found suggestions to use the open command, instead:
> open -a Terminal ~/Documents
Will open a new Terminal window with ~/Documents as the working directory. This is already really close to what I wanted.
I created this function in my shell configuration file (bash, zsh):
# creates a new terminal window
function new() {
if [[ $# -eq 0 ]]; then
open -a "Terminal" "$PWD"
else
open -a "Terminal" "$@"
fi
}
With this, I can now type
> new Projects/desktoppr
and get a new Terminal window there. This is very useful when combined with the history substitution variable!$ (last argument of previous command):
> mkdir Projects/great_new_tool
> new !$
And an unexpected, but useful side effect is that the new function can also open an ssh session in a new window:
I have pushed an update for the “Moving to zsh” book.
Just a few changes and fixes that have accumulated over the past two weeks. Much of this has been from feedback of readers. Thanks to everyone who sent in their notes.
The update is free if you have already purchased the book. You should get a notification from the Books application to update. (On macOS, I have seen that it can help to delete the local download of the book to force the update.)
If you are enjoying the book, please rate it on the Books store, or (even better) leave a review. These really help, thank you!
Also, please recommend the book to friends, co-workers, and anyone else (not just MacAdmins) who might be facing the zsh transition as they upgrade to Catalina.
The changes in v3 are listed here. This list is also in the ‘Version History’ section in the book. There, you will get links to the relevant section of the book, so you can find the changes quickly.
Added a section explaining how to work with upper- or lower-case strings in zsh scripts
Added a section explaining the differences in the read built-in command
Clarified the section on Connected Variables
Fixed file names in the table for Configuration Files and added a note for how to use configuration files with python environments
As usual, several typos and clarifications (Thanks to many readers)
While I have used this for a long time, the limited number of colors has always annoyed me. And with the introduction of Dark Mode in macOS it seemed just not useful anymore.
Mike’s approach to create the color files with a python script sent me down a rabbit hole to recreate this in Swift. I actually succeeded in creating such a Swift tool, but then, when I worked on connecting the tool with Terminal, I found an even simpler, and arguably better way to do this.
Surprisingly, it involved AppleScript.
Changing the Terminal Background color
Terminal has a powerful AppleScript library, which allows to read and change (among other things) the background color of a Terminal window or tab:
tell application "Terminal"
get background color of selected tab of window 1
--> {65535, 65533, 65534}
end tell
The background color is returned as a list of three RGB numbers ranging from 0 to 65535 (216 – 1). You can also set the background color:
tell application "Terminal"
set background color of selected tab of window 1 to {60000, 45000, 45000}
end tell
This will set the background color of the current window to pastel pink (salmon?).
Armed with this knowledge it is fairly straight forward to write a script that will set the background color to a random color:
#!/usr/bin/osascript
on backgroundcolor()
set maxValue to (2 ^ 16) - 1
set redValue to random number from 0 to maxValue
set greenValue to random number from 0 to maxValue
set blueValue to random number from 0 to maxValue
return {redValue, greenValue, blueValue}
end backgroundcolor
set newcolor to backgroundcolor()
tell application "Terminal"
set the background color of the selected tab of window 1 to newcolor
end tell
You can paste this code in Script Editor and hit the ‘play’ button and watch the Terminal window change. But since we want to use from within Terminal, we will take a different approach: paste the code into your favorite text editor and save it as a text file named randombackground (no extension).
Then open Terminal and change directory to where you save the file and set its executable bit:
> chmod +x randombackground
Now you can run this AppleScript file like any other script file from Terminal:
> ./randombackground
This is fun!
I am not the first to discover and use this, Daniel Jalkut and Erik Barzeski have documented this in 2006.
Enter Dark Mode
Fast forward back to 2018: Along with the rest of macOS, Terminal gained “Dark Mode” in macOS Mojave.
The default “Basic” window profile in Terminal has black text on a white background in light mode and white text on a black background in dark mode. There is some “magic” that happens when the system switches to Dark or Light mode.
However, once we customize the background color (or any other color) the magic does not work any more. When our random backgrounds are too dark in light mode (or vice versa), they don’t really look nice any more, and the text becomes hard to read or completely illegible.
So, we want to change the script to detect dark or light mode and limit the colors accordingly. You can detect dark mode in AppleScript with:
tell application "System Events"
get dark mode of appearance preferences
end tell
This will return true for dark mode and false for light mode. We modify the script to use just a subrange of all available colors, depending on the mode:
#!/usr/bin/osascript
on backgroundcolor()
set maxValue to (2 ^ 16) - 1
tell application "System Events"
set mode to dark mode of appearance preferences
end tell
if mode then
set rangestart to 0
set rangeend to (maxValue * 0.4)
else
set rangestart to (maxValue * 0.6)
set rangeend to maxValue
end if
set redValue to random number from rangestart to rangeend
set greenValue to random number from rangestart to rangeend
set blueValue to random number from rangestart to rangeend
return {redValue, greenValue, blueValue}
end backgroundcolor
set newcolor to backgroundcolor()
tell application "Terminal"
set the background color of the selected tab of window 1 to newcolor
end tell
When you run this from Terminal for the first time, it may prompt you to allow access to send events to “System Events.” Click ‘OK’ to confirm that:
Automatically setting the background color
Now we can randomize the color by running the command. For simplicity, you may want to put the script file somewhere in your PATH. I put mine in ~/bin/, a folder which a few useful tools and I also added to my PATH(in bash and in zsh).
It is still annoying that it doesn’t happen automatically when we create a new window or tab, but that is exactly what the shell configuration files are for. Add this code to your bash or zsh configuration file.
# random background color
if [[ $TERM_PROGRAM == "Apple_Terminal" ]]; then
if [[ -x ~/bin/randombackground ]]; then
~/bin/randombackground
fi
fi
Our script will likely fail when the shell is run in any other terminal application or context (such as over ssh). The first if clause checks if the shell is running in Terminal.app. Then the code check to see if the script exists and is executable, and then it executes the script.
This will result in random Terminal colors, matching your choice of dark or light mode.
Note: macOS Catalina added the option to automatically switch the theme depending on the time of day. This script will detect the mode correctly when creating a new window, but Terminal windows and tabs that are already open will retain their color. I am working on a solution…
String comparison in most programming languages is case-sensitive. That means that the string 'A' and 'a' are considered different. Humans usually don’t think that way, so there is bound to be trouble and confusion.
If you are looking at single letters, the bracket expansion can be quite useful:
case $input in
[aA])
# handle 'a'
;;
[bB])
# handle 'b'
;;
[qQ])
# handle 'q': quit
exit 0
;;
*)
echo "Option is not available. Please try again"
;;
esac
However, for longer strings the bracket expansion gets cumbersome. To cover all case combinations of the word cat you need [cC][aA][tT]. For longer, or unknown strings, it is easier to convert a string to uppercase (all capitals) or lower case before comparing.
sh and bash3
Bash3 and sh have no built-in means to convert case of a string, but you can use the tr tool:
name="John Doe"
# sh
echo $(echo "$name" | tr '[:upper:]' '[:lower:]' )
john doe
# bash3
echo $(tr '[:upper:]' '[:lower:]' <<< "$name")
john doe
Switch the [:upper:] and [:lower:] arguments to convert to upper case.
There are many other tools available that can provide this functionality, such as awk or sed.
Bash 5
Bash 5 has a special parameter expansion for upper- and lowercasing strings:
name="John Doe"
echo ${name,,}
john doe
echo ${name^^}
JOHN DOE
% name="John Doe"
% echo ${(L)name}
john doe
% echo ${(U)name}
JOHN DOE
In zsh you can even declare a variable as inherently lower case or upper case. This will not affect the contents of the variable, but it will automatically be lower- or uppercased on expansion:
% typeset -l name
% name="John Doe"
% echo $name
john doe
% typeset -u name
% echo $name
JOHN DOE
Another excerpt from the book “Moving to zsh.” I found this one so useful, I thought I’d like to share it.
You can get the version of zsh with the ZSH_VERSION variable:
% echo $ZSH_VERSION
5.7.1
And you can get the version of macOS with the sw_vers command:
% sw_vers -productVersion
10.15.1
Comparing version strings is usually fraught with potential errors. Strings are compared by the character code for each character.‘2’ is alphabetically greater than ‘10’ when compared as strings, because the character code for 2 is greater than the character code for 1. So, a string comparison of macOS version numbers will return that 10.9.5 is greater than 10.15.1.
Zsh, however, provides a function is-at-least which helps with version string comparisons.
With a single argument, is-at-least will return if the current zsh version matches or is higher than a given number:
if ! is-at-least 2.6-17; then
echo "is-at-least is not available"
fi
When you provide two arguments to is-at-least, then the second argument is compared (using version string rules) with the first and needs to match or be higher:
autoload is-at-least
if is-at-least 10.9 $(sw_vers -productVersion); then
echo "can run Catalina installer"
else
echo "cannot run Catalina installer"
fi
Note: when used in a script, you will probably have to autoload is-at-least before using it. In an interactive shell, it is often already loaded, because many other autoloader functions will have already loaded it.
Ironically, macOS can unarchive xz archives when you double click them in the Finder, but there is no command line tool on macOS to unarchive them. In the previous post, I ran into the same problem, and there you can find instructions on how to install the xz tools on macOS.
Update 2020-03-26: I was unnecessarily complicating this. You can use tar to unarchive this:
tar -xf shellcheck-latest.darwin.x86_64.tar.xz
After downloading and un-archiving, you can manually move the shellcheck binary to a suitable directory. The standard location is /usr/local/bin.
For manual installations, this is it! Much simpler than before. Thank you!
Note: if you want the man page as well, you still need to build it with pandoc from the source.
Build a pkg for managed deployment
If you are a MacAdmin and want to distribute shellcheck with your management system, you will need to build an installer package (pkg).
Instead of copying the binary to /usr/local/bin, place it in a payload folder in a project folder. Then build the pkg with pkgbuild:
And because all of this isn’t really that difficult, I built autopkg recipes for Shellcheck You can find them in my recipe repository or with autopkg search shellcheck. Enjoy!
