Scripting macOS, part 2: The Script File

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

  • Part 1: First Script
  • Part 2: The Script File
  • Part 3: The Code
  • Part 4: Running the Script
  • Part 5: Lists of Commands
  • Part 6: Turning it off and on again

I will publish one part every week over the summer. Enjoy!

The Script File

In the previous part, we created a text file and named it hello.sh. As we have mentioned before, all shell script files are text files.

File Extension

The file extension .sh is entirely optional. When you rename the file to merely hello it will remain a functional script:

> mv hello.sh hello
> ./hello
Hello, World!

When you use the script frequently as a command line tool, typing the additional .sh is cumbersome and does not ‘fit in’ very well with all the other command line tools. It is very common to remove the file extension in these cases.

However, the .sh file extension tells Finder and other applications what kind of file this is. You can assign the .sh file extension to be opened in your favored text editor when you double-click the file in Finder.

To do this, rename the file so it has the extension again:

> mv hello hello.sh

Then select the script file in Finder and open its Info panel (⌘I).

In the ‘General’ section of the info panel, you can see that Finder recognizes this file extension as a shell script.

In the ‘Open with’ section you can tell Finder which application to use to open this file. The popup list will show all applications you have installed that can open .sh files. Select your favorite text editor.

To tell the system to open all files with a .sh extension with your favorite text editor, click on the ‘Change All…’ button here. This will set your favorite text editor as the default application for the .sh file extension.

Some people like to use .bash and .zsh file extensions to further distinguish scripts written specifically in bash or zsh. While this can be useful in some workflows—especially when translating scripts from bash to zsh—it is generally not necessary. The .sh file extension works well for all these scripts. The .bash and .zsh file extensions are also not automatically recognized by the Finder or text editors as shell scripts.

Executable Bit

Aside from the actual script code, a shell script requires one more thing that distinguishes it from any other text file. You have to the tell the system that this file can be executed as a command.

This information is stored in the executable bit of the file privileges or file mode. You can see this information with the long form of the ls command:

> ls -l hello.sh 
-rwxr-xr-x@ 1 armin  staff  hello.sh

The file mode is displayed in the first ten characters in this output: -rwxr-xr-x.

The leading dash designates this item as a file. (A directory will show d, a symbolic link will show l, etc.)

The file access privileges are shown next. There are three sets of three flags or bits. The first three (in this case rwx) are the privileges for the file’s owner. The second set (r-x) are the privileges for group access and the last set (also r-x) shows the privileges for all other user on this system.

The three flags in each set are r, w, and x, in that order. They stand for read (r), write (w) and execute (x) and show the actions the users associated with each set can perform.

The file’s owner can read, write (or change), and execute (or run) this file. Users who are a member of the staff group, can read and execute this file, as well as every other user on this system.

Most documents and files merely contain text and data. The execute bit is not required. When you create a new file, the executable flag is disabled by default:

> touch newfile.txt
> ls -l newfile.txt
-rw-r--r--  1 armin  staff  newfile.txt

Having the execute bit disabled by default is a sensible security measure. You will have to explicitly enable it when building scripts or other executables.

When you are writing shell scripts, you eventually want to execute them. To do this you have to enable the execute bit. When then execute bit is unset, you cannot run the script from the command line.

To demonstrate this—and to gain some familiarity with the command line tools required to manipulate the file mode—disable the executable bit of our hello.sh script:

> chmod -x hello.sh

You can verify that the executable bit has been removed with the ls -l command:

> ls -l hello.sh
-rw-r--r--@ 1 armin  staff  45 Feb 17 14:36 hello.sh

When you now attempt to run the command, it will fail and the shell will tell you the reason: you do not have the right permissions. Without the executable script, your shell script is merely a text file.

> ./hello.sh
zsh: permission denied: ./hello.sh

You can re-set the executable bit with

> chmod +x hello.sh

You do not need write (w) permissions to execute a script, but you require read (r) permissions.

You can remove the executable privilege for group and/or other users for security. Since most Macs are used as single user systems this is not really relevant here. When you create, install, and run shell scripts on shared servers with other Unix or unix-like servers with many users, the proper access privileges may be extremely important. Please consult with the system administrator.

You will usually have to remember to set the executable bit when you create a new script. Some text editors may set it for you when they recognize you are working on a shell script.

Note: The executable bit has a different meaning for directories than it does for files. You can find the details in the chmod man page or in my book ‘macOS Terminal and Shell

Scripting macOS, part 1: First Script

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

  • Part 1: First Script
  • Part 2: The Script File
  • Part 3: The Code
  • Part 4: Running the Script
  • Part 5: Lists of Commands
  • Part 6: Turning it off and on again

I will publish one part every week over the summer. Enjoy!

First Script

When learning programming languages it is tradition that the first program you write displays ‘Hello, World!’

Printing these words to the screen is usually a really simple task. In most programming languages it requires just a single line. Nevertheless, creating a simple program or script like this will teach you a lot about all the other tasks you need to create and run a script.

Before you start, you should create a folder where you store all the scripts you will create. A Documents subfolder is a good location. I have a Projects folder in my home directory where I create subfolders for all my scripting and programming projects. You may already have some structure like this that makes sense for you.

You can create this in Finder and the navigate there in Terminal. But, just to practice working in Terminal, you can create your script project folder in the command line:

> cd ~/Documents
> mkdir ScriptingMacOS
> cd ScriptingMacOS

Where- and however you create this folder, we will refer to it as your script project folder from now on. When you interact with the scripts from the command line, always remember to change the working directory there.

Note: When you know how to use a version control system to manage code, such as git, you can set up or initialize this project folder as repository now. Shell scripts are well suited to be managed with version control systems and I would recommend to use it when building scripts. But explaining the details of git as well as teaching shell scripting would be overwhelming and beyond the scope of this series.

Create your first script

Create a new text file in your favored text editor.

Note: If you do not have a favorite text editor yet, I recommend BBEdit. You can use it for free or pay the license fee to unlock the full feature set. After installing BBEdit, be sure to run ‘Install Command Line Tools…’ from the BBEdit menu.

I will be using the bbedit command line tools in my examples, but other text editors have similar commands.

When you are using a text editor with a command line tool, you can create a new empty file from the command line and open it in the text editor like this:

> bbedit hello.sh

Enter the following text into the text document:

#!/bin/zsh

# Greetings
echo "Hello, World!"

Tip: You can copy and paste the code from this post into the text editor. In general, I approve of this, since it speeds up the process and avoids typing errors. But in the beginning, I would recommend typing out the commands and modifying the script manually. This will engage your brain and memory differently and help you memorize some of the standard steps and commands.

Save the text file you created to a file named hello.sh in the script project folder.

In Terminal, ensure that the working directory is your script project folder. Then enter the following:

> chmod +x hello.sh

This will make your script file executable. You are telling the system this file has code that can be run as a program rather than just data. We will look at what this means in detail later.

Now you can run or execute your script:

> ./hello.sh
Hello, World!

If you get a different result, please verify that you typed the script exactly as given above. Pay extra attention to spaces, quotes and other special characters. Then save the text file and try running the script again:

> ./hello.sh
Hello, World!

Even for a script as simple as this, there is a lot that happened here. We will look at the pieces and steps in detail in the next part.

Get Password from Keychain in Shell Scripts

MacAdmin scripts often require passwords, mostly for interactions with APIs.

It is easiest to store the password in clear text, but that is obviously a terrible solution from a security perspective. You can pass the password as an argument to your script, but that is inconvenient and may still appear in clear text in the ps output or the shell history.

You can obfuscate the password with base64, but that is easily reversible. You can even try to encrypt the password, but since the script needs to be able to decrypt the password, you are just adding a layer of complexity to the problem.

macOS has a keychain, where the user can store passwords and allow applications and processes to retrieve them. We can have our script retrieve a password from a local keychain.

There are limitations to this approach:

  • the password item has to be created in the keychain
  • the user has to approve access to the password at least once
  • the keychain has to be unlocked when item is created and when the script runs—this usually requires the user to be logged in
  • the user and other scripts can find and read the password in the Keychain Access application or with the security tool

Because of these limitations, this approach is not useful for scripts that run without any user interaction, e.g. from a management system. Since the user can go and inspect the key in the Keychain Access is also not well suited for critical passwords and keys.

However, it is quite useful for workflow scripts that you run interactively on your Mac. This approach has the added benefit, that you do not have to remember to remove or anonymize any keys or passwords when you upload a script to GitHub or a similar service.

Note: Mischa used this in his ‘OnAirScanner’ script.

Update: I didn’t remember this, but Graham Pugh has written about this before.

How to Store a Password in the Keychain

Since adding the password to your keychain is a one-time task, you can create the password manually.

Open the Keychain Access application and choose “New Password Item…” from the Menu. Then enter the Keychain Item Name, Account Name and the password into the fields. The “Keychain Item Name” is what we are going to use later to retrieve the password, so watch that you are typing everything correctly.

You can also add the password from the command line with the security command.

> security add-generic-password -s 'CLI Test'  -a 'armin' -w 'password123' 

This will create an item in the Keychain with the name CLI Test and the account name armin and the horribly poor password password123.

How to Retrieve the Password in the Script

To retrieve a password from the keychain in a script, use the security command.

> security find-generic-password -w -s 'CLI Test' -a 'armin'

This will search for an item in the keychain with a name of CLI Test and an account name of armin. When it finds an item that matches the name and account it will print the password.

The first time you run this command, the system will prompt to allow access to this password. Enter your keychain password and click the ‘Always Allow’ button to approve the access.

This will grant the /usr/bin/security binary access to this password. You can see this in the Keychain Access application in the ‘Access Control’ tab for the item.

When you create the item with the security add-generic-password binary, you can add the -T /usr/bin/security option to immediately grant the security binary access.

Whether you grant access through the UI or with the command, keep in mind that a every other script that uses the security binary will also gain access to this password.

For very sensitive passwords, you can just click ‘Allow’ rather than ‘Always Allow.’ Then the script will prompt interactively for access every time. This is more secure, but also requires more user interaction.

Once you have tested that you can retrieve the password in the interactive shell, and you have granted access to the security binary, you can use command substitution in the script to get the password:

cli_password=$(security find-generic-password -w -s 'CLI Test' -a 'armin')

This command might fail for different reasons. The keychain could be locked, or the password cannot be found. (Because it was either changed, deleted or hasn’t been created yet.) You want to catch that error and exit the script when that happens:

pw_name="CLI Test"
pw_account="armin"

if ! cli_password=$(security find-generic-password -w -s "$pw_name" -a "$pw_account"); then
  echo "could not get password, error $?"
  exit 1
fi

echo "the password is $cli_password"

Book Update for Big Sur – Moving to zsh v5

I have pushed an update for the “Moving to zsh” book.

Since I have also published a new book “macOS Terminal and Shell” last month, you might be wondering whether you need both books, or just one.

Moving to zsh” is the book where I documented my journey from using bash in Terminal on macOS to using zsh. Before Apple announced that they would switch from bash to zsh as the default shell with macOS Catalina, I used bash “because it was the deafult.” In this book, I describe how to move from bash to zsh. Because of this, “Moving to zsh” is aimed at a user who is already conformtable using Terminal with bash and is wondering what the change means and how to get some extra features and productivity out of zsh.

macOS Terminal and Shell” is the book for those that have no or little experience with using Terminal and probably don’t even know why bash or zsh matters. Or maybe you have a bit experience, but just don’t feel comfortable because you have the feeling you are not quite understanding what is going on. This book will teach you to use Terminal and the shell with confidence, and it will show how you can configure it to be more productive. Since zsh is the current default shell on macOS Catalina and Big Sur, we will focus on zsh, but explain differences to bash where necessary.

As usual, the update to “Moving to zsh” is free if you have already purchased the book. You should get a notification from the Books application to update. (On macOS, I have seen that it can help to delete the local download of the book to force the update.)

If you are enjoying the book, please rate it on the Books store, or even leave a review. These really help, thank you!

The changes are listed here. This list is also in the ‘Version History’ section in the book. There, you will get links to the relevant section of the book, so you can find the changes quickly.

  • Updated list of other books with ‘macOS Terminal and Shell’
  • Added the vared command (variable editor) as an alternative to read
  • Many typos and other minor corrections and clarifications

Update: desktoppr v0.4

I have posted an update for desktoppr. You can download it from the repository’s releases page.

This update adds no features. It does provide support for the Apple silicon Macs with a Univeral binary and installer pkg.

In my initial testing desktoppr v0.3 worked fine on Apple Silicon Macs even without re-compiling, so I didn’t feel pressure to build and provide a universal binary.

However, since then I have learned that the package installation might trigger Rosetta installation and fail if there is no UI at that point. Also, managing the Desktop picture might happen very early in your deployment workflow, so Rosetta might not be available at that time yet.

Either way, having a universal binary and a properly configured installer pkg will be helpful in either case. If you have to support Apple silicon Macs, be sure to use desktoppr v0.4.

Deploying the Big Sur Installer Application

When you want to provide automated workflows to upgrade to or erase-install macOS Big Sur, you can use the startosinstall tool. You can find this tool inside the “Install macOS Big Sur” application at:

/Applications/Install macOS Big Sur.app/Contents/Resources/startosinstall

Note: Apple calls the “Install macOS *” application “InstallAssistant.” I find this a useful shorthand and will use it.

Before you can use startosinstall, you need to somehow deploy the InstallAssitant on the client system. And since the “Install macOS Big Sur” application is huge (>12GB) it poses its own set of challenges.

Different management systems have different means of deploying software. If you are using Munki (or one of the management systems that has integrated Munki, like SimpleMDM or Workspace One) you can wrap the application in a dmg. Unfortunately, even though “app in a dmg” has been a means of distributing software on macOS for nearly 20 years, most management systems cannot deal with this and expect an installer package (pkg).

You can use pkgbuild to build an installer package from an application, like this:

pkgbuild --component "/Applications/Install macOS Catalina.app" InstallCatalina-10.15.7.pkg

This works for all InstallAssistants up to and including Catalina. With a Big Sur installer application this command will start working, but then fail:

% pkgbuild --component "/Applications/Install macOS Big Sur.app/" InstallBigSur20B29.pkg
pkgbuild: Adding component at /Applications/Install macOS Big Sur.app/
pkgbuild: Inferred install-location of /Applications
pkgbuild: error: Cannot write package to "InstallBigSur20B29.pkg". (The operation couldn’t be completed. File too large)

The reason for this failure is that the Big Sur installer application contains a single file Contents/SharedSupport/SharedSupport.dmg which is larger than 8GB. While a pkg file can be larger than 8GB, there are limitations in the installer package format which preclude individual files in the pkg payload to be larger than that.

When you want to distribute the “Install macOS Big Sur” application to the clients in your fleet, either to upgrade or for an erase-and-install workflow, this limitation introduces some challenges.

You can use Composer with Jamf to create a Jamf dmg style deployment, but that will only work with Jamf Pro. You could further wrap and split the application in different containers, but that will increase the creation and deployment time.

There are a number of solutions. Each with their own advantages and downsides, some supported and recommended by Apple and some… less so. Different management and deployment styles will require different solutions and approaches.

App Deployment with MDM/VPP

When you have your MDM hooked up to Apple Business Manager or Apple School Manager, you can push applications “purchased” in the “Apps and Books” area with MDM commands. This was formerly known as “VPP” (Volume Purchase Program and I will continue to use that name, because “deploy with Apps and Books from Apple Business Manager or Apple School Manager” is just unwieldly and I don’t care what Apple Marketing wants us to call it.

Since the “Install macOS Big Sur” application is available for free on the Mac App Store, you can use VPP to push it to a client from your MDM/management system.

When you do this, the client will not get the full InstallAssistant application, but a ‘stub’ InstallAssistant. This stub is small in size (20-40MB).

The additional resouces required for the actual system upgrade or installation which are GigaBytes worth of data will be loaded when they are needed. It doesn’t matter whether the process is triggered by the user after opeing the application or by using the startosinstall or createinstallmedia tool. Either workflow will trigger the download of the additional resources.

This has the advantage of being a fast initial installation of the InstallAssistant, but then the actual upgrade or re-installation process will take so much longer, because of the large extra download before the actual installation can even begin. For certain deployment workflows, this is an acceptable or maybe even desireable trade-off.

The extra download will use a Caching Server. This approach is recommended and supported by Apple.

Mac App Store and/or System Preferences

For some user-driven deployment styles, having the user download the InstallAssistant themselves can be part of the workflow. This way, the user can control the timing of the large download and make sure they are on a “good” network and the download will not interfere with video conferences or other work.

You can direct then to the Big Sur entry in the Mac App Store with a link. You cannot search for older versions of macOS Installers in the Mac App Store, but Apple has a kbase article with direct links.

You can also use a link that leads a user directly to the Software Update pane in System Preferences and prompts the user to start the download:

# Big Sur
x-apple.systempreferences:com.apple.preferences.softwareupdate?client=bau&installMajorOSBundle=com.apple.InstallAssistant.macOSBigSur

# Catalina
x-apple.systempreferences:com.apple.preferences.softwareupdate?client=bau&installMajorOSBundle=com.apple.InstallAssistant.Catalina

When the InstallAssistant is already installed, this link will open the application. When the Mac is already running a newer version of macOS or doesn’t support the version given, it will display an error.

You can use these links from a script with the open command:

open 'x-apple.systempreferences:com.apple.preferences.softwareupdate?client=bau&installMajorOSBundle=com.apple.InstallAssistant.macOSBigSur'

The downloads initiated this way will use a Caching Server. Linking to the Mac App Store is supported and recommended by Apple. The x-apple.systempreferences links are undocumented.

softwareupdate command

Catalina introduced the --fetch-full-installer option for the softwareupdate command. You can add the --full-installer-version option to get a specific version of the installer, for example 10.15.7.

You can run this command from a managed script on the clients to install the application. The download will use a Caching Server.

This would be a really useful method to automate deployment the InstallAssistant on a client, if it were reliable. However, in my experience and that of many MacAdmins, this command is very fragile and will fail in many circumstances. As of this writing, I have not been able to reliably download a Big Sur InstallAssistant with this command. Most of the time I get

Install failed with error: Update not found 

This approach is often recommended by Apple employees, however it will have to be much more reliable before I will join their recommendation.

Please, use Feedback Assistant, preferably with an AppleSeed for IT account, to communicate your experience with this tool with Apple. If this command were reliable, then it would be my recommended solution for nearly all kinds of deployments.

InstallAssistant pkg

With these solutions so far, we have actually avoided creating an installer package, because we moved the download of the InstallAssistant to the client. A caching server can help with the network load. Nevertheless for some styles of deployments, like schools and universities, using the local management infrastucture (like repositories or distribution points) has great advantages. For this, we need a package installer for the InstallAssistant.

A “magic” download link has been shared frequently in the MacAdmins Slack that downloads an installation package from an Apple URL which installs the Big Sur InstallAssistant.

This pkg from Apple avoids the file size limit for the package payload by not having the big file in the payload and then moving it in the postinstall script. Smart hack.. er… solution!

The URL is a download link from a software update catalog. You can easily find the link for the current version with the SUS Inspector tool.

But it would be really tedious to do this on every update. You, the regular reader, know the “tedious” is a trigger word for me to write a script. In this case it was less writing a script than looting one. Greg Neagle’s installinstallmacos.py had most of the pieces needed to find the InstallAssistant.pkg in the software update catalog and download it. I merely had to put the pieces together somewhat differently.

Nevertheless, I “made” a script that downloads the latest InstallAssistant.pkg for macOS Big Sur. You can then upload this pkg to your management system and distribute it like any other installation package.

It works very much like installinstallmacos.py.

./fetch-installer-pkg.py

When you start the script it will download a lot of data into a content folder in the current working directory, parse through it and determine the Big Sur Installers in the catalog. When it finds more than one installers, it will list them and you can choose one. When it finds only one Installer, it will start downloading that immediately.

You can add the --help option for some extra options (all inherited from installinstallmacos.py.

We will have to wait for the 11.1 release to be sure this actually works as expected, but I am confident we can make it work.

This approach is very likely not supported by Apple. But neither was re-packaging the InstallAssitant from disk in Catalina. This deployment method is likely closer to the supported deployment workflows than some common existing methods.

The download does not use a Caching Server, but since the goal is to obtain a pkg that you can upload to your management server, this is not a big downside.

Big Sur signature verification check

You may have noticed that when you launch the Big Sur InstallAssistant on Big Sur for the first time, it will take a long time to “think” before it actually launches. This is due to a new security feature in Big Sur that verifies the application signature and integrity on first launch. Since this is a “big” application this check takes a while. Unfortunately Big Sur shows no progress bar or other indication. This check occurs when the user double-clicks the app to open it and when you start an upgrade or installation with the startosinstall command.

There does not seem to be a way to skip or bypass this check. You can run startosinstall --usage from a script right after installing the InstallAssistant. This will do nothing really, but force the check to happen. Subsequent launches, either from Finder or with startosinstall will be immediate.

Installomator Updated: v0.3

It’s been more than a month since the last update, and while there has been work on the dev branch, I was quite distracted with other things (like this). The good news is, that there have been quite a few contributions from others! A huge thanks to all who helped make this a better script.

All it took was for me to find some time to put all the contributions together, which I finally found some time for.

What’s new in v0.3:

  • added several new labels for total of 98
  • removed the powershell labels, since the installer is not notarized
  • when run without any arguments, the script now lists all labels
  • changed how zips are expanded because this was broken on Mojave
  • improved logging in some statements
  • several more minor improvements

Get the script and find the instructions on the GitHub repo.

Some of the contributions and requests have not yet been addressed. I believe they will require some more thinking and planning. I would like to approach those in the next version.

If you have any feedback or questions, please join us in the #installomator channel on MacAdmins Slack.

Thanks again to all those who contributed!

Use scout to read Property Lists

I have written a few posts in the past about parsing data from property list files in scripts and Terminal. My usual tool for this is PlistBuddy. However, PlistBuddy’s syntax is… well… eccentric.

Recently, Alexis Bridoux, who is also the main developer on Octory, introduced a command line tool called scout which solves many of the issues I have with PlistBuddy.

For example, you can pipe the output of another command into scout, something you can only convince PlistBuddy to do with some major shell syntax hackery.

So instead of this:

> /usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/armin RealName)


With scout I can use this much clearer syntax:

> dscl -plist . read /Users/armin RealName | scout "dsAttrTypeStandard:RealName[0]"


The tool can also modify existing files, by changing, adding or deleting keys.

scout can also parse JSON and (non plist) XML files, so it can also stand in as a replacement for jq and xpath. It will also color-code output for property list, XML and JSON files.

I have been using scout interactively in the Terminal for a while now. So far, I have been refraining from using scout in scripts I use for deployment. To use a non-system tool in deployment scripts, you need to ensure the tool is deployed early in the setup process. Then you also have to write your scripts in a way that they will gracefully fail or fallback to PlistBuddy in the edge case where scout is not installed:

scout="/usr/local/bin/scout"
if [ ! -x "$scout"]; then
    echo "could not find scout, exiting..."
    exit 1
fi

realName=$( dscl -plist . read /Users/armin RealName | scout "dsAttrTypeStandard:RealName[0]" )


All of this overhead, adds extra burden to using a tool. The good news is that scout comes as a signed and notarized package installer, which minimizes deployment effort.

I wills be considering scout for future projects. If anyone at Apple is reading this: please hire Alexis and integrate scout or something like it in macOS.

About bash, zsh, sh, and dash in macOS Catalina and beyond

This is an excerpt from my book: “Moving to zsh.” At the MacAdmins Conference Campfire session I received quite a few questions regarding this, so I thought it would be helpful information. You can get a lot more detailed information on “Moving to zsh” in the book!

Calls to the POSIX sh /bin/sh in macOS are handled by /bin/bash in sh compatibility mode. You can verify this by asking sh for its version:

% sh --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
Copyright (C) 2007 Free Software Foundation, Inc.

If Apple plans to eventually remove the bash binary, they need to have a replacement which handles sh.

Enter dash

Probably not coincidentally, Apple added the dash shell in Catalina. The Debian Almquist Shell (dash) is a minimal implementation of the Posix sh standard and commonly used on other UNIX and Unix-like systems as a stand-in for sh.

Apple also added dash (but not zsh) to the Recovery system in macOS Catalina. While sh is still interpreted by bash in both the Recovery system and the regular system, this is a strong indicator that Apple eventually wants to use dash as the interpreter for sh scripts.

When your scripts which use the #!/bin/sh shebang strictly follow the POSIX sh standard, you should experience no problems when Apple switches to ‘dash-as-sh.’

Tripping over bashisms

However, there are some quirks of the current ‘bash-as-sh’ implementation in macOS that you need to be aware of. When bash stands in as sh, it will nevertheless continue to interpret ’bashisms’—language features available in bash but not sh—without errors.

For example, consider the following script shtest.sh:

#!/bin/sh

if [[ $(true) == $(true) ]]; then
  echo "still good"
else
  echo "nothing is true"
fi

This script declares the #!/bin/sh shebang and it will work fine on macOS with bash-as-sh.

% shtest.sh
still good

However, when you try to run it with zsh-as-sh or dash-as-sh, it will fail.

You can make dash interpret the script instead of bash by switching the shebang to #!/bin/dash. But macOS Catalina has another, new mechanism available. In Catalina, the symbolic link at /var/select/sh determines which shell stands in as sh. By default the link points to /bin/bash:

% readlink /var/select/sh
/bin/bash 

When you change this link to either /bin/zsh or /bin/dash, the respective other shell binary will stand in as sh.

Switch the sh stand-in to dash with:

% sudo ln -sf /bin/dash /var/select/sh

And then run the script again:

% ./shtest.sh
./shtest.sh: 3 ./shtest.sh: [[: not found
nothing is true

When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh.

Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error.

You can switch back to the default bash-as-sh with:

% sudo ln -sf /bin/bash /var/select/sh

Since macOS has been using bash-as-sh for a long time, there may be many such bashisms lurking in your sh scripts. You can change the above symbolic link to test your scripts with dash-as-sh.

Some common ‘bashisms’ are:

  • double square brackets [[ ... ]]
  • here documents and strings (<<< and << operators)
  • double equals operator == for tests

Shellcheck to the rescue

You can also use the shellcheck tool to detect bashisms in your sh scripts:

% shellcheck shtest.sh                                          

In shtest.sh line 3:
if [[ $(true) == $(true) ]]; then
   ^----------------------^ SC2039: In POSIX sh, [[ ]] is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC2039 -- In POSIX sh, [[ ]] is undefined.

When you change the double square brackets for single square brackets, then you get this:

% shellcheck shtest.sh

In shtest.sh line 3:
if [ "$(true)" == "$(true)" ]; then
               ^-- SC2039: In POSIX sh, == in place of = is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC2039 -- In POSIX sh, == in place of = is undefined.

Conclusion

In Catalina Apple started warning us about the eventual demise of bash from macOS. Converting your existing bash scripts and workflows to zsh, sh, or bash v5 is an important first step. But you also need to consider that the behavior of sh scripts will change when Apple replaces the sh interpreter.

Random Desktop Background Color with desktoppr

File this one under: probably useless, but fun.

I recently updated desktoppr with the feature to control the scaling of a custom desktop picture. Because a scaled desktop picture might not cover the entire screen, macOS also allows you to choose a custom color to fill the remaining area. desktoppr v0.3 can be used to control both of these settings.

So, I thought I could use this to just set a random single color as the desktop background, similar to how I did it for Terminal windows.

macOS does not allow you to set no desktop picture. So I created a PNG file that is only a transparent background. This image is basically an invisible desktop picture and all you see on screen is the fill color. Then, you can set any fill color with desktoppr color.

I have added a randombackgroundcolor script to the examples in the desktoppr repository on GitHub.

It’s a bit silly, but also kind of fun. Enjoy!