Introducing Installomator

As a System Engineer at an Enterprise Reseller, I have to manage and create many Jamf Pro instances.

Some of them are tightly managed and require version control on the OS and the apps. But, many of them are managed less stringently and often the requirement for applications is “install the latest version.”

This is not a statement which management strategy is ‘better.’ There are pros and cons for each. There are situations where either is really not appropriate. You will likely have to use a mixed approach for different pieces of software. When you are doing the first, more controlled deployment strategy, you really want to use AutoPkg and not this script. You can stop reading here.

Apple’s vision of deployment with ‘Automated App Installation’ through MDM (formerly known as VPP) is similar to the ‘less controlled’ strategy. When you install Mac App Store through MDM commands, then you will get the latest version available.

Not all applications are available on the Mac App Store. And even when they are available, installing applications with VPP is still unreliable and hard to debug, or retry when it fails.

If you are managing with the “just install the latest version” philosophy, then you probably have one or more scripts that will download and install the latest version of some software from the vendor’s website. This avoids the overhead work of having to download, repackage and manage every new update in the management system. (This can be automated with AutoPkg, but if you can avoid it entirely…)

When I started thinking about this, we had at least four different scripts. Most of them were internal, but William Smith’s installer script for Microsoft applications was a huge inspiration. it made me thing that you could generalize much of this.

Security Considerations

The main danger when downloading application archives and installers directly from the vendor is that a malicious actor might intercept the traffic or even hijack the servers and replace the download with a manipulated software that contains and or installs malware. Since management processes run with root privileges, we have to extra careful which files and processes are installed.

For user driven installation, Apple introduced GateKeeper, signed applications and Notarization as a way to verify downloaded software before execution. When you download software with a script, then you are bypassing GateKeeper. This is usually considered a benefit, because in a managed deployment we don’t want to scare and annoy a user with the warning dialogs.

But we can use the GateKeeper verification process in our script to verify that the archive, application, or installer is signed and notarized. With the spctl command, we can run the verification from the script without the user interaction.

We can even go one step further than GateKeeper. GateKeeper is happy when a software is signed and notarized with any Apple Developer ID. Since this script is working with a curated list of software, we can verify that the application is actually signed with the expected vendor’s Developer ID.

This will catch situations where someone creates or steals a Developer ID to sign and notarize a manipulated application. Apple can and will eventually block that Developer ID, but there will be a window where the manipulated application may be downloaded and installed. This is not theoretical, but has happened already. (more than once)

Installomator

With these ideas in mind, I started working on a script to unify all these installer scripts. (‘The one to rule them all.’) I may have gone a little overboard, but it turned into Installomator.

You can run Installomator from the command line or from your management system.

> ./Installomator.sh desktoppr

The script requires a single argument. The argument is a label that chooses which application to download and install. (As of now, Installomator can handle 56 applications, you can see a list of applications in the repository.

Please read the readme in the GitHub repository for more details.

Jamf or not

I have tried to keep Installomator generic enough that it can be used with platforms other than Jamf Pro.

However, we will use it with Jamf Pro, and thus I took the opportunity to add some workflows that Jamf is missing.

Drag’n Drop installations

“Drag this app to the Applications folder” is a common instruction found on downloaded dmg or zip archives for the Mac. The fact that Jamf Pro has always required repackaging and cannot directly handle application dmgs or zips is mystifying. Also, highly ironic, since Jamf delivers their own management applications in a disk image.

Nevertheless, Installomator can deal with apps that are downloaded in zip, tbz, and dmg archives.

Blocking Processes

Jamf will also happily attempt to install over a running application. So, Installomator will check for blocking processes and either stop the installation at that time or prompt the user and give them a chance to quit the application. (Yes, this is inspired by the behavior of Munki.)

Vendor update processes

Since Installomator will download and install the latest version of the application from the vendor website, it can be used for updates as well as first installations.

If an application has a built-in update process that can be triggered by the script, This can be used instead for updates. So, for Microsoft applications, when the script detects that the app is already installed, it will run msupdate instead of downloading a full installer. This way the update process will use Microsoft’s optimized thin updates. (Credit to Isaac Ordonez, Mann consulting for the idea and first implementation.)

So far, this is only implemented for Microsoft applications and Google Chrome. (and quite experimental)

Extensible

So far, the script can install 56 different applications or application suites. More application descriptions can be added fairly easily, by adding the proper variables. You can find more detailed explanations in the ReadMe, and of course, the existing applications serve as examples.

Not all applications are suitable to be installed with Installomator. To be able to install an application, the download URL must be accessible without requiring a login, and there must be some, fairly simple process to determine the URL for the latest version.

Installomator will only install the application itself, it will not configure any settings. You will have to use profiles, or additional scripts and installers for that.

When you add an application for your own workflow, please contribute as an issue or pull request! Thank you!

Installomator and AutoPkg

Obviously, much of Installomator’s workflow has been heavily inspired by AutoPkg. I have been using AutoPkg for a long time and provide a repository of recipes. And I plan to continue to use AutoPkg.

As mentioned before, Installomator is not suitable for every type of deployment. If you require control over the versions of the software deployed, then you need to download, re-package and manage the packages in your management system. This is obviously what AutoPkg was designed for.

Also, not every software can be installed with Installomator, mostly because the installer is not available as a direct download. In these cases, AutoPkg will be useful to automate the management and deployment, even when you management style is less controlling.

Going Forward

We have been using Installomator for the past few weeks in our own deployment and with one customer. We are now at a point, where we believe it is stable enough to share it and get feedback from other MacAdmins. (I have already shared it with a few, and many thanks to those that have given valuable feedback.)

We have been using this script with two smaller deployments and want to roll it out to more of our customers. But we probably haven’t hit all the weird edge cases yet. So, proceed with caution.

Consider this a beta release.

(Sidenote: I have tested the script with 10.14.6 and 10.15.x. Because it uses the notarization verification which is available in 10.14.4 and higher it will probably not run well on older macOS versions. Might be possible to adapt it though.)

If you are as excited about the script as we are, please start testing in your environment and provide feedback. But please, as with anything MacAdmin, don’t just go and push the script to hundreds or thousands of devices, but test, test, test first.

Then please provide any enhancements back on the GitHub repository. I have also created an #installomator channel on the MacAdmin Slack for discussion and questions.

Advanced Quoting in Shell Scripts

Quoting strings and variable substitutions is a bit of a dark art in shell scripts. It looks simple and straightforward enough, but there are lots of small devils in the details, that can come out and haunt you.

Basics: why we quote strings

In shell scripts (sh, bash, and zsh) you use the equals character = to assign a string value to a variable:

> name=John
> dirpath=/Library

As long as there are no special characters in the literal string, there is no need to quote the string.

When you use the variable, you prefix a $ symbol:

> echo $name
John
> cd $dirpath
> pwd
/Library

When the literal string contains special characters, you need to either escape the special characters with the backslash \ or quote the entire string with either single quotes ' or double quotes ". Space is proverbial ‘killer character’, especially for file paths. (More details in this post.)

name='John Doe'
dirpath="/Library/Application Support"

The difference between single quotes and double quotes is important. Single quotes escape every special character except the single quote itself. A single quoted string of '#$"\!' will represent exactly those characters.

Double quotes escape most characters, except the double quote " the backtick `, the dollar sign $, the backslash \, and the exclamation mark !. (There are slight differences between the shells on this.)

This allows us to use old-style command substitution with backticks and variable substitution (dollar sign) within double quoted strings:

> echo "Hello, $name"
Hello, John Doe
> echo "The Computer Name is `scutil --get ComputerName`"

Though you should be using the $(…) syntax for command substitution instead of backticks `. The parenthesis syntax is more readable and can be nested.

In general, it is a good rule to always quote literal strings. Whether you should use double quotes or single quotes depends on the use case.

Combining literal strings with special characters

Things can start getting complicated when you want special characters with their special functionality. For example, when you want to refer to the path ~/Library/Application Support, you should put it in quotes, because of the space. But when you put the ~ in the quotes, it will not be substituted to the user’s home directory path.

There are a few ways to solve this problem. You could escape the space with a backslash. You could use the $HOME variable instead (but be sure you are in a context where this is set). But the easiest is to move the special character out of the quotes:

dirpath=~"/Library/Application Support"

Quotes in quotes

Sometimes it is necessary to have a set of quotes within quotes. A common situation for MacAdmins is the following osascript:

osascript -e 'display dialog "Hello, World"'

The osascript command can be used to run Apple commands or scripts. Since AppleScript uses double quotes for literal strings, the entire AppleScript command is passed in single quotes. This keep the command string together and the double quotes in single quotes don’t confuse the shell.

This works fine, until you want to do something like this:

computerName=$(scutil --get ComputerName)
newName=$(osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "$computerName")')

Again, we put the AppleScript command in single quotes, so we can use double quotes inside. But now, the single quotes are also blocking the variable substitution and we get the literal $computerName in the dialog.

There are a few solutions out of this, I will demonstrate three:

First, you could close the single quotes before the variable substitution and re-open them after:

osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'$computerName'")'

This will in this form as long as $computerName contains no spaces. This is unlikely as the default computer name is something like Armin's MacBook Pro. The shell will consider that space a separator before a new argument, breaking the AppleScript command into meaningless pieces and failing the osascript command. We can avoid that by putting the substitution itself in double quotes:

osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'"$computerName"'")'

This works and is entirely legal syntax, but not very legible.

Escaping the escape characters

Another solution is to use double quotes for the entire AppleScript command, we can use variable substitution inside. But then we have to deal with the double quotes required for the AppleScript string literal. The good news here is that we can escape those with the backslash:

osascript -e "text returned of (display dialog \"Enter Computer Name\" default answer \"$computerName\")"

This doesn’t win prizes for legibility either, but I consider it an improvement over the previous approach.

Here Docs

The above approaches with work in sh, bash, and zsh. But bash and zsh have another tool available that can work here. The ‘here doc’ syntax can be used to include an entire block of AppleScript code in a bash or zsh script:

#!/bin/bash

computerName=$(scutil --get ComputerName)

newName=$(osascript <<EndOfScript
    text returned of (display dialog "Enter Computer Name" default answer "$computerName")
EndOfScript
)

echo "New name: $newName"

The syntax is a bit weird. The <<EndOfScript says: take all the text until the next appearance of EndOfScript and pipe it into the preceding command, in this case osascript.

The ‘marker’ EndOfScript is entirely arbitrary. Many people choose EOF but I prefer something a little more descriptive. Whatever label you choose the ending marker has to stand alone in its line. This is why the parenthesis ) which closes the command substition $( has to stand alone in the next line.

You can still use variable substitution in a here doc, so the variable $computerName will be substituted before the here doc is piped into osascript.

Wrangling Pythons

As I noted in my last Weekly News Summary, several open source projects for MacAdmins have completed their transition to Python 3. AutoPkg, JSSImport and outset announced Python 3 compatible versions last week and Munki already had the first Python 3 version last December.

Why?

Apple has included a version of Python 2 with Mac OS X since 10.2 (Jaguar). Python 3.0 was released in 2008 and it was not fully backwards compatible with Python 2. For this reason, Python 2 was maintained and updated alongside Python 3 for a long time. Python 2 was finally sunset on January 1, 2020. Nevertheless, presumably because of the compatibility issues, Apple has always pre-installed Python 2 with macOS and still does so in macOS 10.15 Catalina. With the announcement of Catalina, Apple also announced that in a “future version of macOS” there will be no pre-installed Python of any version.

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. (macOS 10.15 Catalina Release Notes)

This also applies to Perl and Ruby runtimes and other libraries. I will be focussing on Python because it is used more commonly for MacAdmin tools, but most of this post will apply equally to Perl and Ruby. Just mentally replace “Python” for your preferred language.

The final recommendation is what AutoPkg and Munki are following: they are bundling their own Python runtime.

How to get Python

There is a second bullet in the Catalina release notes, though:

Use of Python 2.7 isn’t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS won’t include Python 2.7. Instead, it’s recommended that you run python3 from within Terminal. (51097165)

This is great, right? Apple says there is a built-in Python 3! And it’s pre-installed? Just move all your scripts to Python 3 and you’ll be fine!

Unfortunately, not quite. The python3 binary does exist on a ‘clean’ macOS, but it is only a stub tool, that will prompt a user to download and install the Command Line Developer Tools (aka “Developer Command Line Tools” or “Command Line Tools for Xcode”). This is common for many tools that Apple considers to be of little interest to ‘normal,’ non-developer users. Another common example is git.

Dialog prompting to install the Command Line Tools
Dialog prompting to install the Command Line Tools

When you install Xcode, you will also get all the Command Line Developer Tools, including python3 and git. This is useful for developers, who may want to use Python scripts for build operation, or for individuals who just want to ‘play around’ or experiment with Python locally. For MacAdmins, it adds the extra burden of installing and maintaining either the Command Line Developer Tools or the full Xcode install.

Python Versions, a multitude of Snakes

After installing Xcode or the Command Line Developer Tools, you can check the version of python installed: (versions on macOS 10.15.3 with Xcode 11.3.1)

> python --version    
Python 2.7.16
> python3 --version    
Python 3.7.3

When you go on the download page for Python.org, you will get Python 3.8.1 (as of this writing). But, on that download page, you will also find download links for “specific versions” which include (as of this writing) versions 3.8.1, 3.7.6, 3.6.10, 3.5.9, and the deprecated 2.7.17.

The thing is, that Python isn’t merely split into two major release versions, which aren’t fully compatible with each other, but there are several minor versions of Python 3, which aren’t fully compatible with each other, but are still being maintained in parallel.

Developers (individuals, teams, and organisations) that use Python will often hold on to a specific minor (and sometimes even patch) version for a project to avoid issues and bugs that might appear when changing the run-time.

When you install the latest version of Munki, it will install a copy of the Python framework in /usr/local/munki/ and create a symbolic link to that python binary at /usr/local/munki/python. You can check its version as well:

 % /usr/local/munki/python --version
Python 3.7.4

All the Python code files for Munki will have a shebang (the first line in the code file) of

#!/usr/local/munki/python

This ensures that Munki code files use this particular instance of Python and no other copy of Python that may have been installed on the system.

The latest version of AutoPkg has a similar approach:

> /usr/local/autopkg/python --version    
Python 3.7.5

In both cases the python binary is a symbolic link. This allows the developer to change the symbolic link to point to a different Python framework. The shebangs in the all the code files point to the symbolic link, which can be changed to point to a different Python framework.

This is useful for testing and debugging. Could MacAdmins use this to point both tools to the same Python framework? Should they?

The Bridge to macOS

On top of all these different versions of Python itself, many scripts, apps, and tools written in Python rely on ‘Python modules.’ These are libraries (or frameworks) of code for a certain task, that can be downloaded and included with a Python installation to extend the functionality of Python.

The most relevant of these modules for MacAdmins is the “Python Objective-C Bridge.” This module allows Python code to access and use the native macOS Cocoa and CoreFoundation Frameworks. This not only allows for macOS native GUI applications to be written in Python (e.g. AutoDMG and Munki’s Managed Software Center [update: MSC was re-written in Swift last year]), but also allows short scripts to access system functions. This is sometimes necessary to get a data that matches what macOS applications “see” rather than what the raw unix tools see.

For example, the defaults tool can be used to read the value of property lists on disk. But those might not necessarily reflect the actual preference value an application sees, because that value might be controlled by a different plist file or configuration profile.

(Shameless self-promotion) Learn more about Property lists, Preferences and Profiles

You could build a tool with Swift or Objective-C that uses the proper frameworks to get the “real” preference value. Or you can use Python with the Objective-C bridge:

#!/usr/bin/python
from Foundation import CFPreferencesCopyAppValue
print CFPreferencesCopyAppValue("idleTime", "com.apple.screensaver")

Three simple lines of Python code. This will work with the pre-installed Python 2.7, because Apple also pre-installs the Python Objective-C bridge with that. When you try this with the Developer Tools python3 you get an error:

ModuleNotFoundError: No module named 'Foundation'

This is because the Developer Tools do not include the Objective-C bridge in the installation. You could easily add it with:

> sudo python3 -m pip install pyobjc

But again, while this command is “easy” enough for a single user on a single Mac, it is just the beginning of a Minoan labyrinth of management troubles.

Developers and MacAdmins, have to care about the version of the Python they install, as well as the list of modules and their versions, for each Python version.

It is as if the Medusa head kept growing more smaller snakes for every snake you cut off.

(Ok, I will ease off with Greek mythology metaphors.)

You can get a list of modules included with the AutoPkg and the Munki project with:

> /usr/local/munki/python -m pip list
> /usr/local/autopkg/python -m pip list

You will see that not only do Munki and AutoPkg include different versions of Python, but also a different list of modules. While Munki and AutoPkg share many modules, their versions might still differ.

Snake Herding Solutions

Apple’s advice in the Catalina Release Notes is good advice:

It’s recommended that you bundle the runtime within the app.

Rather than the MacAdmin managing a single version of Python and all the modules for every possible solution, each tool or application should provide its own copy of Python and its required modules.

If you want to build your own Python bundle installer, you can use this script from Greg Neagle.

This might seem wasteful. A full Python 3 Framework uses about 80MB of disk space, plus some extra for the modules. But it is the safest way to ensure that the tool or application gets the correct version of Python and all the modules. Anything else will quickly turn into a management nightmare.

This is the approach that Munki and AutoPkg have chosen. But what about smaller, single script solutions? For example simple Python scripts like quickpkg or prefs-tool?

Should I bundle my own Python framework with quickpkg or prefs-tool? I think that would be overkill and I am not planning to do that. I think the solution that Joseph Chilcote chose for the outset tool is a better approach for less complex Python scripts.

In this case, the project is written to run with Python 3 and generic enough to not require a specific version or extra modules. An admin who wants to use this script or tool, can change the shebang (the first line in the script) to point to either the Developer Tool python3, the python3 from the standard Python 3 installer or a custom Python version, such as the Munki python. A MacAdmin would have to ensure that the python binary in the shebang is present on the Mac when the tool runs.

You can also choose to provide your organization’s own copy Python with your chosen set of modules for all your management Python scripts and automations. You could build this with the relocatable Python tool and place it in a well-known location the clients. When updates for the Python run-time or modules are required, you can build and push them with your management system. (Thanks to Nathaniel Strauss for pointing out this needed clarifying.)

When you build such scripts and tools, it is important to document which Python versions (and module versions) you have tested the tool with.

(I still have to do that for my Python tools.)

What about /usr/bin/env python?

The env command will determine the path to the python binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the python binary is unknown.

This is useful when developers want to use the same script in different environments across different computers, user accounts, and platforms. However, this renders the actual version of python that will interpret the script completely unpredictable.

Not only is it impossible to predict which version of Python will interpret a script, but you cannot depend on any modules being installed (or their versions) either.

For MacAdmin management scripts and tools, a tighter control is necessary. You should use fixed, absolute paths in the shebang.

Conclusion

Managing Python runtimes might seem like a hopeless sisyphean task. I believe Apple made the right choice to not pre-install Python any more. Whatever version and pre-selection of module versions Apple would have chosen, it would only have been the correct combination for a few Python solutions and developers.

While it may seem wasteful to have a multitude of copies of the Python frameworks distributed through out the system, it is the easiest and most manageable solution to ensure that each tool or application works with the expected combination of run-time and modules.

Getting the current user in macOS – Update

I recently posted an article on how to get the current user in macOS. If you read through that entire post, it’ll be obvious that it was actually an excuse to write about the deprecation of python 2 and /bin/bash and the situation with sh ‘compatible’ code.

The ‘tl;dr’ of that post was to stop using the python one-liner and use this scutil based one-liner, based on Erik Berglund’s post, instead.

zsh and bash scripts

loggedInUser=$( scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

sh scripts

loggedInUser=$( echo "show State:/Users/ConsoleUser" | scutil | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

From now on, I will call these the ‘SystemConfiguration’ solution (this also includes the python one-liner). Both the python and the scutil based one-liners go through the SystemConfiguration framework to get at the desired information: which user is using the ‘Console?’

The ‘console’ in this case is an abstract definition, which encompasses the screen output and most inputs. I could not find a clear definition of what the ‘Console’ is exactly on macOS. (If you can point me to one, I’d be grateful.) But it serves to think of the console as all the processes, inputs and outputs attached to a user’s login session.

The other one-liner

There is another solution that keeps popping up in MacAdmins Slack and blog posts.

loggedInUser=$(stat -f %Su /dev/console)

(The stat solution.) This certainly looks much shorter and simpler. But does it return the same in all situations? Is it actually the same?

Testing the Edge

I have done some testing with Fast User Switching and a second remote Screen Sharing session. These are the weird edge cases for which the original python one-liner was devised.

I had a remote ssh session open from my main Mac while I logged two test users in and out of a second test Mac. I ran the commands for the SystemConfiguration and stat solutions in the ssh session, and compared the return values.

In (nearly) all multiple user sessions combinations that I tested, both the SystemConfiguration and the stat solution return the same value.

It is worth pointing out, that when you have two users with graphical sessions at the same time, neither of the two solutions will actually return the “currently” active user session, but the user owning the sessions that was last logged in to. This is especially relevant when someone use Screen Sharing to create a remote graphical session on an Mac. In this case both methods will return the last user that logged in, even though both users currently have active sessions.

How the stat solution works

The stat command shows the user owning the /dev/console file. When we look at that ‘file’ with the ls command:

> ls -al /dev/console  
crw-------  1 armin  staff    /dev/console

We get a file type (the first character of the listing) of ‘c’ which represents a ‘character special file.’ All the ‘files’ in /dev are of this file type. When you look closely you will ‘files’ in here representing the disks connected to your Mac (disk*), the Bluetooth devices (cu.*), and some other weird ‘files.’

This stems back to the Unix underpinnings of macOS where everything has a representation in the file system hierarchy, whether it is actual data on a disk or not. The ‘files’ we are seeing here are an abstraction of devices and other objects in the system.

With that background in mind, it should be clear that we are looking at the same data from the system, abstracted in different ways. Either through the SystemConfiguration framework or the Unix file system. So, we get the same data either way, right?

The difference at login window

Not quite. I found one difference. When the Mac is at the login window, the SystemConfiguration solution returns an empty string and the stat solution returns root.

This is not the entire truth, though. When you look closely at the SystemConfiguration solution, you will that there is a clause, in both the python and scutil versions, that catches a return value of loginwindow and returns and empty string instead.

When you look at the raw output of the scutil command used, we actually get quite a lot of information:

> scutil <<< "show State:/Users/ConsoleUser"
<dictionary> {
  GID : 20
  Name : armin
  SessionInfo : <array> {
    0 : <dictionary> {
      kCGSSessionAuditIDKey : 100009
      kCGSSessionGroupIDKey : 20
      kCGSSessionIDKey : 257
      kCGSSessionLoginwindowSafeLogin : FALSE
      kCGSSessionOnConsoleKey : TRUE
      kCGSSessionSystemSafeBoot : FALSE
      kCGSSessionUserIDKey : 501
      kCGSSessionUserNameKey : armin
      kCGSessionLoginDoneKey : TRUE
      kCGSessionLongUserNameKey : Armin Briegel
      kSCSecuritySessionID : 100009
    }
  }
  UID : 501
}

The awk statement after the scutil command filters out the information we need:

awk '/Name :/ && ! /loginwindow/ { print $3 }' 

This will return lines containing Name : and not containing loginwindow. In other words, if the scutil command returns loginwindow in the line, the entire scutil ... | awk ... construct will return nothing or an empty string.

When you leave out the extra awk clause to filter the loginwindow return value, you will get loginwindow while the login window is shown:

scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/  { print $3 }'

The edge of the edge case

So, the SystemConfiguration solution returns loginwindow or nothing when the Mac is sitting at a login window and the stat solution returns root. Not a big deal right? I just have to consider this difference when I test whether a user is logged in?

I found one edge case (to the edge case) where this difference matters: the stat solution will return root when at the login window and when the root account or System Administrator is logged in.

Now, enabling the root account and logging in and working as root is most definitely not recommended on macOS. So, there is an argument that this edge case should not occur.

But users have a way of ignoring these kind of recommendations. The SystemConfiguration solution will allow your scripts to actually distinguish between a Mac sitting at the login window and the (admittedly rare but extremely dangerous) situation where a user is logged in as root.

Conclusions

So, in most cases the stat solution is identical to the SystemConfiguration solution. It has the advantage of being shorter and easier to read.

When you anticipate that you might need to distinguish between the computer being at the login window and a root user being logged in, or when you are just paranoid about catching all edge cases, the SystemConfiguration solution is more complete.

If in doubt, use the SystemConfiguration solution.

Random Terminal Background Colors

In an older post, I showed a trick to get random terminal backgrounds from a selection of colors. Others have used similar approaches.

While I have used this for a long time, the limited number of colors has always annoyed me. And with the introduction of Dark Mode in macOS it seemed just not useful anymore.

Mike’s approach to create the color files with a python script sent me down a rabbit hole to recreate this in Swift. I actually succeeded in creating such a Swift tool, but then, when I worked on connecting the tool with Terminal, I found an even simpler, and arguably better way to do this.

Surprisingly, it involved AppleScript.

Changing the Terminal Background color

Terminal has a powerful AppleScript library, which allows to read and change (among other things) the background color of a Terminal window or tab:

tell application "Terminal"
    get background color of selected tab of window 1
            --> {65535, 65533, 65534}
end tell

The background color is returned as a list of three RGB numbers ranging from 0 to 65535 (216 – 1). You can also set the background color:

tell application "Terminal"
    set background color of selected tab of window 1 to {60000, 45000, 45000}
end tell

This will set the background color of the current window to pastel pink (salmon?).

Armed with this knowledge it is fairly straight forward to write a script that will set the background color to a random color:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    
    set redValue to random number from 0 to maxValue
    set greenValue to random number from 0 to maxValue
    set blueValue to random number from 0 to maxValue
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

You can paste this code in Script Editor and hit the ‘play’ button and watch the Terminal window change. But since we want to use from within Terminal, we will take a different approach: paste the code into your favorite text editor and save it as a text file named randombackground (no extension).

Then open Terminal and change directory to where you save the file and set its executable bit:

> chmod +x randombackground

Now you can run this AppleScript file like any other script file from Terminal:

> ./randombackground

This is fun!

I am not the first to discover and use this, Daniel Jalkut and Erik Barzeski have documented this in 2006.

Enter Dark Mode

Fast forward back to 2018: Along with the rest of macOS, Terminal gained “Dark Mode” in macOS Mojave.

The default “Basic” window profile in Terminal has black text on a white background in light mode and white text on a black background in dark mode. There is some “magic” that happens when the system switches to Dark or Light mode.

However, once we customize the background color (or any other color) the magic does not work any more. When our random backgrounds are too dark in light mode (or vice versa), they don’t really look nice any more, and the text becomes hard to read or completely illegible.

So, we want to change the script to detect dark or light mode and limit the colors accordingly. You can detect dark mode in AppleScript with:

tell application "System Events"
  get dark mode of appearance preferences
end tell

This will return true for dark mode and false for light mode. We modify the script to use just a subrange of all available colors, depending on the mode:

#!/usr/bin/osascript

on backgroundcolor()
    set maxValue to (2 ^ 16) - 1
    tell application "System Events"
        set mode to dark mode of appearance preferences
    end tell
    
    if mode then
        set rangestart to 0
        set rangeend to (maxValue * 0.4)
    else
        set rangestart to (maxValue * 0.6)
        set rangeend to maxValue
    end if
    
    set redValue to random number from rangestart to rangeend
    set greenValue to random number from rangestart to rangeend
    set blueValue to random number from rangestart to rangeend
    
    return {redValue, greenValue, blueValue}
end backgroundcolor

set newcolor to backgroundcolor()

tell application "Terminal"
    set the background color of the selected tab of window 1 to newcolor
end tell

When you run this from Terminal for the first time, it may prompt you to allow access to send events to “System Events.” Click ‘OK’ to confirm that:

Automatically setting the background color

Now we can randomize the color by running the command. For simplicity, you may want to put the script file somewhere in your PATH. I put mine in ~/bin/, a folder which a few useful tools and I also added to my PATH(in bash and in zsh).

It is still annoying that it doesn’t happen automatically when we create a new window or tab, but that is exactly what the shell configuration files are for. Add this code to your bash or zsh configuration file.

# random background color
if [[ $TERM_PROGRAM == "Apple_Terminal" ]]; then
    if [[ -x ~/bin/randombackground ]]; then
        ~/bin/randombackground
    fi
fi

Our script will likely fail when the shell is run in any other terminal application or context (such as over ssh). The first if clause checks if the shell is running in Terminal.app. Then the code check to see if the script exists and is executable, and then it executes the script.

This will result in random Terminal colors, matching your choice of dark or light mode.

Note: macOS Catalina added the option to automatically switch the theme depending on the time of day. This script will detect the mode correctly when creating a new window, but Terminal windows and tabs that are already open will retain their color. I am working on a solution…

Upper- or lower-casing strings in bash and zsh

String comparison in most programming languages is case-sensitive. That means that the string 'A' and 'a' are considered different. Humans usually don’t think that way, so there is bound to be trouble and confusion.

If you are looking at single letters, the bracket expansion can be quite useful:

case $input in
    [aA])
        # handle 'a'
        ;;   
    [bB])   
        # handle 'b'
        ;;
    [qQ])
        # handle 'q': quit
        exit 0
        ;;
    *)  
        echo "Option is not available. Please try again"
        ;;
esac

However, for longer strings the bracket expansion gets cumbersome. To cover all case combinations of the word cat you need [cC][aA][tT]. For longer, or unknown strings, it is easier to convert a string to uppercase (all capitals) or lower case before comparing.

sh and bash3

Bash3 and sh have no built-in means to convert case of a string, but you can use the tr tool:

name="John Doe"
# sh
echo $(echo "$name" |  tr '[:upper:]' '[:lower:]' )
john doe

# bash3
echo $(tr '[:upper:]' '[:lower:]' <<< "$name")
john doe

Switch the [:upper:] and [:lower:] arguments to convert to upper case.

There are many other tools available that can provide this functionality, such as awk or sed.

Bash 5

Bash 5 has a special parameter expansion for upper- and lowercasing strings:

name="John Doe"
echo ${name,,}
john doe
echo ${name^^}
JOHN DOE

Zsh

In zsh you can use expansion modifiers:

% name="John Doe"
% echo ${name:l}
john doe
% echo ${name:u}
JOHN DOE

You can also use expansion flags:

% name="John Doe"    
% echo ${(L)name}     
john doe
% echo ${(U)name}
JOHN DOE

In zsh you can even declare a variable as inherently lower case or upper case. This will not affect the contents of the variable, but it will automatically be lower- or uppercased on expansion:

% typeset -l name      
% name="John Doe"
% echo $name        
john doe
% typeset -u name      
% echo $name        
JOHN DOE

Comparing Version strings in zsh

Another excerpt from the book “Moving to zsh.” I found this one so useful, I thought I’d like to share it.

You can get the version of zsh with the ZSH_VERSION variable:

% echo $ZSH_VERSION
5.7.1

And you can get the version of macOS with the sw_vers command:

% sw_vers -productVersion
10.15.1

Comparing version strings is usually fraught with potential errors. Strings are compared by the character code for each character.‘2’ is alphabetically greater than ‘10’ when compared as strings, because the character code for 2 is greater than the character code for 1. So, a string comparison of macOS version numbers will return that 10.9.5 is greater than 10.15.1.

Zsh, however, provides a function is-at-least which helps with version string comparisons.

With a single argument, is-at-least will return if the current zsh version matches or is higher than a given number:

if ! is-at-least 2.6-17; then
  echo "is-at-least is not available"
fi

When you provide two arguments to is-at-least, then the second argument is compared (using version string rules) with the first and needs to match or be higher:

autoload is-at-least
if is-at-least 10.9 $(sw_vers -productVersion); then
  echo "can run Catalina installer"
else
  echo "cannot run Catalina installer"
fi

Note: when used in a script, you will probably have to autoload is-at-least before using it. In an interactive shell, it is often already loaded, because many other autoloader functions will have already loaded it.

Install shellcheck binary on macOS (updated)

A few months back I wrote a post on how to compile and build an installer for the shellcheck binary for macOS.

Just a few weeks later, the shellcheck project added a download for a pre-compiled binary for macOS. You can now download the binary with this link:

https://shellcheck.storage.googleapis.com/shellcheck-latest.darwin.x86_64.tar.xz

Ironically, macOS can unarchive xz archives when you double click them in the Finder, but there is no command line tool on macOS to unarchive them. In the previous post, I ran into the same problem, and there you can find instructions on how to install the xz tools on macOS.

Update 2020-03-26: I was unnecessarily complicating this. You can use tar to unarchive this:

tar -xf shellcheck-latest.darwin.x86_64.tar.xz

After downloading and un-archiving, you can manually move the shellcheck binary to a suitable directory. The standard location is /usr/local/bin.

For manual installations, this is it! Much simpler than before. Thank you!

Note: if you want the man page as well, you still need to build it with pandoc from the source.

Build a pkg for managed deployment

If you are a MacAdmin and want to distribute shellcheck with your management system, you will need to build an installer package (pkg).

Instead of copying the binary to /usr/local/bin, place it in a payload folder in a project folder. Then build the pkg with pkgbuild:

% mkdir -p ShellcheckPkg/payload
% cp ~/Downloads/shellcheck-latest/shellcheck ShellcheckPkg/payload
% pkgbuild --root ShellcheckPkg/payload --identifier com.example.shellcheck --version 0.7.0 --install-location /usr/local/bin shellcheck-0.7.0.pkg

Replace the 0.7.0 with the actual version number.

Automated Package creation with autopkg

And because all of this isn’t really that difficult, I built autopkg recipes for Shellcheck You can find them in my recipe repository or with autopkg search shellcheck. Enjoy!

Associative arrays in zsh

This is an excerpt from my book “Moving to zsh” which is available for order on the Apple Books Store.

One of the advantages of zsh over bash 3 is the support of “associative arrays,” a data structure known as hash tables or dictionaries in other languages.

In associative arrays, you can store a piece of data, or value with an identifying ‘key’. For example, the associative array userinfo has multiple values, each identified with a key:

% echo $userinfo[name]
armin
% echo $userinfo[shell]
bash
% echo $userinfo[website]
scriptingosx.com

Note: bash 4 also added associative arrays, but they are implemented slightly differently.

Creating associative arrays

In zsh, before you can use a variable as an associative array, you have to declare it as one with

declare -A userinfo

This will tell the shell that the userinfo variable is an associative array. You can also use typeset -A as an alternative syntax. You can verify the type of the variable:

% echo ${(t)userinfo}
association

You can then set the key-value pairs of the userinfo associative array individually:

userinfo[name]="armin"
userinfo[shell]=bash
userinfo[website]="scriptingosx.com"

When you set the value for an existing key again, it will overwrite the existing value:

% echo $userinfo[shell]
bash
% userinfo[shell]=zsh
% echo $userinfo[shell]
zsh

Setting the values for each key is useful in some situations, but can be tedious. You can also set the entire associative array at once. There are two syntaxes for this in zsh:

userinfo=( name armin shell zsh website scriptingosx.com )

This format follows the format ( key1 value1 key2 value2 ...). The other syntax is more verbose and expressive:

userinfo=( [name]=armin [shell]=zsh [website]="scriptingosx.com" )

When you set the associative array variable this way, you are overwriting the entire array. For example, if you set the userinfo for ‘armin’ like above and then set it later like this, the website key and value pair will have been overwritten as well:

% userinfo=( [name]=beth [shell]=zsh )
% if [[ -z $userinfo[website] ]]; then echo no value; fi
no value

If you want to partially overwrite an existing associative array, while leaving the other key/value pairs intact, you can use the += operator:

% userinfo+=( [shell]=fish [website]=fishshell.com )
% echo $userinfo[name]                                           
beth
% echo $userinfo[shell]
fish
% echo $userinfo[website]
fishshell.com

To clear an associative array, you can use:

% userinfo=( )

Retrieving data from an associative array

We have already seen you can get the value for a given key with the ‘subscript’ notation:

% echo $userinfo[name]                                           
beth

When you access the $userinfo variable directly, you will get a normal array of the value:

% echo $userinfo
beth fish fishshell.com

You can also get an array of the keys with this syntax:

% echo ${(k)userinfo}
name shell website

or a list of both keys and values:

% echo ${(kv)userinfo} 
website fishshell.com shell fish name beth

You can use this to copy the data from one associative array to another:

% declare -A otherinfo
% otherinfo=( ${(kv)userinfo )
% echo $otherinfo[name]
beth

You can also use this to loop through all the keys and values of an associated array:

for key value in ${(kv)userinfo}; do
    echo "$key -> $value"
done

#output
website -> fishshell.com
shell -> fish
name -> beth

Limitations

Associative arrays have their uses, but are not as powerful as dictionaries in more powerful languages. In zsh, you cannot nest associative arrays in normal arrays, which limits their use for complex data structures.

There is also no functionality to transfer certain file formats, like XML or property lists directly in to associative arrays or back.

Shell scripting was never designed for complex data structures. When you encounter these limitations, you should move “up” to a higher level language, such as Python or Swift.

Get Current User in Shell Scripts on macOS

…or, how to deal with deprecated bash and python…

There are many solutions to get the current logged in user in macOS to use in a shell script. However, the semi-official, “sanctioned” method has always involved a rather elaborate python one-liner, originally published by Ben Toms:

loggedInUser=$(/usr/bin/python -c 'from SystemConfiguration import SCDynamicStoreCopyConsoleUser; import sys; username = (SCDynamicStoreCopyConsoleUser(None, None, None) or [None])[0]; username = [username,""][username in [u"loginwindow", None, u""]]; sys.stdout.write(username + "\n");')

Update, because this a FAQ:

There are various other solutions to get the current user which use stat, who, or last commands. These will work in most situations. But there are edge cases, mostly with Fast User Switching, where these methods don’t return the correct user.

From python to scutil

While this has worked wonderfully and more reliably than other solutions, it always looked inelegant, and added a dependency on the python binary. This used to be fine, as the python binary shipped with macOS. But in the macOS Catalina release notes, we have learned that some, yet undefined, future version of macOS will not include the Python, Ruby, and Perl interpreters any more.

With prescient timing, Erik Berglund figured out a different method, which still accesses the same system framework, but through the scutil command rather than the PyObjC bridge.
Over time, different contributors in the MacAdmins Slack have optimized this command:

loggedInUser=$( scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

Sidenote: the scutil version runs 15–20x faster than the python version. Speed is not usually a major consideration for MacAdmin scripts, but it’s a nice improvement.

From bash to sh

This works great in bash and zsh. However, the built-in bash has been deprecated as well. (I have talked about this, a lot.)

While zsh is a great replacement for bash for interactive terminal use and scripting on the ‘full’ macOS system, there are environments (Recovery) where zsh is not present.

The only shell Mac Admins can rely on in all contexts is now /bin/sh.

Because of this I recommend using /bin/sh for installer scripts (pre- and postinstall scripts in pkgs).

The Posix sh standard does not include the ‘here string’ <<< used in Erik’s command. Nevertheless, when you use the above construct in sh on macOS it will still work fine. This is because sh on macOS is actually handled by bash in sh compatibility mode.

And while bash in sh compatibility mode will recreate the odd syntax and quirks of sh for compatibility, it will also happily run ‘bashims’ that don’t even exist in sh. E.g. double brackets [[...]] or here strings <<<.

As long as bash is taking care of sh scripts, you will be fine. Myself, I would not have noticed this if I had not ran a sh script through shellcheck. Shellcheck told me that Posix sh should not understand the here string:

loggedInUser=$( scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/ && ! /loginwindow/ { print $3 }' )
                       ^-^ SC2039: In POSIX sh, here-strings are undefined.

But we’re fine on macOS, because sh is really bash right? No need to worry…

Predicting the demise of bash

We don’t yet know when, but the /bin/bash binary will eventually be removed from macOS. The version included with macOS has not been updated since 2014. It is unlikely that the twelve year old bash v3.2 will receive patches for security vulnerabilities in the future.

With the switch to zsh as the default shell in Catalina Apple is laying the groundwork to remove bash v3.2 from the system in a future release. cron and login hooks were infamously deprecated in macOS years ago and are still around, so we could be looking at years. But my suspicion is that bash is just one security vulnerability away from being removed.

With the High Sierra and Mojave updates, Apple has shown that they do not need to wait for major releases to remove or add functionality or restrictions to the system.

I believe /bin/bash has more than a few months still. However, Apple’s messaging on this switch is uncharacteristically strong. It would not be terribly surprising if the Catalina Spring Update in March 2020 removed bash.

The amount of old installer packages which would break on such a change is terrifying. Apple might be willing to pay this price, to avoid a publicized security vulnerability. A severe bash vulnerability would definitely get a lot of publicity. Remember Shellshock? That was the last time bash v3.2 was patched.

Instead of removing /bin/bash Apple might be able to replace it with zsh in bash emulation mode. So far, I have found no signs for this. While it wouldn’t be perfect, it would be a safer move.

Python 2.7 will still get updates until the end of 2019. There are also some commands and tools in the system written in python 2.7. It will be harder for Apple to migrate these to other languages, so I think Python has a longer countdown clock. But it cannot hurt to start preparing now.

Whither sh?

Whenever /bin/bash is removed, what will happen with /bin/sh at that point? The answer can be deduced from the support article on the shell switch.

How to test your shell scripts

To test script compatibility with Bourne-compatible shells in macOS Catalina, you can change /var/select/sh to /bin/bash, /bin/dash, or /bin/zsh. If you change /var/select/sh to a shell other than bash, be aware that scripts that make use of bashisms may not work properly.

zsh can be made to emulate sh by executing the command zsh --emulate sh.

We can actually see that Apple has two options here. sh processes could be run by zsh in emulation mode or by dash.

dash is the Debian Almquist shell, which is a minimal implementation of the Posix sh standard. dash is a newcomer to macOS on Catalina. New arrivals on a release where Apple seems intent on cleaning out ballast (32-bit, Kernel Extensions, bash v3, Python, Perl, and Ruby) are immediately of interest.

When you check on Catalina Recovery system, you can see that /bin/dash has been added, but /bin/zsh is absent. (/bin/bash is still present on Catalina Recovery and the sh binary is also still bash.)

Putting all of this together, I would hazard that Apple is planning to use dash to take over from ‘bash as sh.’ This will also bring macOS in line with most other Unix and Linux distributions where dash commonly handles sh processes.

With the symbolic link mechanism described in the support article, Apple could switch sh to dash in an update while bash is still present. This could allow Developers or Mac Admins to switch their managed systems back to bash when they need to mitigate problems.

When Apple switches to dash, sh scripts that use bashisms will break.

You can start testing this in Catalina by changing the symbolic link at /var/select/sh:

% sudo ln -sf /bin/dash /var/select/sh

(Change /bin/dash to /bin/bash to revert to default.)

When I test the above command to get the current user with dash, it returns:

Syntax error: redirection unexpected

(You can also change the shebang of a script you want to test to #!/bin/dash then you do not have to change the system sh.)

Getting the Current User in sh, Future Proof (I hope)

The solution is easy enough. Replace the here string with a pipe. For sh scripts, change the syntax to:

loggedInUser=$( echo "show State:/Users/ConsoleUser" | scutil | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

This will make Posix sh, dash, and shellcheck happy and will still work in bash and zsh.

This syntax (echoing into a pipe) is considered inelegant and inefficient, because it requires an additional process (echo) and an additional pipe. Here docs and here strings were introduced exactly to avoid this. Obviously, if you prefer, you can continue to use the here string form of the command for bash and zsh.

Conclusion

There is no need panic and search and replace the python command in all your scripts immediately. I would recommend replacing this line when you update a script. Keep all of this in mind when working on your administration, installation and workflow scripts going forward.

You may also find scripts in third party documentation and installers that need updating, and you should let the author or vendor know.

  • /bin/bash (v3.2) is deprecated and has not received updates since 2014
  • /usr/bin/python (v2.7) is deprecated and will not receive new updates starting in Jan 2020
  • both will eventually be removed from macOS
  • we don’t know when exactly, could be years, could be months
  • even when you do not use Python for scripting, you may be using Python ‘one-liners’ in shell scripts
  • bash scripts should be moved to zsh (general use) or sh (general and installation scripts)
  • sh processes in macOS will likely be handled by dash in the future
  • use shellcheck to find and correct bashisms in sh scripts
  • shellcheck does not work for zsh scripts