MDOYVR 22 Talk: The Encyclopedia of macOS Automation

Last week I had the pleasure and honor of participating and presenting at MacDevOps YVR. The videos for the sessions are now appearing on YouTube.

There is a page for my talk “The Encyclopedia of macOS Automation,” in which I discuss the options for scripting and automation on macOS, with extra links and notes. You can go directly to the video here.

The talks this year were graphic recorded by the amazing Ashton Rodenhiser (website, twitter). The graphic at the top of this post was made by her while I was presenting.

As always, I had a lot of fun at this conference. Many thanks to the organizers and all the other speakers. Until next year!

Update Installomator: v9.2

We have updated Installomator. This brings Installomator to 465(!) applications! Many thanks to everyone who contributed.

Note: Both Google and Mozilla recommend using the pkg installers instead of the dmg downloads for managed deployments. So far, Installomator has provided labels for both. (googlechrome and googlechromepkgor firefox and firefoxpkg, respectively) Since there are problems with the dmg downloads, a future release of Installomator will disable the firefox and googlechrome dmg labels. You should switch to using the firefoxpkg or googlechromepkg labels instead.

  • bug and documentation fixes
  • 40 new, and 26 updated labels

You can find more details in the release notes.

Launching Scripts #4: AppleScript from Shell Script

In the last post, we discussed how to run shell commands and scripts from an Apple Script environment. In this post, we will look at how we can run AppleScript commands and scripts from the shell environment.

Open Scripting Architecture

The key to running AppleScript from the shell is the osascript command. OSA is short for ‘Open Scripting Architecture’ which is the framework that powers AppleScript. This framework allows AppleScript to have its native language, but also use JavaScript syntax.

The osascript command allows us to run AppleScript commands from Terminal and shell. The most common use is the user interaction commands from AppleScript, like display dialog:

osascript -e 'display dialog "Hello from shell"'

The -e option tells osascript that it will get one or more lines of statements as arguments. The following argument is AppleScript code. You can have multiple -e options which will work like multiple lines of a single AppleScript:

> osascript -e 'display dialog "Hello from shell"' -e 'button returned of result'
OK

osascript prints the value of the last command to stdout. In this case, it is the label of the button clicked in the dialog. (The ‘Cancel’ button actually causes the AppleScript to abort with an error, so no label will be returned for that.)

When you have multiple lines of script, using multiple -e statements will quickly become cumbersome and unreadable. It is easier to use a heredoc instead:

osascript <<EndOfScript
   display dialog "Hello from shell"
   return button returned of result
EndOfScript

This also avoids the problem of nested quotation marks and simplifies shell variable substitution.

Shell variables and osascript

There are a few ways to pass data into osascript from the shell.

Since the shell substitutes variables with their value before the command itself is actually executed, this works in a very straightforward manner:

computerName=$(scutil --get ComputerName)

newName=$(osascript -e "text returned of (display dialog \"Enter Computer Name\" default answer \"$computerName\")")

echo "New Name: $newName"

This works well, but because we want to use shell variable substitution for the $computerName, we have to use double quotes for the statement. That means we have to escape the internal AppleScript double quotes and everything starts to look really messy. Using a heredoc, cleans the syntax up:

computerName=$(scutil --get ComputerName)

newName=$(osascript <<EndOfScript
    display dialog "Enter Computer Name" default answer "$computerName"
    return text returned of result
EndOfScript
)

echo "New name: $newName"

I have a detailed post: Advanced Quoting in Shell Scripts.

Environment Variables

Generally, variable substitution works well, but there are some special characters where it might choke. A user can put double quotes in the computer name. In that case, the above code will choke on the substituted string, since AppleScript believes the double quotes in the name end the string.

If you have to expect to deal with text like this, you can pass data into osascript using environment variables, and using the AppleScript system attribute to retrieve it:

computerName=$(scutil --get ComputerName)

newName=$(COMPUTERNAME="$computerName" osascript <<EndOfScript
    set computerName to system attribute "COMPUTERNAME"
    display dialog "Enter Computer Name" default answer computerName
    return text returned of result
EndOfScript
)

echo "New name: $newName"

The shell syntax

VAR="value" command arg1 arg2...

sets the environment variable VAR for the process command and that command only. It is very useful.

Retrieving environment variables in AppleScript using system attribute is generally a good tool to know.

Interpret this!

osascript can also work as a shebang. That means you can write entire scripts in AppleScript and receive arguments from the shell. For example, this script prints the path to the front most Finder window:

#!/usr/bin/osascript

tell application "Finder"
    if (count of windows) is 0 then
        set dir to (desktop as alias)
    else
        set dir to ((target of Finder window 1) as alias)
    end if
    return POSIX path of dir
end tell

You can save this as a text file and set the executable bit. I usually use the .applescript extension.

> print_finder_path.applescript
/Users/armin/Documents

To access arguments passed into a script this way, you need to wrap the main code into a run handler:

#!/usr/bin/osascript

on run arguments
    if (count of arguments) is 0 then
        error 2
    end if
    return "Hello, " & (item 1 of arguments)
end

You can combine this into a longer script:

macOS Privacy and osascript

When you ran the above script, you may have gotten this dialog:

If you didn’t get this dialog, you must have gotten it at an earlier time and already approved the access.

AppleEvents between applications are controlled by the macOS Privacy architecture. Without this, any process could use AppleEvents to gather all kinds of data from any process. These dialogs are easy enough to deal with when running from Terminal. But if you put your AppleScript code (or shell scripts calling AppleScript) into other apps or solutions, it could get messy quite quickly.

Mac Admins generally want their automations to run without any user interactions. You can avoid these dialogs by creating PPPC (Privacy Preferences Policy Control) profiles that are distributed from an MDM server. In this case you have to pre-approve the application that launches the script, which can sometimes also be challenge. The other option is to find solutions that avoid sending AppleEvents altogether.

I have a longer post detailing this: Avoiding AppleScript Security and Privacy Requests

osascript and root

Management scripts often run as a privileged user or root. In this case, certain features of AppleScript may behave strangely, or not at all. I generally recommend to run osascript in the user context, as detailed in this post: Running a Command as another User

Conclusion

AppleScript’s bad reputation may be deserved, because its syntax is strange, and often very inconsistent. Nevertheless, it has features which are hard to match with other scripting languages. You can use the strategies from this and the previous posts to combine AppleScript with Shell Scripting and other languages to get the best of both worlds.

Launching Scripts #3: Shell scripts from AppleScript

In this series of posts, I am exploring the various different ways of launching scripts on macOS. In the first two posts, we explored what happens when you launch scripts from Terminal. We already explored some concepts such as the shell environment and how that affects scripts. In this post we are going to explore a different, but very common way to launch shell commands and scripts: AppleScript’s do shell script command.

do shell script

When AppleScript made the transition from Classic Mac OS 9 and earlier to Mac OS X, it gained one command that allowed AppleScripts to interact with Mac OS X’s Unix environment. The do shell script command executes a command or script in the shell and returns the output as a text to the AppleScript.

We have used this in an earlier post:

do shell script "echo $PATH"
    --> "/usr/bin:/bin:/usr/sbin:/sbin"

But you can use this to run any shell command:

do shell script "mdfind 'kMDItemCFBundleIdentifier == org.mozilla.firefox'"
    --> "/Applications/Firefox.app"

Note the use of single quotes inside the double quotes which delineate the AppleScript text. I have a post on the challenges of quoting in these mixed scripting environments.

You can assemble the command you pass into do shell script using AppleScript text operators:

set bundleID to "org.mozilla.firefox"
do shell script "mdfind 'kMDItemCFBundleIdentifier == " & bundleID & "'"
    --> "/Applications/Firefox.app"

Note that the PATH variable for AppleScripts that are run from Script Editor or as an AppleScript applet is different than the PATH in your interactive environment. Most notably, it does not include /usr/local/bin. When you want to use a command or script that is not stored in the four default directories, you will have to use the full path in the do shell script:

do shell script "/usr/local/bin/desktoppr"

(Desktoppr is a small tool I built to work with desktop pictures on macOS, you can get it here.)

When you are unsure what the full path to a command is, you can use the which command in Terminal:

> which desktoppr
/usr/local/bin/desktoppr

Errors

Keep in mind that which uses the same logic to lookup a command as the shell does when it looks up a command with no path. So, if you think you can trick AppleScript by using the which command to lookup a non-standard command, it will still fail:

do shell script "which desktoppr"
    --> error "The command exited with a non-zero status." number 1

When the command in do shell script returns a non-zero exit code, you will get an interactive dialog informing the user of the error. The AppleScript will not continue after the error. You can handle the error the same way you would handle any AppleScript error, with a try… on error block:

set filepath to "/unknown/file"

try
    do shell script "/usr/local/bin/desktoppr" & quoted form of filepath
on error
    display alert "Cannot set desktop picture to '" & filepath & "'"
end try

Files and Paths

AppleScript has its own methods of addressing files and folders. Actually, there are multiple ways, which is one of the confusing things about AppleScript. Neither of the native forms of addressing files and folder in AppleScript use the standard Unix notation with forward slashes separating folders. But there are built-in tools to convert from Unix notation to AppleScript and back.

Use the POSIX path attribute to get a Unix style file path from an AppleScript file or alias. Unix style paths used with commands need spaces and other special characters escaped. You can use the quoted form attribute to escape any AppleScript string. This is usually used directly with POSIX path:

set imagefile to choose file "Select a Desktop"
    --> alias "Macintosh HD:Library:Desktop Pictures:BoringBlueDesktop.png"
set imagepath to quoted form of POSIX path of imagefile
    --> '/Library/Desktop Pictures/BoringBlueDesktop.png'
do shell script "/usr/local/bin/desktoppr " & imagepath

You can convert a Unix style file path into an AppleScript file with the POSIX file type:

set bundleID to "org.mozilla.firefox"
set appPaths to do shell script "mdfind 'kMDItemCFBundleIdentifier == " & bundleID & "'"
    --> "/Applications/Firefox.app"

if (count of paragraphs of appPaths) = 0 then
    display alert "No app found"
else
    set appPath to first paragraph of appPaths
    -- convert the path to an AppleScript file
    set appFile to POSIX file appPath
        --> file "Macintosh HD:Applications:Firefox.app:"
    tell application "Finder" to reveal appFile
end if

Shell Scripts in AppleScript Bundles

Sometimes you are writing an AppleScript and want to use a script for some functionality which is difficult to achieve in AppleScript. When you have an AppleScript file and a shell script file that work together this way, you want to store them together and also have an easy way for one to get the location of the other.

For this, AppleScript provides the notion of script bundles and AppleScript applets (which are also bundles). A script bundle is not a flat file, but a folder which contains the script itself and other resources, such as script libraries, or shell scripts. The script can easily locate items in its bundle and call them.

For example, we want a script that needs to unzip a file. We can use the unzip command line tool to do that, but in my experience it is better to use ditto. The ditto expansion seems to be closer to how the expansion from Archive Utility works and do better with extended attributes and resource forks and other macOS specific things.

This is the shell script for our example:

#!/bin/sh

# target dir for expansion
targetdir="/Users/Shared/Script Bundle Demo"

# sanity checks for argument 1/filepath
if [ -z "$1" ]; then
    exit 2
fi

filepath=$1

# is it a file?
if [ ! -f "$filepath" ]; then
    exit 3
fi

# note: ditto seems to work better than unzip
ditto -x -k "$filepath" "$targetdir"

This is simple enough that you could just do it in a one-line do shell script, but I want to keep it simple. You could extend this shell script to use different tools to expand different types of archives, such as xar, tar, aa etc. If you want a more complex script, feel free to build one!

Now we can build the AppleScript/shell script combination. Open Script Editor, create a new script and save it right away. In the save dialog, change the ‘File Format’ to ‘Script bundle’ before saving.

After you have saved the Script as a bundle, you can see the Bundle Info in a pane on the right side of the script window. If you don’t see this pane, choose ‘Show Bundle Contents’ from the ‘View’ menu, or click the right most icon in the tool bar.

In this pane, you can set the name, identifier, version and some other data for the script bundle. You can also see a list of the ‘Resources’ which shows the contents of the Contents/Resources folder in the script’s bundle. When you find the script you save, you will see it has a scptd file extension and when you open the context menu on it in Finder, you can choose ‘Show Package Contents’ and dig into the bundle contents.

Note: AppleScript applications (or applets) work the same way. Their .app bundles have a few more sub folders, but the Resources work the same way. The difference is that AppleScript applets work on double-click, drag’n drop, and some other events that we will get to in later posts. Script bundles have to run from Script Editor.

Save the shell script from above into the script bundle’s Resources sub-directory with the name unarchive.sh. You should see it appear in the ‘Resources’ list in the script window.

This way, the AppleScript bundle can contain all the resources it might need, including shell (or other) scripts.

Now we still need to find a way to access the Resources from the script. To run our shell script, add the following code to the AppleScript in Script Editor:

-- Script Bundle Demo
set theArchive to choose file "Select a zip archive:" of type {"zip"}
set archivePath to quoted form of POSIX path of theArchive

-- assemble command
set scriptPath to quoted form of POSIX path of my (path to resource "unarchive.sh")
set commandString to scriptPath & space & archivePath

-- for debugging 
log (commandString)

do shell script commandString

First we prompt the user to choose a file with a zip extension, and the we convert the result into a quoted Unix path.

Then, we use the path to resource "unarchive.sh" to get the path to our shell script in the Resources folder in the bundle. Then we get the quoted Unix notation, and assemble the command for the do shell script. The log command there will print the commandString to the output in the script window and is useful for debugging. Then we run the command with do shell script.

Environment for do shell script

Our example script expands the archive into a subfolder of /Users/Shared. If you wanted to use a different location, you could use a second argument in the script.

There is a different way of passing data into scripts and that is environment variables.

First of all it is important to note that the shell environment for commands and scripts run with the do shell script command from an AppleScript in Script Editor or an AppleScript application is very different from the shell environment in an interactive shell in the Terminal. We have already seen that the PATH environment variable has a different value, which influences the command lookup behavior.

You can check the environment variable by running the env command. This will list all environment variables. (To be nitpicky, there is more to a shell environment than just the env variables, there are also shell options, but those will be different for each shell, sh, bash or zsh, anyway.)

do shell script "env"
    --> "SHELL=/bin/zsh
TMPDIR=/var/folders/2n/q0rgfx315273pb4ycsystwg80000gn/T/
USER=armin
COMMAND_MODE=unix2003
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/usr/bin:/bin:/usr/sbin:/sbin
__CFBundleIdentifier=com.apple.ScriptEditor2
PWD=/
XPC_FLAGS=0x0
SHLVL=1
HOME=/Users/armin
LOGNAME=armin
_=/usr/bin/env"

Interestingly, we have USER and HOME to use in this environment.

We can also add environment variables to a do shell script command:

do shell script "TARGET_DIR='/Users/Shared/Script Bundle Demo' " & scriptPath & space & filePath

You can use this to set the value of the TARGET_DIR env variable for the next command, which is our script in the script bundle.

Administrator Privileges

No matter which way you use do shell script, it has one big benefit. You can prompt the user to get the command or script to run with administrative privileges:

do shell script "systemsetup -getRemoteLogin" with administrator privileges

This will prompt for the user name and password of an administrator user. This can allow to build some simple workflows that require user interaction and administrative privileges really easily.

Conclusion

Combining Script Bundles and AppleScript Applications with shell scripts can create a powerful combination. You can use the “best of both worlds” with some of AppleScript’s user interaction commands and the shell’s strength in file manipulation and similar workflows. You can also sign AppleScript applications with a valid Apple Developer ID and pre-approve them for privacy exemptions with a PPPC profile.

If this explanation is not detailed enough for you, there is an amazing Tech Note in Apple’s Documentation Archive.

This post covered launching shell scripts from AppleScript. In the next post we will launch AppleScript code from shell scripts.

Launching Scripts #2: Launching Scripts from Finder

In this series of posts, I will explore the many ways that you can launch a script on macOS. In the previous, inaugural post, I described what happens when you launch a script from an interactive terminal shell.

There are several virtual terminal applications available for macOS. iTerm is very popular. Some text editors like Visual Studio Code and Nova, have terminals built-in. Since the actual launching of an executable is done by the shell running inside the virtual terminal, the launch process remains the same.

That said, Terminal app has a useful trick up its sleeves.

command file extension

When you change the file extension of a script to .command, double-clicking the file will open it in a new Terminal window and run it there. Any input or output the script requires will happen in that Terminal window. When the script exits, the shell session in the Terminal window will exit.

Let’s take this simple script:

#!/bin/sh
echo "Enter your name: "
read -r username
echo "Hello, $username"

When you put this this in a .command file and double-click it, you get a new Terminal window with:

/Users/armin/Desktop/hello_name.command ; exit;                                 
~ % /Users/armin/Desktop/hello_name.command ; exit;
Enter your name: 

You can see that Terminal opens a new window with a new, default shell and all your configurations, then launches the script right away. The script prints its output and then waits for the user input (the read command). When you enter the name at the prompt, the script continues.

Armin
Hello, Armin

Saving session...completed.

[Process completed]

When the script ends, the shell in the Terminal window exits, as well. No more interactive prompt will be shown.

This script expects user input in the Terminal and then presents output to stdout in the same window. While you could re-write a script to use AppleScript’s display dialog to handle both in the input and the output, it would make the script significantly more complex.

Instead, you can change the file extension to .command and then a double-click will create a new Terminal window where the user interaction (input and output) takes place. For the right kind of user and workflow, this can be a sufficient solution with practically no overhead.

You can also remove the file extension completely. The behavior when you double-click such a file in Finder will be the same. Extension-less executables also get a different the icon. Either way, you need to have the executable bit set for the script.

Note: iTerm can also open .command files, but I have had some trouble with user interaction in these cases. Since I usually don’t use iTerm, maybe I have something setup wrong?

Quarantine and Gatekeeper

A Terminal window is not a user interface that many users will appreciate, but this allows you share scripts with other users in a form they understand. “Double-click this to run” is something that fits with most users’ idea of how macOS works.

Before you start creating dozens of .command scripts and share them, there is a major tripwire that macOS security has set up.

When you share an executable file through a website, email or a chat message, macOS will attach a quarantine flag. With applications, this flag triggers a GateKeeper scan before the app is launched and it will show the standard dialog, even when the app is signed and notarized and a much more “scary” warning when it is not.

When you launch a command or script from Terminal, either directly or indirectly with a double-click, and it still has the quarantine flag set, it will not launch. You will not get one of the standard Gatekeeper dialogs. Just an opaque operation not permitted error in the Terminal output.

You can check if a file has the quarantine flag set with the xattr command:

> xattr hello_name.command
com.apple.TextEncoding
com.apple.lastuseddate#PS
com.apple.quarantine 

Your list list of extended attributes may be different. The quarantine flag has the label com.apple.quarantine.

The xattr command also can remove the quarantine flag:

> xattr -d com.apple.quarantine hello_name.command

Apple seems to assume that when you are using Terminal, you know what you are doing. That means you can bypass most of the mechanisms that attach a quarantine flag with command line tools. When you download something with curl it won’t get quarantined. You can install an unsigned, unnotarized pkg installer using the installer command. Because this is possible, it doesn’t mean it is always wise. Piping a curl command directly into sh or bash or any interpreter is still poor security.

Most of the time though, a script file shared to another Mac or another user will almost certainly get the quarantine flag. Users who are comfortable with using Terminal should be able to use the xattr command to disable this protection, but this is not something for ‘normal’ users. So quarantine, makes the use of the command file extensions far less effective than it could be. This is probably intentional, since executables that are opened by double-click can be an easy way to sneak malware and other unwanted software onto a system, leveraging a user’s ignorance of what is happening.

This is generally true when you move scripts and executable files between macOS systems. I have also seen the quarantine flag getting set when you store a script in a cloud sync service (especially iCloud) or when you edit an executable with a sandboxed application.

One way around the quarantine, would be to distribute and properly install the scripts with an installer pkg. Then you can properly sign and notarize the installer pkg and the scripts will not be quarantined, as they come from a trusted and verified source. This may be a good solution for some workflows, but generally feels a bit “over-designed.”

Output flashes by so quickly

There is a setting in Terminal’s preferences which determines what happens with windows when the shell exits. You can find it under the ‘Shell’ tab in the ‘Profiles’ area. This setting can be different for each profile. Under ‘When the shell exits’ there is a popup menu with the options ‘Close the window,’ ‘Close if the shell exited cleanly,’ and ‘Don’t close the window.’ The last ‘Don’t close’ is the default.

When you have this option set to ‘Close the window,’ the new Terminal window from a command file might only be active and visible for a short time. This may or may not be a good thing, depending on what you want.

Conclusion

Script files with the command file extension can be a simple, straightforward way to make scripts easily ‘launchable’ from Finder. You can also put them in the Dock or in the Login Items. The user experience is, well, a terminal, so not terribly nice, but it can be useful, and does not require any modification of the script.

When you share executable scripts, whether they have the command file extension or not, Gatekeeper quarantine on macOS can prevent the script from running. You should get familiar with the quarantine flag and the xattr command to manipulate it.

In the next post, we launch shell scripts from AppleScripts.]

On env Shebangs

There was a comment to my previous post about using the /usr/bin/env shebang to gain system portability.

Regarding shebang, a more system independent way would be to use e.g. ‘#!/usr/bin/env bash’. With this the script would also be usable on FreeBSD, where bash would be installed from FreeBSD Ports and then be available as ‘/usr/local/bin/bash’ or some Linux systems (e.g. Debian) where it is ‘/usr/bin/bash’. Unfortunately there are some other unixode systems around, where ‘env’ is not in ‘/usr/bin/’ and so the shebang needs to be adjusted.

(I replied to the comment there, but then realized this deserves its own post.)

The /usr/bin/env shebang provides a means for system portability. It has many valid use cases. However, you don’t just magically gain portability by switching to an env shebang. There are many trade-offs to consider.

The note on how the env binary may not be in /usr/bin on all platforms, hints at some of these trade-offs, but there are more.

The trade-offs are a loss of predictability and reliability, or functionality, or increased maintenance and management.

Let me elaborate…

How the /usr/bin/env shebang works

When used in the shebang, the /usr/bin/env binary will use the current environment’s PATH to lookup the interpreter binary for the script, in the same way the shell looks up commands.

As an example, let us imagine you have installed bash v5 on your Mac. Either manually or using brew or some other package management system.

This will (usually) put the bash v5 binary at /usr/local/bin/bash. The /usr/local/bin directory is a common choice for custom command line tools, because it is part of the default PATH for interactive shells on macOS and not protected by SIP/SSV. The default PATH on macOS for interactive shells is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Some installations will put the binary in a different location in the file system. Then you you have to pre-pend the directory containing the binary to your PATH variable in your shell configuration. The order of the directories in the PATH is important, because the shell and env will stop the search when they find the first match. If /usr/local/bin came after /bin in the PATH the new binary would not be ‘seen’ since the pre-installed, old /bin/bash binary is found first.

Some installations solve this by placing a symbolic link to the binary in /usr/local/bin.

When you run a script from the interactive shell with a shebang of #!/usr/bin/env bash, then env would find the bash v5 binary first in /usr/local/bin, so your script is interpreted with bash v5. This is probably what you were hoping for, when you installed bash v5 in the first place.

When you run the same script on a different Mac (same macOS version, but it doesn’t have bash v5 installed) env will pick up /bin/bash. Your script will work even though that other Mac doesn’t have /usr/local/bin/bash, so you gained portability.

However, /bin/bash is bash v3.2, so your script may behave differently. If the script uses bash v5 features that are not available in the 15-year-old bash v3.2, it will generate errors. Since you actively chose to install bash v5 on the first Mac, it is likely you needed some of these bash v5 features, so it is likely your script will fail on other Macs, which don’t have bash v5 installed.

You lost either predictability and reliability (which version and features are available? Does my script run successfully?), or you lose functionality (the features added to bash v5 since v3.2). To retain reliability, you can restrict the script to features that work in both bash versions. But then using the env shebang gives you no advantage, and you might as well use /bin/bash as the shebang.

Some solutions

One alternative is to use a /usr/local/bin/bash shebang for scripts which use bash v5 functionality and continue to use /bin/bash for scripts that need to run across multiple Macs, where you pay attention to using only features available in bash v3.2. You gain predictability and reliability, but your bash v5 scripts aren’t portable to other Macs. They may even fail on other Macs with bash v5 installed, if the bash v5 binary is installed in a different location.

When you use /usr/bin/env bash for a bash v5 script, it will run fine on all Macs which have bash v5 installed and the PATH configured properly to find it. (Configuring and maintaining the PATH does not happen on its own.) But the script will still fail on Macs without any bash v5. You can (and probably should) add a version check to the script, but now you are increasing code maintenance.

When you are managing a fleet of Macs, you also have the option (or in this case, I would say, the duty) to install bash v5 in a consistent location and version across all Macs in your fleet and pre-configure the proper PATH in the contexts the script will run in. Then you get predictability and functionality, but it requires extra effort in deployment and maintenance.

This requires a decently experienced MacAdmin and the proper tooling, neither of which comes for free.

Note: There are great open source solutions for macOS in this area, but I consider them ‘free, as in puppy,’ so they come with higher skill requirements and/or maintenance effort for the admin. And this isn’t supposed to imply that all commercial solutions are ‘easy to use,’ either. It’s trade-offs all the way down.

Context changes the PATH

Notice that so far I kept talking about “the default PATH for the interactive shell.”

The PATH variable may be different depending on the context, even on the same Mac with the same user. For example, when you run your script with the AppleScript do shell script command, the PATH in that context is not the same as the PATH in your interactive shell. It will be:

/usr/bin:/bin:/usr/sbin:/sbin

You can verify this by opening Script Editor and running the do shell script "echo $PATH". Other context, like scripts in installation packages will see other PATH values.

Most importantly, the PATH in these other contexts, does not contain /usr/local/bin, or any other addition you made to your PATH in the shell configuration files. An /usr/bin/env shebang will not ‘see’ a bash 5 binary you installed on the system. The same script with the same user on the same computer, will behave differently when run in a different context.

These different PATH values are an intentional choice. In these contexts, especially installation package scripts, reliability and predictability are extremely important. You do not want user and third-party installed custom binaries to interfere with the command lookup.

Sidenote on Python

With python and python3 and other run time interpreters, it gets even more difficult. There may multiple different versions installed and the behavior and functionality between versions varies more. My Mac currently has four different Python 3 binaries, each with a different version, and I not even remotely a full-time Python developer. When you call python3 on a non-developer Mac it will trigger the ‘You have to install Developer Command Line Tools’ dialog when Xcode is not installed. (Developers seem to have a hard time considering that there are Macs without Xcode installed.)

With the demise of python 2 in macOS 12.3, some developers reacted by changing the shebang in their python scripts from /usr/bin/python to /usr/bin/env python which solves nothing, when the binary goes away without replacement. Some switched to /usr/bin/env python3 which can makes things worse, by triggering the Developer Tools installation or picking a random python3 binary of the wrong version.

The only reliable solution for the mess that is python is to deploy a consistent version of Python 3 in a consistent location. You can do this by either bundling the python framework and module your tool needs together with the tool, or by deploying and maintaining the Python frameworks and modules with a management system.

MacAdmin perspective

As a MacAdmin, my scripts don’t need to be portable to systems other than macOS. They usually use tools and access files and folders that only exist on macOS. Predictability and reliability, however, are paramount. Configuration and installation scripts need to run reliably on thousands of Macs across multiple versions of macOS (even future versions) regardless of what else is installed.

As MacAdmins, we also (should) have the tools and experience to deploy and maintain binaries in predictable locations. But then, like everyone, we only have limited time, and often need to prioritize business critical software over our own tooling. So, the pre-installed interpreter binaries have little ‘friction’ to use, even if they may have a reduced functionality when compared to the latest version available elsewhere.

This is the reason bash v3.2 is still present on macOS 12.3 and it will never be easy when Apple ultimately decides to remove it. So many tools and scripts rely on /bin/bash.

(I don’t expect the removal to be any time soon, but there is a limit to how long Apple will or can keep this interpreter from 2007 on the system. We got the first warning when Apple switched the default interactive shell to zsh in Catalina. There will be more warnings …I hope. With the removal of the Python 2 binary we saw that Apple can move quickly when they feel the need. They did not even wait for a major macOS release.)

In this context, there is no gain in using /usr/bin/env. The trade-offs favor the absolute shebang very strongly.

Cross-platform portability

After this rant, you may think that I recommend against using /usr/bin/env shebangs always. But there are very good use cases. Using /usr/bin/env shebangs is the solution for workflows where cross-platform portability is required.

When your scripts need to run across multiple platforms, installing the binaries in the same location in the file system may not be possible, or require an unreasonable effort. For example, the /bin and /usr/bin are protected by SIP and the Sealed System Volume on macOS, so you cannot add your tooling there without significantly impacting the integrity and security of the entire system.

In these cases, using a /usr/bin/env shebang provides the required flexibility, so your scripts can work across platforms. But the portability does not come magically from just switching the shebang.

The target platforms need the binary to be installed and the versions should match. The installation location of the binary has to be present in the PATH in the context the script runs in. To get reliable behavior, the systems you are porting between need to be well managed, with a predictable setup and configuration of the interpreter binary and environment.

When your scripts work ‘effortlessly’ across systems with the env shebang, it is thanks to the work of the platform developers and your sysadmins/devops team for creating and maintaining this consistency. Even if you are the sole developer and devops admin, maintaining all the systems, you have to put in this work. Also the platform developers put in a lot of effort to achieve much of this consistency out of the box. As the commenter noted, some platforms don’t even agree where the env binary should be.

You gain portability at the price of increased maintenance.

Trade-offs all the way down

Alternatively, you can keep the scripts simple – restricted to a subset of features common across platforms and versions – so that the differences have no impact. Then you trade for reliability and portability at the price of functionality.

This is often the trade-off with Python (restrict the use of python features to those common among versions) and one of the reasons Python 2 was kept around for so long on macOS.

Using POSIX sh, instead of bash or zsh, is another option to gain portability, but that has its own trade-offs. Most of these trade-offs will be in functionality, but also consider not all sh emulations are equal, and supporting multiple different emulators or the real common subset of functionality, requires extra effort.

Conclusion

Shebangs with absolute paths have their strengths and weaknesses, as do shebangs with /usr/bin/env. Each has their use case and you have to understand the trade-offs when using either. Neither env shebangs nor absolute path shebangs are generally ‘better.’ Either one may get you in trouble when used in the wrong context.

When someone says ‘you should be using,’ or ‘a better way would be,’ you always need to consider their context, use case, and which trade-offs they are accepting. You need to understand the tools to use them efficiently.

Use shellcheck with BBEdit

When I teach or mentor shell scripting on macOS, I always have two recommendations: use BBEdit as your text editor and use shellcheck to verify sh and bash scripts.

I only discovered recently, that the two combine wonderfully!

It used to be that installing the shellcheck command line tool on macOS was a complicated process. Thankfully, the GitHub project now provides a pre-compiled binary for macOS, so you don’t have to mess with all of that anymore. The project labels the macOS version as ‘darwin.’ The binary is x86_64 (Intel) only and requires Rosetta on Apple silicon, but this doesn’t reduce its usefulness.

Installing shellcheck

The easiest way to install the command is to download the archive from the repo, expand the archive, and copy the shellcheck binary to /usr/local/bin/. You can follow the instructions from this earlier post to do it in Terminal or do it with your browser and in the Finder. When you download and unarchive manually, you will have to remove the quarantine flag from the shellcheck command, before you can use it.

> xattr -d com.apple.quarantine shellcheck-latest/shellcheck

I also created shellcheck recipes for AutoPkg, that automate creating a package installer for mass deployment.

Using shellcheck

Once you have the shellcheck command in place, you can use it from the command line:

> shellcheck my_great_script.sh

Sadly, shellcheck will only work with sh and bash scripts, not with zsh scripts. On the other hand, many of the common mistakes that shellcheck catches, (mostly quoting variables) are not mistakes in zsh. Still, I miss it a lot when working on large zsh scripts.

shellcheck with BBEdit

Once you have the shellcheck command installed, you can also invoke from within BBEdit: When you have a script open in BBEdit, verify that the script is recognized as a ‘Unix shell script.’ Then you can select ‘Check Syntax…’ from the ‘#!’ menu (keyboard shortcut ⌘-K). This will open a second window with all the issues shellcheck has found.

This feature was added in BBEdit 13.1, but it took me quite a while to discover it. Now I find it indispensable.

Enjoy!

Update: Installomator 9.1

We have updated Installomator. This brings Installomator to 407(!) applications! Many thanks to everyone who contributed.

Note: Both Google and Mozilla recommend using the pkg installers instead of the dmg downloads for managed deployments. So far, Installomator has provided labels for both. (googlechrome and googlechromepkg or firefox and firefoxpkg, respectively) Since there are problems with the dmg downloads, a future release of Installomator will disable the firefox and googlechrome dmg labels. You should switch to using the respective pkg labels instead.

  • added option for Microsoft Endpoint Manager (Intune) to LOGO
  • minor fixes
  • the googlechrome label now always downloads the universal version
  • 16 new labels
  • 6 updated labels

Full release notes in the repo.

Update: Installomator 9.0.1

We found a bug had snuck in to Installomator 9.0 which broke applications that download as pkgs wrapped in dmgs, so we have a bug fix update. While we were at it, there were a few other minor changes as well:

  • improved logging levels throughout the script
  • fixed a bug for pkgindmg style labels
  • changed the criteria used to locate an app in the case the it cannot be found in the default locations, this should help with some apps with similar name (Virtual Box and Box Drive)
  • new label: WhiteBox Packages (packages)
  • modified label: loom (added Apple silicon download)

You can get more details and download the pkg installer from the Installomator repo’s release page.

The unexpected return of JavaScript for Automation

Monterey has deprecated the pre-installed python on macOS. To be precise, built-in python has been deprecated since macOS Catalina, but Monterey will now throw up dialogs warning the user that an app or process using built-in python needs to be updated.

I and others have written about this before:

So far, I have recommended to build native Swift command line tools to replace python calls. However, from discussions in MacAdmins Slack, a new option has emerged. Most of the credit for popularizing and explaining this goes to @Pico (@RandomApps on Twitter) in the #bash and #scripting channels.

(Re-)Introducing JavaScript for Automation

AppleScript has been part of macOS since System 7.1. In the late nineties, there was concern that it wouldn’t make the transition to Mac OS X, but AppleScript made the jump and has happily co-existed with the Terminal and shell scripting as an automation tool on macOS. AppleScript has a very distinct set of strengths (interapplication communication) and weaknesses (awkward syntax and inconsitent application functionality and dictionaries) but it has been serving its purpose well for many users.

With Mac OS X 10.4 Tiger, Apple introduced Automator, which provided a neat UI to put together workflows. Much of Automator was based on AppleScript and users expected a more and improved AppleScript support because of that going forward. Instead, we saw AppleScript’s support from Apple and third parties slowly wane over the years.

AppleScript is stil very much present and functional in recent versions of macOS. It just seems like it hasn’t gotten much love over the last decade or so. Now that Shortcuts has made the jump from iOS, there may be hope for another revival?

The last major changes to AppleScript came with Mavericks and Yosemite. Mavericks (10.9) included a JavaScript syntax for the Open Scripting Architecture (OSA), which is the underlying framework for all AppleScript functionality. Apple called this “JavaScript for Automation.” Because this is a mouthful, it often abbreviated as JXA.

The JavaScript syntax and structure is more like a “real” programming language, than the “english language like” AppleScript. Once again this raised hopes that this could attract more scripters to AppleScript and thus encourage Apple and third party developers to support more AppleScript. But unfortunately, this positive re-inforcement did not take off.

Then Yosemite (10.10) made the AppleScript-Objective-C bridge available everywhere in AppleScript. Previously, the Objective-C bridge was only available when you built AppleScript GUI applications using AppleScript Studio in Xcode. The Objective-C bridge allows scripters to access most of the functionality of the system frameworks using AppleScript or JXA.

The coincidence of these two new features might be the reason that the ObjC bridge works much better using JXA than it does with the native AppleScript syntax.

JXA and Python

What does JXA and the AppleScriptObjC bridge have to do with the Python deprecation in modern macOS?

One reason python became so popular with MacAdmins, was that the pre-installed python on Mac OS X, also came with PyObjC, the Objective-C bridge for python. This allowed python to build applications with a native Cocoa UI, such as AutoDMG and Munki’s Managed Software Center. It also allowed for short python scripts or even one-liners to access system functionality that was otherwise unavailable to shell scripts.

For example, to determine if a preference setting in macOS is enforced with a configuration profile, you can use CFPreferences or NSUserDefaults.

Objective-C/C:

BOOL isManaged =CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

Swift:

let isManaged = CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

The Objective-C bridge allows to use this call from python, as well:

from Foundation import CFPreferencesAppValueIsForced
isManaged=CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

With JXA and the AppleScriptObjC bridge, this will look like this:

ObjC.import('Foundation');
$.CFPreferencesAppValueIsForced(ObjC.wrap('idleTime'), ObjC.wrap('com.apple.screensaver'))

Now, this looks really simple, but working with any Objective-C bridge is always fraught with strange behaviors, inconsistencies and errors and the JXA ObjC implementation is no different.

For example, I wanted to change the code above to return the value of the setting instead of whether it is managed. The CFPreferences function for that is called CFPreferencesCopyAppValue and it works fine in Swift and Python, but using JXA it only ever returned [object Ref]. The easiest solution was to switch from the CFPreferences functions to using the NSUserDefaults object:

ObjC.import('Foundation');
ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('$1').objectForKey('$2'))

(Once again many thanks to @Pico on the MacAdmins Slack for helping me and everyone else with this and also pointing out, that there is a different, somewhat complicated, solution to the object Ref problem. I will keep that one bookmarked for situations where there is no alternative Cocoa object.)

We used this to remove the python dependency from Mischa van der Bent’s CIS-Scripts.

JXA in shell scripts

To call JXA from a shell script, you use the same osascript command as for normal AppleScript, but add the -l option option to switch the language to JavaScript:

osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation');
    ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('idleTime').objectForKey('com.apple.screensaver'))
EndOfScript

For convenience, you can wrap calls like this in a shell function:

function getPrefValue() { # $1: domain, $2: key
      osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation');
    ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('$1').objectForKey('$2'))
EndOfScript
}

function getPrefIsManaged() { # $1: domain, $2: key
     osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation')
     $.CFPreferencesAppValueIsForced(ObjC.wrap('$1'), ObjC.wrap('$2'))
EndOfScript
}

echo $(getPrefValue "com.apple.screensaver" "idleTime")
# -> actual value
echo $(getPrefIsManaged "com.apple.screensaver" "idleTime")
# -> true/false

Note that the $ character does a lot of work here. It does the shell variable substitution for the function arguments in the case of $1 and $2. These are substituted before the here doc is piped into the osascript command. The $. at the beginning of the command is a shortcut where $ stands in for the current application and serves as a root for all ObjC objects.

There is also a $(…) function in JXA which is short for ObjC.unwrap(…) but I would recommend against using that in combination with shell scripts as shell’s command substitution has the same syntax and would happen before the JavaScript is piped into osascript.

There is a GitHub wiki with more detailed documentation on using JXA, and the JXA Objective-C bridge in particular.

JXA for management tasks

I’ll be honest here and admit that working with JXA seems strange, inconsistent, and — in weird way — like a step backwards. Putting together a Command Line Tool written in Swift feels like a much more solid (for lack of a better word) way of solving a problem.

However, the Swift binary command line tool has one huge downside: you have to install the binary on the client before you can use it in scripts and your management system. Now, as MacAdmins, we usually have all the tools and workflows available to install and manage software on the client. That’s what we do.

On the other hand, I have encountered three situations (set default browser, get free disk space, determine if a preference is managed) where I needed to replace some python code in the last few months and I would have no trouble finding a few more if I thought about it. Building, maintaining, and deploying a Swift CLI tool for each of these small tasks would add up to a lot of extra effort, both for me as the developer and any MacAdmin who wants to use the tools.

Alternatively, you can deploy and use a Python 3 runtime with PyObjC, like the MacAdmins Python and continue to use python scripts. That is a valid solution, especially when you use other tools built in python, like Outset or docklib. But it still adds a dependency that you have to install and maintain.

In addition to being extra work, it adds some burden to sharing your solutions with other MacAdmins. You can’t just simply say “here’s a script I use,” but you have to add “it depends on this runtime or tool, which you also have to install.

Dependencies add friction.

This is where JXA has an advantage. Since AppleScript and its Objective-C bridge are present on every Mac (and have been since 2014 when 10.10 was released) there is no extra tool to install and manage. You can “just share” scripts you build this way, and they will work on any Mac.

For example, I recently built a Swift command line tool to determine the free disk space. You can download the pkg, upload it to your management system, deploy it on your clients and then use a script or extension attribute or fact or something like to report this value to your management system. Since there is a possibility that the command line tool is not yet installed when the script runs, you need to add some code to check for that. All-in-all, nothing here is terribly difficult or even a lot of work, but it adds up.

Instead you can use this script (sample code for a Jamf extension attribute):

#!/bin/sh

freespace=$(/usr/bin/osascript -l JavaScript << EndOfScript
    ObjC.import('Foundation')
    var freeSpaceBytesRef=Ref()
    $.NSURL.fileURLWithPath('/').getResourceValueForKeyError(freeSpaceBytesRef, 'NSURLVolumeAvailableCapacityForImportantUsageKey', null)
    ObjC.unwrap(freeSpaceBytesRef[0])
EndOfScript
)

echo "<result>${freespace}</result>"

Just take this and copy/paste it in the field for a Jamf Extension Attribute script and you will get the same same free disk space value as the Finder does. If you are running a different management solution, it shouldn’t be too difficult to adapt this script to work there.

The Swift tool is nice. Once it is deployed, there are some use cases where it could be useful to have a CLI tool available. But most of the time, the JXA code snippet will “do the job” with much less effort.

Note on Swift scripts

Some people will interject with “but you can write scripts with a swift shebang!” And they are correct. However, scripts with a swift shebang will not run on any Mac. They will only run with Xcode, or at least the Developer Command Line Tools, installed. And yes, I understand this is hard for developers to wrap their brains around, but most people don’t have or need Xcode installed.

When neither of these are installed yet, and your management system attempts to run a script with a swift shebang, it will prompt the user to install the Developer command line tools. This is obviously not a good user experience for a managed deployment.

As dependencies go, Xcode is a fairly gigantic installation. The Developer Command Line Tools much less so, but we are back in the realm of “install and manage a dependency.”

Parsing JSON

Another area where JXA is (not surprisingly) extremely useful is JSON parsing. There are no built-in tools in macOS for this so MacAdmins either have to install jq or scout or fall back to parsing the text with sed or awk. Since JSON is native JavaScript, JXA “just works” with it.

For example the new networkQuality command line tool in Monterey has a -c option which returns JSON data instead of printing a table to the screen. In a shell script, we can capture the JSON in a variable and substitute it into a JXA script:

#!/bin/sh

json=$(networkQuality -c)

osascript -l JavaScript << EndOfScript
    var result=$json
    console.log("Download:  " + result.dl_throughput)
    console.log("Upload:    " + result.ul_throughput)
EndOfScript

Update: (2021-11-24) Paul Galow points out that this syntax might allow someone to inject code into my JavaScript. This would be especially problematic with MacAdmin scripts as those often run with root privileges. The way to avoid this injection is too parse the JSON data with JSON.parse :

#!/bin/sh 

json=$(networkQuality -c) 

osascript -l JavaScript << EndOfScript     
  var result=JSON.parse(\`$json\`)     
  console.log("Download:  " + result.dl_throughput)     
  console.log("Upload:    " + result.ul_throughput) 
EndOfScript

(I am leaving the original code up there for comparison.)

Conclusion

After being overlooked for years, JXA now became noticeable again as a useful tool to replace python in MacAdmin scripts, without adding new dependencies. The syntax and implementation is inconsistent, buggy, and frustrating, but the same can be said about the PyObjC bridge, we are just used it. The community knowledge around the PyObjC bridge and solutions goes deeper.

However, as flawed as it is, JXA can be a simple replacement for the classic python “one-liners” to get data out of a macOS system framework. Other interesting use cases are being discovered, such as JSON parsing. As such, JavaScript for Automation or JXA should be part of a MacAdmins tool chest.