On env Shebangs

There was a comment to my previous post about using the /usr/bin/env shebang to gain system portability.

Regarding shebang, a more system independent way would be to use e.g. ‘#!/usr/bin/env bash’. With this the script would also be usable on FreeBSD, where bash would be installed from FreeBSD Ports and then be available as ‘/usr/local/bin/bash’ or some Linux systems (e.g. Debian) where it is ‘/usr/bin/bash’. Unfortunately there are some other unixode systems around, where ‘env’ is not in ‘/usr/bin/’ and so the shebang needs to be adjusted.

(I replied to the comment there, but then realized this deserves its own post.)

The /usr/bin/env shebang provides a means for system portability. It has many valid use cases. However, you don’t just magically gain portability by switching to an env shebang. There are many trade-offs to consider.

The note on how the env binary may not be in /usr/bin on all platforms, hints at some of these trade-offs, but there are more.

The trade-offs are a loss of predictability and reliability, or functionality, or increased maintenance and management.

Let me elaborate…

How the /usr/bin/env shebang works

When used in the shebang, the /usr/bin/env binary will use the current environment’s PATH to lookup the interpreter binary for the script, in the same way the shell looks up commands.

As an example, let us imagine you have installed bash v5 on your Mac. Either manually or using brew or some other package management system.

This will (usually) put the bash v5 binary at /usr/local/bin/bash. The /usr/local/bin directory is a common choice for custom command line tools, because it is part of the default PATH for interactive shells on macOS and not protected by SIP/SSV. The default PATH on macOS for interactive shells is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Some installations will put the binary in a different location in the file system. Then you you have to pre-pend the directory containing the binary to your PATH variable in your shell configuration. The order of the directories in the PATH is important, because the shell and env will stop the search when they find the first match. If /usr/local/bin came after /bin in the PATH the new binary would not be ‘seen’ since the pre-installed, old /bin/bash binary is found first.

Some installations solve this by placing a symbolic link to the binary in /usr/local/bin.

When you run a script from the interactive shell with a shebang of #!/usr/bin/env bash, then env would find the bash v5 binary first in /usr/local/bin, so your script is interpreted with bash v5. This is probably what you were hoping for, when you installed bash v5 in the first place.

When you run the same script on a different Mac (same macOS version, but it doesn’t have bash v5 installed) env will pick up /bin/bash. Your script will work even though that other Mac doesn’t have /usr/local/bin/bash, so you gained portability.

However, /bin/bash is bash v3.2, so your script may behave differently. If the script uses bash v5 features that are not available in the 15-year-old bash v3.2, it will generate errors. Since you actively chose to install bash v5 on the first Mac, it is likely you needed some of these bash v5 features, so it is likely your script will fail on other Macs, which don’t have bash v5 installed.

You lost either predictability and reliability (which version and features are available? Does my script run successfully?), or you lose functionality (the features added to bash v5 since v3.2). To retain reliability, you can restrict the script to features that work in both bash versions. But then using the env shebang gives you no advantage, and you might as well use /bin/bash as the shebang.

Some solutions

One alternative is to use a /usr/local/bin/bash shebang for scripts which use bash v5 functionality and continue to use /bin/bash for scripts that need to run across multiple Macs, where you pay attention to using only features available in bash v3.2. You gain predictability and reliability, but your bash v5 scripts aren’t portable to other Macs. They may even fail on other Macs with bash v5 installed, if the bash v5 binary is installed in a different location.

When you use /usr/bin/env bash for a bash v5 script, it will run fine on all Macs which have bash v5 installed and the PATH configured properly to find it. (Configuring and maintaining the PATH does not happen on its own.) But the script will still fail on Macs without any bash v5. You can (and probably should) add a version check to the script, but now you are increasing code maintenance.

When you are managing a fleet of Macs, you also have the option (or in this case, I would say, the duty) to install bash v5 in a consistent location and version across all Macs in your fleet and pre-configure the proper PATH in the contexts the script will run in. Then you get predictability and functionality, but it requires extra effort in deployment and maintenance.

This requires a decently experienced MacAdmin and the proper tooling, neither of which comes for free.

Note: There are great open source solutions for macOS in this area, but I consider them ‘free, as in puppy,’ so they come with higher skill requirements and/or maintenance effort for the admin. And this isn’t supposed to imply that all commercial solutions are ‘easy to use,’ either. It’s trade-offs all the way down.

Context changes the PATH

Notice that so far I kept talking about “the default PATH for the interactive shell.”

The PATH variable may be different depending on the context, even on the same Mac with the same user. For example, when you run your script with the AppleScript do shell script command, the PATH in that context is not the same as the PATH in your interactive shell. It will be:

/usr/bin:/bin:/usr/sbin:/sbin

You can verify this by opening Script Editor and running the do shell script "echo $PATH". Other context, like scripts in installation packages will see other PATH values.

Most importantly, the PATH in these other contexts, does not contain /usr/local/bin, or any other addition you made to your PATH in the shell configuration files. An /usr/bin/env shebang will not ‘see’ a bash 5 binary you installed on the system. The same script with the same user on the same computer, will behave differently when run in a different context.

These different PATH values are an intentional choice. In these contexts, especially installation package scripts, reliability and predictability are extremely important. You do not want user and third-party installed custom binaries to interfere with the command lookup.

Sidenote on Python

With python and python3 and other run time interpreters, it gets even more difficult. There may multiple different versions installed and the behavior and functionality between versions varies more. My Mac currently has four different Python 3 binaries, each with a different version, and I not even remotely a full-time Python developer. When you call python3 on a non-developer Mac it will trigger the ‘You have to install Developer Command Line Tools’ dialog when Xcode is not installed. (Developers seem to have a hard time considering that there are Macs without Xcode installed.)

With the demise of python 2 in macOS 12.3, some developers reacted by changing the shebang in their python scripts from /usr/bin/python to /usr/bin/env python which solves nothing, when the binary goes away without replacement. Some switched to /usr/bin/env python3 which can makes things worse, by triggering the Developer Tools installation or picking a random python3 binary of the wrong version.

The only reliable solution for the mess that is python is to deploy a consistent version of Python 3 in a consistent location. You can do this by either bundling the python framework and module your tool needs together with the tool, or by deploying and maintaining the Python frameworks and modules with a management system.

MacAdmin perspective

As a MacAdmin, my scripts don’t need to be portable to systems other than macOS. They usually use tools and access files and folders that only exist on macOS. Predictability and reliability, however, are paramount. Configuration and installation scripts need to run reliably on thousands of Macs across multiple versions of macOS (even future versions) regardless of what else is installed.

As MacAdmins, we also (should) have the tools and experience to deploy and maintain binaries in predictable locations. But then, like everyone, we only have limited time, and often need to prioritize business critical software over our own tooling. So, the pre-installed interpreter binaries have little ‘friction’ to use, even if they may have a reduced functionality when compared to the latest version available elsewhere.

This is the reason bash v3.2 is still present on macOS 12.3 and it will never be easy when Apple ultimately decides to remove it. So many tools and scripts rely on /bin/bash.

(I don’t expect the removal to be any time soon, but there is a limit to how long Apple will or can keep this interpreter from 2007 on the system. We got the first warning when Apple switched the default interactive shell to zsh in Catalina. There will be more warnings …I hope. With the removal of the Python 2 binary we saw that Apple can move quickly when they feel the need. They did not even wait for a major macOS release.)

In this context, there is no gain in using /usr/bin/env. The trade-offs favor the absolute shebang very strongly.

Cross-platform portability

After this rant, you may think that I recommend against using /usr/bin/env shebangs always. But there are very good use cases. Using /usr/bin/env shebangs is the solution for workflows where cross-platform portability is required.

When your scripts need to run across multiple platforms, installing the binaries in the same location in the file system may not be possible, or require an unreasonable effort. For example, the /bin and /usr/bin are protected by SIP and the Sealed System Volume on macOS, so you cannot add your tooling there without significantly impacting the integrity and security of the entire system.

In these cases, using a /usr/bin/env shebang provides the required flexibility, so your scripts can work across platforms. But the portability does not come magically from just switching the shebang.

The target platforms need the binary to be installed and the versions should match. The installation location of the binary has to be present in the PATH in the context the script runs in. To get reliable behavior, the systems you are porting between need to be well managed, with a predictable setup and configuration of the interpreter binary and environment.

When your scripts work ‘effortlessly’ across systems with the env shebang, it is thanks to the work of the platform developers and your sysadmins/devops team for creating and maintaining this consistency. Even if you are the sole developer and devops admin, maintaining all the systems, you have to put in this work. Also the platform developers put in a lot of effort to achieve much of this consistency out of the box. As the commenter noted, some platforms don’t even agree where the env binary should be.

You gain portability at the price of increased maintenance.

Trade-offs all the way down

Alternatively, you can keep the scripts simple – restricted to a subset of features common across platforms and versions – so that the differences have no impact. Then you trade for reliability and portability at the price of functionality.

This is often the trade-off with Python (restrict the use of python features to those common among versions) and one of the reasons Python 2 was kept around for so long on macOS.

Using POSIX sh, instead of bash or zsh, is another option to gain portability, but that has its own trade-offs. Most of these trade-offs will be in functionality, but also consider not all sh emulations are equal, and supporting multiple different emulators or the real common subset of functionality, requires extra effort.

Conclusion

Shebangs with absolute paths have their strengths and weaknesses, as do shebangs with /usr/bin/env. Each has their use case and you have to understand the trade-offs when using either. Neither env shebangs nor absolute path shebangs are generally ‘better.’ Either one may get you in trouble when used in the wrong context.

When someone says ‘you should be using,’ or ‘a better way would be,’ you always need to consider their context, use case, and which trade-offs they are accepting. You need to understand the tools to use them efficiently.

Use shellcheck with BBEdit

When I teach or mentor shell scripting on macOS, I always have two recommendations: use BBEdit as your text editor and use shellcheck to verify sh and bash scripts.

I only discovered recently, that the two combine wonderfully!

It used to be that installing the shellcheck command line tool on macOS was a complicated process. Thankfully, the GitHub project now provides a pre-compiled binary for macOS, so you don’t have to mess with all of that anymore. The project labels the macOS version as ‘darwin.’ The binary is x86_64 (Intel) only and requires Rosetta on Apple silicon, but this doesn’t reduce its usefulness.

Installing shellcheck

The easiest way to install the command is to download the archive from the repo, expand the archive, and copy the shellcheck binary to /usr/local/bin/. You can follow the instructions from this earlier post to do it in Terminal or do it with your browser and in the Finder. When you download and unarchive manually, you will have to remove the quarantine flag from the shellcheck command, before you can use it.

> xattr -d com.apple.quarantine shellcheck-latest/shellcheck

I also created shellcheck recipes for AutoPkg, that automate creating a package installer for mass deployment.

Using shellcheck

Once you have the shellcheck command in place, you can use it from the command line:

> shellcheck my_great_script.sh

Sadly, shellcheck will only work with sh and bash scripts, not with zsh scripts. On the other hand, many of the common mistakes that shellcheck catches, (mostly quoting variables) are not mistakes in zsh. Still, I miss it a lot when working on large zsh scripts.

shellcheck with BBEdit

Once you have the shellcheck command installed, you can also invoke from within BBEdit: When you have a script open in BBEdit, verify that the script is recognized as a ‘Unix shell script.’ Then you can select ‘Check Syntax…’ from the ‘#!’ menu (keyboard shortcut ⌘-K). This will open a second window with all the issues shellcheck has found.

This feature was added in BBEdit 13.1, but it took me quite a while to discover it. Now I find it indispensable.

Enjoy!

Update: Installomator 9.1

We have updated Installomator. This brings Installomator to 407(!) applications! Many thanks to everyone who contributed.

Note: Both Google and Mozilla recommend using the pkg installers instead of the dmg downloads for managed deployments. So far, Installomator has provided labels for both. (googlechrome and googlechromepkg or firefox and firefoxpkg, respectively) Since there are problems with the dmg downloads, a future release of Installomator will disable the firefox and googlechrome dmg labels. You should switch to using the respective pkg labels instead.

  • added option for Microsoft Endpoint Manager (Intune) to LOGO
  • minor fixes
  • the googlechrome label now always downloads the universal version
  • 16 new labels
  • 6 updated labels

Full release notes in the repo.

Update: Installomator 9.0.1

We found a bug had snuck in to Installomator 9.0 which broke applications that download as pkgs wrapped in dmgs, so we have a bug fix update. While we were at it, there were a few other minor changes as well:

  • improved logging levels throughout the script
  • fixed a bug for pkgindmg style labels
  • changed the criteria used to locate an app in the case the it cannot be found in the default locations, this should help with some apps with similar name (Virtual Box and Box Drive)
  • new label: WhiteBox Packages (packages)
  • modified label: loom (added Apple silicon download)

You can get more details and download the pkg installer from the Installomator repo’s release page.

The unexpected return of JavaScript for Automation

Monterey has deprecated the pre-installed python on macOS. To be precise, built-in python has been deprecated since macOS Catalina, but Monterey will now throw up dialogs warning the user that an app or process using built-in python needs to be updated.

I and others have written about this before:

So far, I have recommended to build native Swift command line tools to replace python calls. However, from discussions in MacAdmins Slack, a new option has emerged. Most of the credit for popularizing and explaining this goes to @Pico (@RandomApps on Twitter) in the #bash and #scripting channels.

(Re-)Introducing JavaScript for Automation

AppleScript has been part of macOS since System 7.1. In the late nineties, there was concern that it wouldn’t make the transition to Mac OS X, but AppleScript made the jump and has happily co-existed with the Terminal and shell scripting as an automation tool on macOS. AppleScript has a very distinct set of strengths (interapplication communication) and weaknesses (awkward syntax and inconsitent application functionality and dictionaries) but it has been serving its purpose well for many users.

With Mac OS X 10.4 Tiger, Apple introduced Automator, which provided a neat UI to put together workflows. Much of Automator was based on AppleScript and users expected a more and improved AppleScript support because of that going forward. Instead, we saw AppleScript’s support from Apple and third parties slowly wane over the years.

AppleScript is stil very much present and functional in recent versions of macOS. It just seems like it hasn’t gotten much love over the last decade or so. Now that Shortcuts has made the jump from iOS, there may be hope for another revival?

The last major changes to AppleScript came with Mavericks and Yosemite. Mavericks (10.9) included a JavaScript syntax for the Open Scripting Architecture (OSA), which is the underlying framework for all AppleScript functionality. Apple called this “JavaScript for Automation.” Because this is a mouthful, it often abbreviated as JXA.

The JavaScript syntax and structure is more like a “real” programming language, than the “english language like” AppleScript. Once again this raised hopes that this could attract more scripters to AppleScript and thus encourage Apple and third party developers to support more AppleScript. But unfortunately, this positive re-inforcement did not take off.

Then Yosemite (10.10) made the AppleScript-Objective-C bridge available everywhere in AppleScript. Previously, the Objective-C bridge was only available when you built AppleScript GUI applications using AppleScript Studio in Xcode. The Objective-C bridge allows scripters to access most of the functionality of the system frameworks using AppleScript or JXA.

The coincidence of these two new features might be the reason that the ObjC bridge works much better using JXA than it does with the native AppleScript syntax.

JXA and Python

What does JXA and the AppleScriptObjC bridge have to do with the Python deprecation in modern macOS?

One reason python became so popular with MacAdmins, was that the pre-installed python on Mac OS X, also came with PyObjC, the Objective-C bridge for python. This allowed python to build applications with a native Cocoa UI, such as AutoDMG and Munki’s Managed Software Center. It also allowed for short python scripts or even one-liners to access system functionality that was otherwise unavailable to shell scripts.

For example, to determine if a preference setting in macOS is enforced with a configuration profile, you can use CFPreferences or NSUserDefaults.

Objective-C/C:

BOOL isManaged =CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

Swift:

let isManaged = CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

The Objective-C bridge allows to use this call from python, as well:

from Foundation import CFPreferencesAppValueIsForced
isManaged=CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

With JXA and the AppleScriptObjC bridge, this will look like this:

ObjC.import('Foundation');
$.CFPreferencesAppValueIsForced(ObjC.wrap('idleTime'), ObjC.wrap('com.apple.screensaver'))

Now, this looks really simple, but working with any Objective-C bridge is always fraught with strange behaviors, inconsistencies and errors and the JXA ObjC implementation is no different.

For example, I wanted to change the code above to return the value of the setting instead of whether it is managed. The CFPreferences function for that is called CFPreferencesCopyAppValue and it works fine in Swift and Python, but using JXA it only ever returned [object Ref]. The easiest solution was to switch from the CFPreferences functions to using the NSUserDefaults object:

ObjC.import('Foundation');
ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('$1').objectForKey('$2'))

(Once again many thanks to @Pico on the MacAdmins Slack for helping me and everyone else with this and also pointing out, that there is a different, somewhat complicated, solution to the object Ref problem. I will keep that one bookmarked for situations where there is no alternative Cocoa object.)

We used this to remove the python dependency from Mischa van der Bent’s CIS-Scripts.

JXA in shell scripts

To call JXA from a shell script, you use the same osascript command as for normal AppleScript, but add the -l option option to switch the language to JavaScript:

osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation');
    ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('idleTime').objectForKey('com.apple.screensaver'))
EndOfScript

For convenience, you can wrap calls like this in a shell function:

function getPrefValue() { # $1: domain, $2: key
      osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation');
    ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('$1').objectForKey('$2'))
EndOfScript
}

function getPrefIsManaged() { # $1: domain, $2: key
     osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation')
     $.CFPreferencesAppValueIsForced(ObjC.wrap('$1'), ObjC.wrap('$2'))
EndOfScript
}

echo $(getPrefValue "com.apple.screensaver" "idleTime")
# -> actual value
echo $(getPrefIsManaged "com.apple.screensaver" "idleTime")
# -> true/false

Note that the $ character does a lot of work here. It does the shell variable substitution for the function arguments in the case of $1 and $2. These are substituted before the here doc is piped into the osascript command. The $. at the beginning of the command is a shortcut where $ stands in for the current application and serves as a root for all ObjC objects.

There is also a $(…) function in JXA which is short for ObjC.unwrap(…) but I would recommend against using that in combination with shell scripts as shell’s command substitution has the same syntax and would happen before the JavaScript is piped into osascript.

There is a GitHub wiki with more detailed documentation on using JXA, and the JXA Objective-C bridge in particular.

JXA for management tasks

I’ll be honest here and admit that working with JXA seems strange, inconsistent, and — in weird way — like a step backwards. Putting together a Command Line Tool written in Swift feels like a much more solid (for lack of a better word) way of solving a problem.

However, the Swift binary command line tool has one huge downside: you have to install the binary on the client before you can use it in scripts and your management system. Now, as MacAdmins, we usually have all the tools and workflows available to install and manage software on the client. That’s what we do.

On the other hand, I have encountered three situations (set default browser, get free disk space, determine if a preference is managed) where I needed to replace some python code in the last few months and I would have no trouble finding a few more if I thought about it. Building, maintaining, and deploying a Swift CLI tool for each of these small tasks would add up to a lot of extra effort, both for me as the developer and any MacAdmin who wants to use the tools.

Alternatively, you can deploy and use a Python 3 runtime with PyObjC, like the MacAdmins Python and continue to use python scripts. That is a valid solution, especially when you use other tools built in python, like Outset or docklib. But it still adds a dependency that you have to install and maintain.

In addition to being extra work, it adds some burden to sharing your solutions with other MacAdmins. You can’t just simply say “here’s a script I use,” but you have to add “it depends on this runtime or tool, which you also have to install.

Dependencies add friction.

This is where JXA has an advantage. Since AppleScript and its Objective-C bridge are present on every Mac (and have been since 2014 when 10.10 was released) there is no extra tool to install and manage. You can “just share” scripts you build this way, and they will work on any Mac.

For example, I recently built a Swift command line tool to determine the free disk space. You can download the pkg, upload it to your management system, deploy it on your clients and then use a script or extension attribute or fact or something like to report this value to your management system. Since there is a possibility that the command line tool is not yet installed when the script runs, you need to add some code to check for that. All-in-all, nothing here is terribly difficult or even a lot of work, but it adds up.

Instead you can use this script (sample code for a Jamf extension attribute):

#!/bin/sh

freespace=$(/usr/bin/osascript -l JavaScript << EndOfScript
    ObjC.import('Foundation')
    var freeSpaceBytesRef=Ref()
    $.NSURL.fileURLWithPath('/').getResourceValueForKeyError(freeSpaceBytesRef, 'NSURLVolumeAvailableCapacityForImportantUsageKey', null)
    ObjC.unwrap(freeSpaceBytesRef[0])
EndOfScript
)

echo "<result>${freespace}</result>"

Just take this and copy/paste it in the field for a Jamf Extension Attribute script and you will get the same same free disk space value as the Finder does. If you are running a different management solution, it shouldn’t be too difficult to adapt this script to work there.

The Swift tool is nice. Once it is deployed, there are some use cases where it could be useful to have a CLI tool available. But most of the time, the JXA code snippet will “do the job” with much less effort.

Note on Swift scripts

Some people will interject with “but you can write scripts with a swift shebang!” And they are correct. However, scripts with a swift shebang will not run on any Mac. They will only run with Xcode, or at least the Developer Command Line Tools, installed. And yes, I understand this is hard for developers to wrap their brains around, but most people don’t have or need Xcode installed.

When neither of these are installed yet, and your management system attempts to run a script with a swift shebang, it will prompt the user to install the Developer command line tools. This is obviously not a good user experience for a managed deployment.

As dependencies go, Xcode is a fairly gigantic installation. The Developer Command Line Tools much less so, but we are back in the realm of “install and manage a dependency.”

Parsing JSON

Another area where JXA is (not surprisingly) extremely useful is JSON parsing. There are no built-in tools in macOS for this so MacAdmins either have to install jq or scout or fall back to parsing the text with sed or awk. Since JSON is native JavaScript, JXA “just works” with it.

For example the new networkQuality command line tool in Monterey has a -c option which returns JSON data instead of printing a table to the screen. In a shell script, we can capture the JSON in a variable and substitute it into a JXA script:

#!/bin/sh

json=$(networkQuality -c)

osascript -l JavaScript << EndOfScript
    var result=$json
    console.log("Download:  " + result.dl_throughput)
    console.log("Upload:    " + result.ul_throughput)
EndOfScript

Update: (2021-11-24) Paul Galow points out that this syntax might allow someone to inject code into my JavaScript. This would be especially problematic with MacAdmin scripts as those often run with root privileges. The way to avoid this injection is too parse the JSON data with JSON.parse :

#!/bin/sh 

json=$(networkQuality -c) 

osascript -l JavaScript << EndOfScript     
  var result=JSON.parse(\`$json\`)     
  console.log("Download:  " + result.dl_throughput)     
  console.log("Upload:    " + result.ul_throughput) 
EndOfScript

(I am leaving the original code up there for comparison.)

Conclusion

After being overlooked for years, JXA now became noticeable again as a useful tool to replace python in MacAdmin scripts, without adding new dependencies. The syntax and implementation is inconsistent, buggy, and frustrating, but the same can be said about the PyObjC bridge, we are just used it. The community knowledge around the PyObjC bridge and solutions goes deeper.

However, as flawed as it is, JXA can be a simple replacement for the classic python “one-liners” to get data out of a macOS system framework. Other interesting use cases are being discovered, such as JSON parsing. As such, JavaScript for Automation or JXA should be part of a MacAdmins tool chest.

Monterey, python, and free disk space

With Montery, many MacAdmins have been seeing dialogs that state:

“ProcessName” needs to be updated

and often the “ProcessName” is your management system. As others have already pointed out, the process, or scripts this process is calling, is using the pre-installed Python 2.7 at /usr/bin/python.

This is Apple’s next level of warning us that that the pre-installed Python (and Perl and Ruby) is deprecated and going away in “future version of macOS.” I have written about this before.

Even though the management system will be identified as the process that “needs to be updated,” the culprits are scripts and scriptlets that the management system calls for for management tasks (e.g. policies, tasks, scripts) and information gathering (e.g. extension attributes, facts, etc.). Ben Tom’s post above has information on how to identify scripts which may use python in a Jamf Pro server.

You can suppress the warning using a configuration profile. While this a useful measure to avoid confusing users with scary dialogs, you will have to start identifying and fixing scripts that are written entirely in python or just use simple python calls, and replacing them with non-python solutions.

Python 2.7 is not getting any more security patches and I assume Apple is eager to remove it from macOS. The clock is really ticking on this one.

Current User

The most common python call is probably the one which determines the currently logged in user. The python call for this was developed by Mike Lynn and popularized by Ben Toms in this post and has been a reliable MacAdmin tool for years. I have written about this and introduced a shell-based solution discovered by Erik Berglund.

But there are other use cases, where it is not so straight forward to replace the python code. The built-in python is so popular for MacAdmin tasks because it comes with PyObjC which allows access to the macOS system frameworks. With a few python calls you can avoid having to build an Objective-C or Swift command line tool.

Desktop Picture

I built desktoppr for this reason. The standard way to set a desktop picture with locking it down was a line of AppleScript. But, starting in macOS Mojave, sending AppleEvents to another process (in this case Finder) required a PPPC profile. You can also set the desktop picture using a framework call. There were python scripts out there, but the Swift solution will survive them…

Available Disk Space

Yesterday, I came across another such problem. With the recent versions of macOS, getting a value of the available disk space is not as strightforward as it used to be. There are a lot of files and data on the system, which will be cleared out when some process requires more disk space. Most of this is cache data or data that can be restored from cloud storage. But this ‘flexible’ available disk space will not be reported by the traditional tools, such as df or diskutil. The available disk space these tools report will be woefully low.

The available disk space which Finder reports will usually be much higher. There is functionality in the macOS system frameworks where apps can get the values for available that takes the ‘flexible’ files into account. There is even useful sample code!

Starting with this sample code, I built a command line tool that reports the different levels of ‘available’ disk space. When you run diskspace it will list them all. There are raw and ‘human-readable’ formats.

> diskspace                  
Available:      70621810688
Important:      231802051028
Opportunistic:  214051607271
Total:          494384795648
> diskspace -H              
Available:      70.62 GB
Important:      231.8 GB
Opportunistic:  214.05 GB
Total:          494.38 GB

The ‘Available’ value matches the actually unused disk space that df and diskutil will report. The ‘Important’ value matches what Finder will report as available. The ‘Opportunistic’ value is somewhat lower, and from Apple’s documentation on the developer page, that seems to be what we should use for automated background tasks.

For use in scripts, you can get each raw number with some extra flags:

> diskspace --available               
70628638720
> diskspace --important
231808547284
> diskspace --opportunistic
214057661159
> diskspace --total
494384795648

You can get more detail by running diskspace --help.

In Scripts

If you wanted to check if there is enough space to run the macOS Monterey upgrade (26 GB) you could do something like this:

if [[ $(/usr/local/bin/diskspace --opportunistic ) -gt 26000000000 ]]; then
     echo "go ahead"
else
    echo "not enough free disk space"
fi

Jamf Extension Attributes

Or, you can use diskspace in a Jamf Extension Attribute:

#!/bin/sh

diskspace="/usr/local/bin/diskspace"

# test if diskspace is installed
if [ ! -x "$diskspace" ]; then
    # return a negative value as error
    echo "<result>-1</result>"
fi

echo "<result>$($diskspace --opportunistic)</result>"

Since, this extension attribute relies on the diskspace tool being installed, you should have a ‘sanity check’ to see that the tool is there.

Get and install the tool

You can get the tool from the GitHub repo and I have created a (signed and notarized) installer pkg that will drop the tool in /usr/local/bin/diskspace.

Scripting macOS, part 7: Download and Install Firefox

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Follow this blog or the Twitter account for updates on the book’s progress!

Download and Install Firefox

To further illustrate the progress from the idea of a workflow to a working script, let us look at another, more involved example.

To download and install the latest version of Firefox a user has to go to the Firefox website and download the latest version, which will come as a disk image (dmg) file. Then the user needs locate the dmg in the ~/Downloads folder and open it to mount the virtual disk image. Finally, they need to copy the Firefox application from the virtual disk to the Applications folder.

When we want to automate the task ‘Download and Install Firefox,’ we have the following steps:

  • download latest Firefox disk image
  • mount downloaded disk image
  • copy Firefox application to /Applications
  • unmount disk image

From this list of steps, we can build the first ‘frame’ of our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

This breaks the workflow into smaller pieces, that we will now tackle individually.

Download from the Command Line

You can use the curl command to download data in the command line. The curl command is very complex and has many options. We will only discuss the few options that we require for our task here. As always, you can find a detailed description of the curl command and its options in the curl man page.

The URI to download the latest Firefox is
https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US

However, when you try to curl this URI, you only get the following:

> curl "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US"
<a href="https://download-installer.cdn.mozilla.net/pub/firefox/releases/86.0.1/mac/en-US/Firefox%2086.0.1.dmg">Found</a>.

This is a re-direction, that is commonly used to have a single URI, that is redirected to different final URIs, so that when the software updates, the same URI always returns the latest version.

We can tell curl to follow these redirections with the --location option.

By default, the curl command will output the download to standard out. To save the download to a file, we can use the --output option with a file name.

> curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" --output Firefox.dmg

This command will download the latest Firefox disk image to a file named Firefox.dmg in your current working directory. We can use this as our first step:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

Note: Like many other command line tools, curl has short and long options. The short options for –location and –output are -L and -o.
Short options are convenient in the interactive shell, as they save typing and reduce the potential for typos. But they are much less readable, and you usually have to look up their function in the documentation. For that reason, I recommend using the long, descriptive options in scripts.

Working with Disk Images

The command line tool to work with disk image (dmg) files on macOS is hdiutil. This is also a very powerful command with many verbs and options. You can find all the detail in the hdiutil man page.

To mount a disk image, use the attach verb:

> hdituil attach Firefox.dmg

This will output some information and mount the virtual disk. The last line ends with the path to the mounted virtual disk /Volumes/Firefox.

By default, you can see the mounted volume in Finder. We do not really need the disk image to appear in Finder while the script is running. We can suppress this behavior with the -nobrowse option.

Since we are only going to read from the disk image, we can tell hdiutil to mount the dmg in readonly mode with the -readonly option. This speeds things up a bit.

> hdiutil attach Firefox.dmg -nobrowse -readonly

You can unmount or eject the virtual disk with

> hdiutil detach -force /Volumes/Firefox

The -force option will unmount the disk image, even when another process is still using it.

Thehdiutil command covers two of our steps, so we can fill them in:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications

# unmount disk image
hdiutil detach /Volumes/Firefox -force

Copying the Application

When you manually install Firefox the disk image shows you a nice graphic that reminds you to drag the app to the Applications folder. Once the disk image is mounted, the cp command can be used to do this in the shell:

> cp -R /Volumes/Firefox/Firefox.app /Applications/

This provides the last missing step in our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications
echo "copying Firefox to /Applications"
cp -R /Volumes/Firefox/Firefox.app /Applications/

# unmount disk image
hdiutil detach /Volumes/Firefox/ -force

You can now test the script. If Firefox is running, you want to quit it before you run the script. You may also want to delete the existing copy of Firefox from the Applications folder, to be sure that your script is doing the work.

Lists of Commands—Conclusion

We have been able to automate a fairly complex workflow with a script of four commands.

To be perfectly honest, this script (as well as all the others we have built so far) is not complete yet.

A ‘proper’ script needs to be able to react to errors that occur. In our example, imagine the download fails. The script should be able to detect the failure before it overwrites the installed, functional Firefox application.

We will get to this kind of error handling later.

Nevertheless, this script is already useful in its current form. You can try to adapt this script to work with some other software you can download as a disk image.

You can also add extra commands that

  • delete the downloaded disk image at the end
  • open the newly installed Firefox app after installation
  • quit or kill the Firefox process before copying the new version

In the book “Scripting macOS”, you will learn more scripting techniques, and we will re-visit some of these sample scripts and keep improving them.

Follow this blog or the Twitter account for updates on the book’s progress!

Note: After using different variations of these kinds of workflows, I did put together a more generic script to download and install various kinds of software, called ‘Installomator.’ You can see the script at its open source repository on GitHub.

Scripting macOS, part 6: Turn it off and on again

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Turn it off an on again

A common strategy to solve problems with computers is ‘turning it off and on again.’ For example, when you have Wi-Fi connectivity problems, one of the first things you should try it to turn the Wi-Fi off, wait a few seconds, and then turn it on again.

Let’s write a script to do that.

Create the ‘frame’

When you start out writing a script, you will usually have only a vague idea of what needs to be done, and even less an idea of how to actually do it. In this case is helps to write out the steps that your script should perform:

  • turn off Wi-Fi
  • wait a few seconds
  • turn on Wi-Fi

With this list we have broken down the problem into smaller, more manageable steps, which we can each solve on their own.

I usually take this list of steps and build a script ‘frame’ from them:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi

# wait for a few seconds

# turn on Wi-Fi

We just copied our list of steps into the text file for our script and made them comments by adding the # character at the beginning of the line. We also added the shebang in the first line and two more comment lines at the beginning, which name the script and have short description on what it is supposed to do.

Since the script in this form only consists of the shebang and comments, it does nothing. But it provides a frame to fill in. Now we can tackle these steps one at a time.

Control Wi-Fi

When you want to control network related settings on macOS, the networksetup command is the first place you should look. The networksetup command allows you to configure the settings in the ‘Network’ pane in ‘System Preferences’— and a few more. The networksetup command has dozens of options. You can get a list and descriptions by running networksetup -help or in the networksetup man page.

To turn off Wi-Fi, you can use this option:

> networksetup -setairportpower <hwport> off

The value you need to use for <hwport> depends on what kind of Mac you are working on. For MacBooks, it will be en0 (unless you have a very old MacBook with a built-in ethernet port). For Macs with a single built-in ethernet port, it will be en1. For Macs with two built-in ethernet ports it will be en2.

You can also use the networksetup command to list all available hardware ports and their names:

> networksetup -listallhardwareports

Hardware Port: Wi-Fi
Device: en0
Ethernet Address: 12:34:56:78:9A:BC

Look for the Hardware Port named Wi-Fi. The value shown next to Device is the one you have to use. So, for me, on a MacBook, I will use:

> networksetup -setairportpower en0 off

We will use en0 in our sample script going forward. If your Wi-Fi port is different, remember to change it in your script going forward

Note: Apple used to brand Wi-Fi as ‘Airport’ and this naming still lingers in some parts of macOS. Changing networksetup’s options to match the new branding would break all existing scripts.

When you replace the off with on it will turn the Wi-Fi back on:

> networksetup -setairportpower en0 on

We have solved the first and third step for our script and tested them in the interactive shell. Now, we can fill them into our script frame:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi
networksetup -setairportpower en0 off 

# wait for a few seconds

# turn on Wi-Fi
networksetup -setairportpower en0 on

You can now save the script as reset_wifi.sh, set its executable bit and run it:

> chmod +x reset_wifi.sh
> ./reset_wifi.sh

This should already work. You should see the Wi-Fi icon in the dock switch to the animation indicating that it is re-connecting with your network.

Taking a break

We still want the script to ‘take a break’ and wait for a few seconds between the turning off and turning back on commands. This will allow other parts of system to ‘settle’ and react to the change.

In the shell you can use the sleep command to achieve this. It takes a single argument, the time in seconds it should pause before continuing:

> sleep 5

When you enter this command in the interactive shell, you should notice that it takes 5 seconds for the next prompt to appear. The sleep command is delaying progress for the given time.

When you watch the CPU load in Activity Monitor while you run the sleep command, you will not see a spike in load. The sleep command merely waits, letting other processes and the system do their thing.

Let us insert a ten second break into our script between the commands to turn Wi-Fi off and on:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again

# turn off Wi-Fi
networksetup -setairportpower en0 off 

# wait for ten seconds
sleep 10

# turn on Wi-Fi
networksetup -setairportpower en0 on

Now, when you run the script. You can see that Wi-Fi icon in the menu bar should switch to ‘disabled’ and then return to the ‘searching’ animation and eventually reconnect after ten seconds.

Feedback

While you are running this script, there is no feedback in the Terminal though. It just takes a disconcerting time for the prompt to return. We can add some output to let the user of the script know what is happening:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi
echo "Disabling Wi-Fi"
networksetup -setairportpower en0 off 

# wait for a ten seconds
echo "Waiting..."
sleep 10

# turn on Wi-Fi
networksetup -setairportpower en0 on
echo "Re-enabled Wi-Fi"

Now, the user will get some feedback in Terminal that lets them know what is going on.

>  ./reset_wifi.sh
Disabling Wi-Fi
Waiting...
Re-enabled Wi-Fi

Script Building Process

We formed an idea or a goal: ‘Turn Wi-Fi off and back on’

Then we described the steps necessary to achieve that goal:

  • turn off Wi-Fi
  • wait a few seconds
  • turn on Wi-Fi

We used these descriptive steps to create a frame or scaffolding script to ‘fill in’ with the proper commands.

Then we explored commands that would achieve these steps in the interactive terminal.

Once we determined the correct commands and options we placed them into our script ‘frame’ in the correct order.

As you add the commands, test whether the script shows the expected behavior.

Then we also added some output, to provide user feedback.

This is a really simple example script to illustrate this process. Nevertheless, we will re-visit these process steps with every script we build and you should follow this process (or a similar flow) when building your own scripts.

The process will not always be quite so clearly defined and you may have to iterate or repeat some of these tasks multiple times, before you find a combination of commands that do what you want. Less experienced scripters will naturally spend more time on ‘exploring commands.’ This is part of the normal learning process.

With larger, more complex scripts, each descriptive step may need to broken into even smaller sub-steps to manage the complexity. Again, experienced scripters will be able tackle more complex steps faster. Do not let that intimidate you when you are just beginning! Experience has to be built ‘the hard way’ by learning and doing.

In many situations you can learn from other people’s scripts that solve a similar problem. This is a well-proven learning strategy. I would still recommend to ‘explore’ the commands used in other people’s scripts before adding them to your scripts, as it will deepen your knowledge and experience with the commands. Sometimes, you may even be able to anticipate and avoid problems.

Next Post: Download and Install Firefox

Installomator v0.7b1 – Prerelease

We have posted a new version of Installomator. This one brings with it major changes in how we assemble the actual script. Since this is such a big change, we decided to do a beta release first.

The changes in detail:

  • script is now assembled from fragments. This helps avoid merging conflicts on git and allows the core team to work on the script logic while also accepting new labels. See the “Assemble Script ReadMe” for details.
  • Change in finding installed apps. We now look in /Applications and /Applications/Utilities first. If not found there, we use spotligt to find it. (We discovered a problem when a user has Parallels Windows installed with Microsoft Edge in it. Then Installomator wanted to update the app all the time, becaus spotligt found that Windows version of the app that Parallels created.)
  • Added bunch of new labels
  • Improved buildCaseStatement.sh a lot. It is a great start when figuring out how to create a new label for an app, or a piece of software.
  • Mosyle changed their app name from Business to Self-Service

Why the changes?

Since the Installomator.sh script has grown to over 3000 lines, its management on git has become very unwieldy. The single file with all the logic and the data required to download and install the applications creates constant merge conflicts which add to the workload of the repo admins, especially when part of the team is working on the logic of the script while we still get PRs to add labels.

Because of that we have split the main script into multiple files which are easier to manage. Having multiple files results in less merge conflicts.

What changes when I use the script?

Nothing. When you just use the Installomator.sh, you still copy its contents from the Installomator.sh script at the root of the repository into your management service (don’t forget to change the DEBUG value). Or you install the script to the clients using the installer pkg from the Releases.

The changes will only affect you when you want to build your own application labels, modify existing labels or other wise modify the script.

How do I build my own labels now?

This is where you need to learn about the new system. To reduce merge conflicts, we have broken the big Installomator.sh script into smaller pieces. There is a utility script that can assemble the script from the pieces and even run it right away fro testing. You can get the details in the “Assemble script ReadMe”

We hope that these changes will make it easier for the Installomator team and other contributors to keep growing and improving the script.

Scripting macOS, part 5: Lists of Commands

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Beyond Hello, World

Now that you have a minimal working script, let’s extend it a bit.

Create a copy of the hello.sh script file, set its executable bit, and open it in your favored text editor:

> cp hello.sh hello_date.sh
> chmod +x hello_date.sh
> bbedit hello_date.sh

A script is a list of commands that the interpreter will process sequentially. Up to now, we only have a single command, the line with echo, in our script. We will add another one:

Change the text in the script file:

#!/bin/zsh

# Greetings
echo "Hello, World!"
date

We have added a line to the script with the date command.

If you are unfamiliar with this command, you can try it out in the interactive command line:

> date    
Tue Feb 23 10:23:05 CET 2021

When you invoke date with out any arguments it will print out the current date and time. The date command has many other functions for doing date and time calculations which we will not use right now, but you can read about in the date man page.

Save the modified script and execute it:

> ./hello_date.sh
Hello, World!
Tue Feb 23 10:27:06 CET 2021

As we can tell from the output, each command in the script was executed one after the other and the output of each command was shown in Terminal.

You can insert more echo commands before the date to make the output prettier:

#!/bin/zsh

# Greetings
echo "Hello, World!"

# print an empty line
echo

# print without line break
echo -n "Today is: "

date

As we have learned earlier, empty lines and lines starting with a # character will be ignored, but can serve to explain and clarify your code. I will be using comments in the example scripts for quick explanations of new commands or options.

Here I added the -n option to the third echo command. This option suppresses the new line character or line break that echo adds automatically at the end of the text. This will then result in the output of the date command to print right after the ‘Today is:’ label, rather than in a line of its own.

Take a look:

> ./hello_date.sh
Hello, World!

Today is: Tue Feb 23 10:30:44 CET 2021

The date command allows for changing the format of the date and time output. It uses the same formatting tokens as the C strftime (string format time) function. You can read the strftime man page for details. The %F format will print the output as ‘year-month-date:’

> date +%F
2021-02-23

You can combine place holders:

> date +"%A, %F"
Tuesday, 2021-02-23

You can experiment with the date formatting placeholders from the strftime man page in the interactive shell. Once you have built a formatter that you like, you can add it to the date command in your script.

#!/bin/zsh

# Greetings
echo "Hello, World!"

# print an empty line
echo

# print without line break
echo -n "Today is: "

date +"%A, %B %d"

It is very common that you will test and iterate a command and arguments in the interactive shell before you add or insert it into your script. This can be a much easier and safer means of testing variations of commands and options than changing and saving the entire script and running it repeatedly.

Scripts are basically ‘lists of commands.’ Once you know the steps to perform a workflow in the interactive terminal, you can start building a script. Let’s look at another example.

Desktop Picture Installer

When you copy an image file to /Library/Desktop Pictures, it will appear in the list of pictures in the ‘Desktop & Screen Saver’ pane in System Preferences. On macOS 10.15 Catalina and higher you may have to create that directory.

You can easily build an installer package (pkg file) that installs an image file into that location with the pkgbuild command.

First, create a directory to hold all the sub-directories files we will need. There needs to be a payload directory in the project directory. Copy the image file into the payload directory:

> mkdir BoringDesktop
> cd BoringDesktop
> mkdir payload
> cp /path/to/BoringBlueDesktop.png payload

You can then build an installer package with the following command:

> pkgbuild --root payload --install-location "/Library/Desktop Pictures/" --identifier blog.scripting.BoringBlueDesktop --version 1.0 BoringDesktop-1.0.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to BoringDesktop-1.0.pkg

This will create a file BoringDesktop-1.0.pkg. When you double-click this file, it will open the Installer application and, when you proceed with the installation, put the image file in /Library/Desktop Pictures.

Note: To learn more about using and building installer package files for macOS, read my book “Packaging for Apple Administrators.”

The pkgbuild command has a lot of arguments and when you get any of them just slightly wrong, it may affect how the installer package works. This is a very simple example, but when you build more complex installer packages, you may be building and re-building many times and you want to avoid errors due to typos.

We can copy this big command and place it in a script file:

#!/bin/zsh
pkgbuild --root payload --install-location "/Library/Desktop Pictures/" --identifier blog.scripting.BoringBlueDesktop --version 1.0 BoringDesktop-1.0.pkg

Then make the script file executable:

> chmod +x buildBoringDesktopPkg.sh

Now you can just run the script and not worry about getting the arguments ‘just right.’

But we can go one step further and make the script more readable. We can add a comment describing what the script does.

The shell usually assumes the line break to be the end of the command. But when you place a single backslash \ as the last character in a line, the shell will continue reading the command in the next line. That way, you can break this long, difficult to read command into multiple lines:

#!/bin/zsh

# builds the pkg in the current directory

pkgbuild --root payload \
         --install-location "/Library/Desktop Pictures/" \
         --identifier blog.scripting.BoringBlueDesktop \
         --version 1.0 \
         BoringDesktop-1.0.pkg

The arguments are now more readable. When you want to change one of the values, e.g. the version, it is easier to locate.

Even though this script only contains a single command, it improves the workflow significantly, as you do not have to remember a long complex command with many arguments.

Of course, you can copy and modify this script for other package building projects.

Next Post: Turning it off an on again