Ten Years of Scripting OS X

The oldest post on this weblog happened ten years ago. It makes me especially proud, that the first post has aged quite well. (The second post, not so much.)

This weblog was (quite obviously) inspired by the weblogs of Greg Neagle, Rich Trouton, Ben Toms, and many others. I wanted to share and give back to the MacAdmins community. I have always enjoyed scripting, in the sense of tinkering and combining tools in useful, or sometimes just funny ways and sharing that seemed like an obvious thing to do.

I wasn’t really planning on honing my writing skills for writing books later. As many of these personal weblogs go, I posted quite irregularly for the first few years. Still, I was often quite excited when a post got dozens of views.

As it turns out, a weblog is also a decent marketing platform for a self-published book (or two, or three, or four). When you are writing a book, there are always a few things that don’t quite “fit in” and that then get turned into a blog post. Once I got into the habit of posting regularly, the viewer numbers and the books sales increased. At some point, my ranking in the weird black magic that is the Google algorithm reached some critical point and the views from searches started pouring in.

Having posts being read and re-shared does give you a bit of a thrill. And I wanted to share that feeling. I started the weekly news summary to have a place where all the great work of fellow MacAdmins (and related experts) is gathered. I got requests to have an email newsletter fairly quickly. We passed 1000 subscribers on the email newsletter in October, and many more read it on the website.

Last week, according to the Jetpack metrics, this site had its millionth unique visitor overall and its 500.000th unique visitor in this calendar year. Yes, that means traffic is more than doubling year over year.

Back when a post would get hundreds of views that was exciting. The traffic now is still exciting, but also a bit humbling. So, thank you all for being here and reading and sharing my posts and books.

There was never a big strategy or plan. But I am very happy how everything worked out. Over the years, writing for the weblog gave me confidence to write and self-publish a book. The books and posts lead to conference presentations, which lead to more posts and books and the newsletter. And the best part is: I got to meet and befriend some pretty great people from all over the world.

And true to the spirit of how the last ten years turned out, I have no plans or strategies for major changes in the near future. I will keep the things that work and try new things as they occur to me. That doesn’t mean that there won’t be a few minor changes happening soon, though.

On to the next ten years!

Weekly News Summary for Admins — 2020-11-20

This summary isn’t quite as big as last week’s, but very close.

This week many people got their hands on the first Macs with Apple silicon M1 chips. The reviews and benchmarks are in and it looks as if Apple wasn’t over promising. Many software vendors are shipping updates for Big Sur and Universal app support. This is definitely an interesting and busy time.

Whether your organisation can dive head-first into Big Sur and Apple silicon deployment or you have to (or want to) hold back for a while, there will be articles in here to help you.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

💻Apple Silicon M1 Macs

🌅macOS 11 Big Sur

⚙️macOS and iOS Updates

🐦MacAdmins on Twitter

  • Laura Rösler: “Every fifth Mac is on Big Sur. After 5 days (including initial crash of Apple services, weekend and issues with macOS 11 Internet Recovery) we’re up to more than 5800 Macs with macOS 11.”
  • macshome: “Remember that if you don’t have installer packages that declare arm64 support then the Installer will trigger the Rosetta 2 install as well. Even if the contents are universal.”
  • Rich Trouton: “Need the model numbers for the new Apple Silicon Macs? Mac Mini: Macmini9,1 MacBook Pro: MacBookPro17,1 MacBook Air: MacBookAir10,1 Developer Transition Kit: ADP3,2”
  • Nathaniel Strauss: “‘If upgrading from macOS Sierra or later, macOS Big Sur requires 35.5GB of available storage to upgrade. If upgrading from an earlier release, macOS Big Sur requires up to 44.5GB of available storage.’ Nearly 1/3 of a 120 GB SSD. “

🐞Bugs and Security

🔨Support and HowTos

🍏Apple Support

♻️Updates and Releases

🎧To Listen

🎈Just for Fun

📚 Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Deploying the Big Sur Installer Application

When you want to provide automated workflows to upgrade to or erase-install macOS Big Sur, you can use the startosinstall tool. You can find this tool inside the “Install macOS Big Sur” application at:

/Applications/Install macOS Big Sur.app/Contents/Resources/startosinstall

Note: Apple calls the “Install macOS *” application “InstallAssistant.” I find this a useful shorthand and will use it.

Before you can use startosinstall, you need to somehow deploy the InstallAssitant on the client system. And since the “Install macOS Big Sur” application is huge (>12GB) it poses its own set of challenges.

Different management systems have different means of deploying software. If you are using Munki (or one of the management systems that has integrated Munki, like SimpleMDM or Workspace One) you can wrap the application in a dmg. Unfortunately, even though “app in a dmg” has been a means of distributing software on macOS for nearly 20 years, most management systems cannot deal with this and expect an installer package (pkg).

You can use pkgbuild to build an installer package from an application, like this:

pkgbuild --component "/Applications/Install macOS Catalina.app" InstallCatalina-10.15.7.pkg

This works for all InstallAssistants up to and including Catalina. With a Big Sur installer application this command will start working, but then fail:

% pkgbuild --component "/Applications/Install macOS Big Sur.app/" InstallBigSur20B29.pkg
pkgbuild: Adding component at /Applications/Install macOS Big Sur.app/
pkgbuild: Inferred install-location of /Applications
pkgbuild: error: Cannot write package to "InstallBigSur20B29.pkg". (The operation couldn’t be completed. File too large)

The reason for this failure is that the Big Sur installer application contains a single file Contents/SharedSupport/SharedSupport.dmg which is larger than 8GB. While a pkg file can be larger than 8GB, there are limitations in the installer package format which preclude individual files in the pkg payload to be larger than that.

When you want to distribute the “Install macOS Big Sur” application to the clients in your fleet, either to upgrade or for an erase-and-install workflow, this limitation introduces some challenges.

You can use Composer with Jamf to create a Jamf dmg style deployment, but that will only work with Jamf Pro. You could further wrap and split the application in different containers, but that will increase the creation and deployment time.

There are a number of solutions. Each with their own advantages and downsides, some supported and recommended by Apple and some… less so. Different management and deployment styles will require different solutions and approaches.

App Deployment with MDM/VPP

When you have your MDM hooked up to Apple Business Manager or Apple School Manager, you can push applications “purchased” in the “Apps and Books” area with MDM commands. This was formerly known as “VPP” (Volume Purchase Program and I will continue to use that name, because “deploy with Apps and Books from Apple Business Manager or Apple School Manager” is just unwieldly and I don’t care what Apple Marketing wants us to call it.

Since the “Install macOS Big Sur” application is available for free on the Mac App Store, you can use VPP to push it to a client from your MDM/management system.

When you do this, the client will not get the full InstallAssistant application, but a ‘stub’ InstallAssistant. This stub is small in size (20-40MB).

The additional resouces required for the actual system upgrade or installation which are GigaBytes worth of data will be loaded when they are needed. It doesn’t matter whether the process is triggered by the user after opeing the application or by using the startosinstall or createinstallmedia tool. Either workflow will trigger the download of the additional resources.

This has the advantage of being a fast initial installation of the InstallAssistant, but then the actual upgrade or re-installation process will take so much longer, because of the large extra download before the actual installation can even begin. For certain deployment workflows, this is an acceptable or maybe even desireable trade-off.

The extra download will use a Caching Server. This approach is recommended and supported by Apple.

Mac App Store and/or System Preferences

For some user-driven deployment styles, having the user download the InstallAssistant themselves can be part of the workflow. This way, the user can control the timing of the large download and make sure they are on a “good” network and the download will not interfere with video conferences or other work.

You can direct then to the Big Sur entry in the Mac App Store with a link. You cannot search for older versions of macOS Installers in the Mac App Store, but Apple has a kbase article with direct links.

You can also use a link that leads a user directly to the Software Update pane in System Preferences and prompts the user to start the download:

# Big Sur
x-apple.systempreferences:com.apple.preferences.softwareupdate?client=bau&installMajorOSBundle=com.apple.InstallAssistant.macOSBigSur

# Catalina
x-apple.systempreferences:com.apple.preferences.softwareupdate?client=bau&installMajorOSBundle=com.apple.InstallAssistant.Catalina

When the InstallAssistant is already installed, this link will open the application. When the Mac is already running a newer version of macOS or doesn’t support the version given, it will display an error.

You can use these links from a script with the open command:

open 'x-apple.systempreferences:com.apple.preferences.softwareupdate?client=bau&installMajorOSBundle=com.apple.InstallAssistant.macOSBigSur'

The downloads initiated this way will use a Caching Server. Linking to the Mac App Store is supported and recommended by Apple. The x-apple.systempreferences links are undocumented.

softwareupdate command

Catalina introduced the --fetch-full-installer option for the softwareupdate command. You can add the --full-installer-version option to get a specific version of the installer, for example 10.15.7.

You can run this command from a managed script on the clients to install the application. The download will use a Caching Server.

This would be a really useful method to automate deployment the InstallAssistant on a client, if it were reliable. However, in my experience and that of many MacAdmins, this command is very fragile and will fail in many circumstances. As of this writing, I have not been able to reliably download a Big Sur InstallAssistant with this command. Most of the time I get

Install failed with error: Update not found 

This approach is often recommended by Apple employees, however it will have to be much more reliable before I will join their recommendation.

Please, use Feedback Assistant, preferably with an AppleSeed for IT account, to communicate your experience with this tool with Apple. If this command were reliable, then it would be my recommended solution for nearly all kinds of deployments.

InstallAssistant pkg

With these solutions so far, we have actually avoided creating an installer package, because we moved the download of the InstallAssistant to the client. A caching server can help with the network load. Nevertheless for some styles of deployments, like schools and universities, using the local management infrastucture (like repositories or distribution points) has great advantages. For this, we need a package installer for the InstallAssistant.

A “magic” download link has been shared frequently in the MacAdmins Slack that downloads an installation package from an Apple URL which installs the Big Sur InstallAssistant.

This pkg from Apple avoids the file size limit for the package payload by not having the big file in the payload and then moving it in the postinstall script. Smart hack.. er… solution!

The URL is a download link from a software update catalog. You can easily find the link for the current version with the SUS Inspector tool.

But it would be really tedious to do this on every update. You, the regular reader, know the “tedious” is a trigger word for me to write a script. In this case it was less writing a script than looting one. Greg Neagle’s installinstallmacos.py had most of the pieces needed to find the InstallAssistant.pkg in the software update catalog and download it. I merely had to put the pieces together somewhat differently.

Nevertheless, I “made” a script that downloads the latest InstallAssistant.pkg for macOS Big Sur. You can then upload this pkg to your management system and distribute it like any other installation package.

It works very much like installinstallmacos.py.

./fetch-installer-pkg.py

When you start the script it will download a lot of data into a content folder in the current working directory, parse through it and determine the Big Sur Installers in the catalog. When it finds more than one installers, it will list them and you can choose one. When it finds only one Installer, it will start downloading that immediately.

You can add the --help option for some extra options (all inherited from installinstallmacos.py.

We will have to wait for the 11.1 release to be sure this actually works as expected, but I am confident we can make it work.

This approach is very likely not supported by Apple. But neither was re-packaging the InstallAssitant from disk in Catalina. This deployment method is likely closer to the supported deployment workflows than some common existing methods.

The download does not use a Caching Server, but since the goal is to obtain a pkg that you can upload to your management server, this is not a big downside.

Big Sur signature verification check

You may have noticed that when you launch the Big Sur InstallAssistant on Big Sur for the first time, it will take a long time to “think” before it actually launches. This is due to a new security feature in Big Sur that verifies the application signature and integrity on first launch. Since this is a “big” application this check takes a while. Unfortunately Big Sur shows no progress bar or other indication. This check occurs when the user double-clicks the app to open it and when you start an upgrade or installation with the startosinstall command.

There does not seem to be a way to skip or bypass this check. You can run startosinstall --usage from a script right after installing the InstallAssistant. This will do nothing really, but force the check to happen. Subsequent launches, either from Finder or with startosinstall will be immediate.

Weekly News Summary for Admins — 2020-11-13

Usually, I gather 30-40 links a week, which I then curate into this newsletter. This week I had more than a hundred links to work through, and discovered a few more posts and articles while I put everything together. What a week!

Macs with M1

New Macs with Apple Silicon M1 chips were introduced at the “One more thing” event on Tuesday. The MacBook Air, two-port MacBook Pro 13″ and the Mac mini are now available to order with the Apple Silicon M1 chip.

While the performance numbers look really exciting, there are a few interesting caveats, mostly with regards to external displays.All Macs with the M1 chip could drive a 6K display, the MacBooks can only drive a single external display and the Mac mini can only connect two external displays (a 6K and a 4K). External GPUs are not supported. (yet?)

The four port 13″ MacBook Pro and the 16″ MacBook Pro are still availble with Intel chips, as well as a Mac mini Intel configuration, which can have more RAM than the M1 Mac mini and optional 10Gig ethernet.

The M1 Macs cap out at 16GB RAM. This is a concern for many technical minded people. However, I would remind that the iPad Pro seems to fine with even less RAM, and that the fast SSD storage used in Macs these days make the penalty for paging to “disk” much less painful than it used to be.

Big Sur

And then we got Big Sur. Apple published macOS Big Sur 11.0.1 yesterday. And then a weird thing happened. We can only guess details, but it seems that the massive 12GB download overwhelmed Apple’s distribution network, but also took down other parts of Apple’s server network. (Then again, cause end effect might be the other way around. We don’t know.) One of the servers that was not reachable for a while was the OCSP server which macOS uses to verify the developer signatures of applications. Because of this app launches were slow on Macs everywhere. Issues were resolved within a few hours.

I would just love to be a fly on the wall in this post-mortem meeting at Apple. The chain of events that must have happened to cause this must be very strange, indeed.

Nevertheless, we got Big Sur.. eventually. And with Big Sur come lots of reviews, app updates (some universal), support articles, blog post.

Enjoy this “extra big” Big Sur edition of the news!

(If I missed something, please let me know. I will add it next week!)

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

💻Apple Silicon M1 Macs

🌅 macOS 11 Big Sur and iOS 14

👩‍💻MacAdmins and Users on Big Sur

MDM vendors on Big Sur

Apple on Big Sur

Apple Developer

What’s new for Enterprise

⚙️macOS Catalina 10.15 and iOS 14 Updates

🐦MacAdmins on Twitter

  • Thomas Reed: “Scam sites that call just about everything a “virus,” and that promote junk apps to “remove” the “virus,” have become a plague. Consider macsecurity dot net, which scares people with things like a “DuckDuckGo virus” as a means for promoting ComboCleaner.” (Thread)
  • Camille Fournier: “Screw “choose boring technology,” today’s mantra is “make boring plans.” AKA, if you can break a problem down well enough that the plans look to an outsider like they are mostly boring and rote, you are probably a damn fine platform engineer.”
  • Bombich Software: “Apple fixed some of the APFS replication issues in the Big Sur 11.0.1 update, and CCC 5.1.23-b1 includes support for making bootable backups on Big Sur. To participate in this beta cycle, go to CCC’s Preferences > Software Update and check the box to be informed of beta releases.”
  • Panic: “Transmit 5.7 is ALSO now Apple silicon native on Mac, which is kind of a beautiful thing: this means that our little truck has run natively on every single Mac architecture in history, from 68k, to PPC, to Intel, now to M1. It just keeps on going!”
  • Longhorn: “So… I heard a lot of people asking: “can you run Windows on Apple M1 Macs?”( Thread)
  • Victor: “You thought kernel extensions on Apple Silicone were dead, but you were wrong.”
  • Adam C. Engst: “Apple has apparently resolved the problem with the certificate revocation server that was timing out and causing apps to open very slowly. Make sure to remove the workaround line from /etc/hosts if you added it!”
  • Carl Ashley: “Real talk @Apple @AppleSupport – the mechanisms available to #macadmins to withold OS updates and new OS releases is absolutely awful. We absolutely need the flexibility of being able to permanently block releases.”
  • Jeff Johnson: “Don’t confuse Developer ID certificate status (/usr/libexec/trustd to ocsp . apple . com) with notarization (/usr/libexec/syspolicyd to api . apple-cloudkit . com). Notarization check only occurs on first launch. Online Certificate Status Protocol can occur on any launch.” (Thread)
  • Keir Thomas: “Nice little Big Sur tweak for Disk Utility. When you click to use First Aid on a disk – which typically locks-up the Mac for a few seconds – it takes away the desktop and your apps temporarily so you CAN’T do anything.”

🐞Bugs and Security

🔨Support and HowTos

Wrangling file paths in Catalina and Big Sur – Howard Oakley

🤖Scripting and Automation

🍏Apple Support

♻️Updates and Releases

📺To Watch

🎧To Listen

📚 Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Book Update for Big Sur – Moving to zsh v4

I have pushed an update for the “Moving to zsh” book.

Big Sur is such an important update that gave it the long-awaited version number ’11.’ Thankfully, it did not bring many changes to the way the Terminal and zsh work.

zsh is still the (new) default shell for new users. bash v3 is (so far) still present on macOS Big Sur, but when you use it as your shell, you will get the warning to switch in Terminal. While Big Sur updates zsh to version 5.8 this doesn’t change any major behavior compared to zsh 5.7.1 in zsh.

Because of all this, I only had to do a few minor updates to the book.

I do anticipate that many user who have been holding off from upgrading to Catalina (or older versions of macOS), will now either upgrade to Catalina or leap frog directly to Big Sur. For those users, this book is ready to help them “Move to zsh.” Please, recommend this book when you encounter one of these users.

As usual, the update is free if you have already purchased the book. You should get a notification from the Books application to update. (On macOS, I have seen that it can help to delete the local download of the book to force the update. It might still take a few hours for the change to propagate through Apple’s server network. Even when you get the older version now, you can re-download the update when it is available.)

When you haven’t gotten the book yet, you can purchase it on the Apple Books store.

If you are enjoying the book, please rate it on the Books store, or even leave a review. These really help, thank you!

The changes are listed here. This list is also in the ‘Version History’ section in the book. There, you will get links to the relevant section of the book, so you can find the changes quickly.

  • Description of the new zsh session restore feature in Big Sur Terminal
  • Updated some images and text with macOS Big Sur information
  • Added a link to a post with instructions to install shellcheck on macOS

Weekly News Summary for Admins — 2020-11-06

The news this week has certainly been… well… captivating on many levels.

Yesterday, Apple also dropped iOS 14.2 (and siblings) and a “macOS Catalina 10.15.7 Supplemental Update.” (I had hoped we were done with these.) Also, a “release candidate” for macOS Big Sur 11.0.1.

Next week, is the long anticipated third Apple Event this Fall, where everybody is expecting the first Apple Silicon Macs to be introduced and a macOS Big Sur release date to be announced. Whatever your process to prepare for a major macOS release is, I hope you are ready!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Headlines

News and Opinion

macOS 11 Big Sur and Apple Silicon

macOS Catalina 10.15 and iOS 14 Updates

Coronavirus and Remote Work

MacAdmins on Twitter

  • Patrick Fergus: “Did you know (an Adobe Admin Console package of) Acrobat won’t install while Safari is running? Has anyone mentioned to the Acrobat team the reason blocking on Safari is because of the Acrobat NPAPI browser plugin Safari hasn’t loaded in a few years?”
  • Tim Perfitt: “I have been working with CryptoTokenKit’s persistent tokens on iOS 14.2 and something we have been hoping for for a long time on iOS. An app can vend a identity (private key + certificate) that can be consumed by other apps AND iOS itself.” (Thread)

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Listen

Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Scripting OS X — Weekly News Summary for Admins — 2020-10-30

A strange thing happened last Friday: while I was putting together last week’s news summary, HP printer connections on Macs all over started showing strange error messages. First suspicions were on the Xprotect update published a day before, but then HP had to admit that they had ‘accidentally’ revoked a certificate that was still very much in active use. As usual, Howard Oakley and Mr Macintosh covered the issue in much detail.

macOS Big Sur is still in beta did a surprising step from “11.0 beta 10” to “11.0.1 beta 1.” This probably means that there will be new Mac hardware soon that ships with a version of 11.0, and immediately prompt for an update when they are run for the first time.

Even though the 2020 conference season is not over yet—you can register for Everyworld here—Jamf has gone ahead and announced that JNUC 2021 will also be virtual. This will planning uncertainty for the organizers, speakers, attendees. Virtual JNUC 2020 was already a great event, but there were many aspects that can be improved on and I am happy that Jamf is taking the time necessary to take that on.

On the topic of conferences, the MacAdmins Podcast had Patrik Jerneheim from the MacSysAdmins Conference talking about the experience of MacSysAdmin Online.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

News and Opinion

macOS 11 Big Sur and iOS 14

Coronavirus and Remote Work

MacAdmins on Twitter

  • Alexander Blach: “TIL: you can use a Lightning to USB3 Camera Adapter to migrate data to a new iPhone using a wired connection (no Wi-Fi) while also charging both iPhones from a single power adapter.”
  • Tim Perfitt: “Yes, this is probably from Xcode, but Xcode isn’t running nor is an iOS simulator. I have no idea what to answer here and I am an Apple developer since before Swift! How would I know if I should trust this? If I deny it, there could be issues with development” (Thread)
  • Mr. Macintosh: “As to why we are getting 11.0.1 This could mean that 11.0 is GM for Apple Silicon Macs currently in production. When they arrive in customers hands, 11.0.1 will show as an available update. Doing it this way hides the AS installer data so we can’t look at the new DeviceIDs.”
  • Erik Gomez: “I think it’s less to hide things and more that Big Sur just isn’t ready.”

Bugs and Security

HP Printer certificates

Support and HowTos

Scripting and Automation

Updates and Releases

To Listen

Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Weekly News Summary for Admins — 2020-10-23

Bit of a lull in the news between the previous iOS 14 and forthcoming macOS 11 releases. Apple is shipping the iPhone 12, iPhone 12 Pro, and iPad Air 4 this week and the devices look good in the reviews.

No reason to be bored, though. You probably haven’t caught up with the sessions from Virtual JNUC and MacSysAdmin Online, yet! (To be honest, there are some Campfire sessions I have not gotten around to watch yet.)

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

News and Opinion

iOS 14 Updates

MacAdmins on Twitter

  • Patrick Wardle: “Previously, a comprehensive macOS firewall could be implemented via a Network Kernel Extension (kext) Apple deprecated kexts, giving us Network Extensions….but apparently (many of) their apps / daemons bypass this filtering mechanism. Are we ok with this!?” (Thread, Michael Tsai)
  • Joel Rennich: “Let’s talk attestation and boxes and what that means for generating keys on an iOS device. For a while folks, myself included, have been really interested in getting Apple to provide attestation statements about a key being in the Secure Enclave.” (Thread)

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Listen

Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Dealing with xpath changes in Big Sur

In one of the recent betas for macOS 11.0 Big Sur, the xpath command line tool changed. Big Sur uses the 5.28 version of the tool, while Catalina defaults to the 5.18 version.

These aren’t version numbers for xpath but actually version numbers for perl. When you look at all the xpath executables you will see that they are actually perl scripts.

> head -n1 /usr/bin/xpath*
==> /usr/bin/xpath <==
#!/usr/bin/perl

==> /usr/bin/xpath5.18 <==
#!/usr/bin/perl5.18 -w

==> /usr/bin/xpath5.28 <==
#!/usr/bin/perl5.28

The perl environment will choose the xpath script ending in the version number matching the perl version automatically. The plain xpath script with no version number serves as fallback.

Either way, Catalina runs perl 5.18 and xpath5.18 and Big Sur runs perl 5.28 and the newer xpath5.28 script.

The problem here is that the newer xpath script has a different syntax:

[5.18] xpath [filename] query
[5.28] xpath [options] -e query [-e query...] [filename...]

Either version will use stdin when there is no file name, but the newer xpath requires the query string to be labeled with a -e argument.

So, your scripts that are using the xpath tool to parse XML data, will fail in Big Sur!

The easiest fix is to change the script to use the new syntax, i.e. insert the -e at the right place and if necessary re-arrange the arguments. But then the script will fail on older versions of macOS. Many of us will have to write our scripts to be able to support the latest and older versions of macOS for a transition time. For some MacAdmins the transition time can be several years.

Since Big Sur still includes xpath5.18, another solution is to just hardcode the version whenever you use xpath. But this will only defer the problem to a future version of macOS, when the 5.18 version of the script is removed.

A better solution is to check the version of macOS that is being used and to call xpath with the proper for each version:

    if [[ $(sw_vers -buildVersion) > "20A" ]]; then
        xpath -e "//query" "/path/to/file"
    else
        xpath "/path/to/file" "//query" 
    fi

This will call xpath with the new syntax on Big Sur (20A) and higher and use the older syntax otherwise. (Why I use the build version.)

If you are using xpath in multiple locations in a script, using this code everywhere will become tedious. In the Installomator script, we often use xpath to parse the download URL out of an XML file. As of now, there are nine occurences of xpath, but as more people contribute to Installomator, this is likely to go up.

There is a nice sleight-of-hand trick you can use to make the script resilient to change in Big Sur, without having to touch every use of xpath in the code.

At the beginning of the script (before the first use of xpath) insert this function:

xpath() {
    # the xpath tool changes in Big Sur 
    if [[ $(sw_vers -buildVersion) > "20A" ]]; then
        /usr/bin/xpath -e "$@"
    else
        /usr/bin/xpath "$@"
    fi
}

Since the shell interpreter (bash or zsh) will prefer a local function over an external executable, this function will now be used for all uses of xpath in the remaining code. We don’t need to touch them at all.

Within the function, we tell the interpreter to use the executable by using its full path.

In Installomator, we pipe data into xpath, so there are no files involved and the above works fine.

When you use xpath with files, it gets a bit more complicated, because the order of the arguments changed between the versions.

With the assumption that the current use is xpath [filename] query we can use the following:

xpath() { # [filename] query
    # xpath in Big Sur changes syntax
    if [[ -n $2 ]]; then
        local query=$2
        local filename=$1
    else 
        local query=$1
        local filename=""
    fi

    if [[ $(sw_vers -buildVersion) > "20A" ]]; then
        /usr/bin/xpath -e "$query" "$filename"
    else
        /usr/bin/xpath "$filename" "$query"
    fi
}

You will want to make a note for some point in the future, when you finally can drop Catalina support, to revisit these scripts and clean them up for the new syntax.

I have not yet encountered other command line tools that change in a similar way in Big Sur. If you do, you should be able to use a similar function to simplify the transtition.

When you do, let us know in the comments!

Installomator Updated: v0.4

I had been fairly busy with both the JNUC and MacSysAdmin presentations, but work on Installomator has continued.

I also did an overview of what Installomator does in my MacSysAdmin Online presentation: “Practical Scripting”.

Many of these new features have been provided from others, either through GitHub issues, pull requests, or through comments in the #installomator channel on MacAdmins Slack. Thanks to all who contributed.

What’s new in v0.4:

  • you can now set script variables as an argument in the form VARIABLE=value. More detail on this in the README file, ‘Configuration from Arguments.’
  • change downloadFromGit to match file types better
  • implemented a workaround for changed behavior of xpath in Big Sur
  • added an option prompt_user_the_kill to BLOCKING_PROCESS_ACTION which will kill the process after the third unsuccessful attempt to quit
  • added several new labels for total of 116

Get the script and find the instructions on the GitHub repo.

If you have any feedback or questions, please join us in the #installomatorchannel on MacAdmins Slack.

Thanks again to all those who contributed!