Dealing with xpath changes in Big Sur

In one of the recent betas for macOS 11.0 Big Sur, the xpath command line tool changed. Big Sur uses the 5.28 version of the tool, while Catalina defaults to the 5.18 version.

These aren’t version numbers for xpath but actually version numbers for perl. When you look at all the xpath executables you will see that they are actually perl scripts.

> head -n1 /usr/bin/xpath*
==> /usr/bin/xpath <==
#!/usr/bin/perl

==> /usr/bin/xpath5.18 <==
#!/usr/bin/perl5.18 -w

==> /usr/bin/xpath5.28 <==
#!/usr/bin/perl5.28

The perl environment will choose the xpath script ending in the version number matching the perl version automatically. The plain xpath script with no version number serves as fallback.

Either way, Catalina runs perl 5.18 and xpath5.18 and Big Sur runs perl 5.28 and the newer xpath5.28 script.

The problem here is that the newer xpath script has a different syntax:

[5.18] xpath [filename] query
[5.28] xpath [options] -e query [-e query...] [filename...]

Either version will use stdin when there is no file name, but the newer xpath requires the query string to be labeled with a -e argument.

So, your scripts that are using the xpath tool to parse XML data, will fail in Big Sur!

The easiest fix is to change the script to use the new syntax, i.e. insert the -e at the right place and if necessary re-arrange the arguments. But then the script will fail on older versions of macOS. Many of us will have to write our scripts to be able to support the latest and older versions of macOS for a transition time. For some MacAdmins the transition time can be several years.

Since Big Sur still includes xpath5.18, another solution is to just hardcode the version whenever you use xpath. But this will only defer the problem to a future version of macOS, when the 5.18 version of the script is removed.

A better solution is to check the version of macOS that is being used and to call xpath with the proper for each version:

    if [[ $(sw_vers -buildVersion) > "20A" ]]; then
        xpath -e "//query" "/path/to/file"
    else
        xpath "/path/to/file" "//query" 
    fi

This will call xpath with the new syntax on Big Sur (20A) and higher and use the older syntax otherwise. (Why I use the build version.)

If you are using xpath in multiple locations in a script, using this code everywhere will become tedious. In the Installomator script, we often use xpath to parse the download URL out of an XML file. As of now, there are nine occurences of xpath, but as more people contribute to Installomator, this is likely to go up.

There is a nice sleight-of-hand trick you can use to make the script resilient to change in Big Sur, without having to touch every use of xpath in the code.

At the beginning of the script (before the first use of xpath) insert this function:

xpath() {
    # the xpath tool changes in Big Sur 
    if [[ $(sw_vers -buildVersion) > "20A" ]]; then
        /usr/bin/xpath -e "$@"
    else
        /usr/bin/xpath "$@"
    fi
}

Since the shell interpreter (bash or zsh) will prefer a local function over an external executable, this function will now be used for all uses of xpath in the remaining code. We don’t need to touch them at all.

Within the function, we tell the interpreter to use the executable by using its full path.

In Installomator, we pipe data into xpath, so there are no files involved and the above works fine.

When you use xpath with files, it gets a bit more complicated, because the order of the arguments changed between the versions.

With the assumption that the current use is xpath [filename] query we can use the following:

xpath() { # [filename] query
    # xpath in Big Sur changes syntax
    if [[ -n $2 ]]; then
        local query=$2
        local filename=$1
    else 
        local query=$1
        local filename=""
    fi

    if [[ $(sw_vers -buildVersion) > "20A" ]]; then
        /usr/bin/xpath -e "$query" "$filename"
    else
        /usr/bin/xpath "$filename" "$query"
    fi
}

You will want to make a note for some point in the future, when you finally can drop Catalina support, to revisit these scripts and clean them up for the new syntax.

I have not yet encountered other command line tools that change in a similar way in Big Sur. If you do, you should be able to use a similar function to simplify the transtition.

When you do, let us know in the comments!

Installomator Updated: v0.4

I had been fairly busy with both the JNUC and MacSysAdmin presentations, but work on Installomator has continued.

I also did an overview of what Installomator does in my MacSysAdmin Online presentation: “Practical Scripting”.

Many of these new features have been provided from others, either through GitHub issues, pull requests, or through comments in the #installomator channel on MacAdmins Slack. Thanks to all who contributed.

What’s new in v0.4:

  • you can now set script variables as an argument in the form VARIABLE=value. More detail on this in the README file, ‘Configuration from Arguments.’
  • change downloadFromGit to match file types better
  • implemented a workaround for changed behavior of xpath in Big Sur
  • added an option prompt_user_the_kill to BLOCKING_PROCESS_ACTION which will kill the process after the third unsuccessful attempt to quit
  • added several new labels for total of 116

Get the script and find the instructions on the GitHub repo.

If you have any feedback or questions, please join us in the #installomatorchannel on MacAdmins Slack.

Thanks again to all those who contributed!

Weekly News Summary for Admins — 2020-10-16

We got the expected new iPhones 12 this week, all four models, and a new HomePod mini. I am actually looking forward to a smaller iPhone 12 mini to replace my iPhone X.

In other (some would say good) news, macOS 11 Big Sur was not released this week, but we got beta 10 instead. There is an expectation for another Apple event introducing the first Apple Silicon based Macs in the upcoming weeks and that would be an obvious time to release Big Sur to everyone.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

News and Opinion

macOS 11 Big Sur, Apple Silicon, and iOS 14

MacAdmins on Twitter

  • Taha Karim: “MacOS Ransomware in one tweet” (Thread, via Mat X)
  • William Smith: “Tomorrow (October 13, 2020): Support for Microsoft Office 2016 for Mac — any version 16.16.x and lower— ends. No more software updates to fix: • Bugs • Compatibility problems • Security vulnerabilities – No macOS Big Sur support”

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

To Listen

Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Weekly News Summary for Admins — 2020-10-09

For now this has been a quiet week. Since Apple announced another event next week, possibly a calm before the storm. After Apple announced iOS 14 with 24 hours notice, MacAdmins are understandably a bit nervous. Also, no new betas this week (so far).

On the other hand, after JNUC last week, we had MacSysAdmin this week. I have not been able to watch all presentations yet, but those that I have seen have been worth it and I am looking forward to the rest.

In other news, this newsletter just passed the 1000 email subscribers number! Thank you all for reading, and on to the next thousand!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

News and Opinion

macOS 11 Big Sur and iOS 14

MacAdmins on Twitter

  • Eliz: “Please don’t say just “Hello” in a chat #NoHello” (Read thread and links for background. I find myself doing that a lot, too, and have resvoled to do better.)

Bugs and Security

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

To Listen

Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Installomator Updated: v0.3

It’s been more than a month since the last update, and while there has been work on the dev branch, I was quite distracted with other things (like this). The good news is, that there have been quite a few contributions from others! A huge thanks to all who helped make this a better script.

All it took was for me to find some time to put all the contributions together, which I finally found some time for.

What’s new in v0.3:

  • added several new labels for total of 98
  • removed the powershell labels, since the installer is not notarized
  • when run without any arguments, the script now lists all labels
  • changed how zips are expanded because this was broken on Mojave
  • improved logging in some statements
  • several more minor improvements

Get the script and find the instructions on the GitHub repo.

Some of the contributions and requests have not yet been addressed. I believe they will require some more thinking and planning. I would like to approach those in the next version.

If you have any feedback or questions, please join us in the #installomator channel on MacAdmins Slack.

Thanks again to all those who contributed!

Installomator updated: v0.2

It’s been nearly a month since I introduced Installomator.

Since then, it has gotten lots of feedback from others and many contributions. As the changes, fixes and additional apps have accumulated, I have created a 0.2 release to get a stable new version. If you like living on the edge you can also use the dev branch for the latest update.

Changes in this version:

  • many fixes for broken URLs and other bugs
  • pkgInDmg and pkgInZip now search for the first pkg file in the archive in case the file name varies with the version
  • notification on successful installation can be suppressed with the NOTIFY variable
  • Apple signed installers and apps that don’t have a Team ID are verified correctly now
    improved logging
  • several new applications: count increased from 62 in v0.1 to 87 in v0.2

Thanks to all who contributed!

Also, if you haven’t already, you want to read Mischa’s guest post on using Installomator with Jamf Pro.

Advanced Quoting in Shell Scripts

Quoting strings and variable substitutions is a bit of a dark art in shell scripts. It looks simple and straightforward enough, but there are lots of small devils in the details, that can come out and haunt you.

Basics: why we quote strings

In shell scripts (sh, bash, and zsh) you use the equals character = to assign a string value to a variable:

> name=John
> dirpath=/Library

As long as there are no special characters in the literal string, there is no need to quote the string.

When you use the variable, you prefix a $ symbol:

> echo $name
John
> cd $dirpath
> pwd
/Library

When the literal string contains special characters, you need to either escape the special characters with the backslash \ or quote the entire string with either single quotes ' or double quotes ". Space is proverbial ‘killer character’, especially for file paths. (More details in this post.)

name='John Doe'
dirpath="/Library/Application Support"

The difference between single quotes and double quotes is important. Single quotes escape every special character except the single quote itself. A single quoted string of '#$"\!' will represent exactly those characters.

Double quotes escape most characters, except the double quote " the backtick `, the dollar sign $, the backslash \, and the exclamation mark !. (There are slight differences between the shells on this.)

This allows us to use old-style command substitution with backticks and variable substitution (dollar sign) within double quoted strings:

> echo "Hello, $name"
Hello, John Doe
> echo "The Computer Name is `scutil --get ComputerName`"

Though you should be using the $(…) syntax for command substitution instead of backticks `. The parenthesis syntax is more readable and can be nested.

In general, it is a good rule to always quote literal strings. Whether you should use double quotes or single quotes depends on the use case.

Combining literal strings with special characters

Things can start getting complicated when you want special characters with their special functionality. For example, when you want to refer to the path ~/Library/Application Support, you should put it in quotes, because of the space. But when you put the ~ in the quotes, it will not be substituted to the user’s home directory path.

There are a few ways to solve this problem. You could escape the space with a backslash. You could use the $HOME variable instead (but be sure you are in a context where this is set). But the easiest is to move the special character out of the quotes:

dirpath=~"/Library/Application Support"

Quotes in quotes

Sometimes it is necessary to have a set of quotes within quotes. A common situation for MacAdmins is the following osascript:

osascript -e 'display dialog "Hello, World"'

The osascript command can be used to run Apple commands or scripts. Since AppleScript uses double quotes for literal strings, the entire AppleScript command is passed in single quotes. This keep the command string together and the double quotes in single quotes don’t confuse the shell.

This works fine, until you want to do something like this:

computerName=$(scutil --get ComputerName)
newName=$(osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "$computerName")')

Again, we put the AppleScript command in single quotes, so we can use double quotes inside. But now, the single quotes are also blocking the variable substitution and we get the literal $computerName in the dialog.

There are a few solutions out of this, I will demonstrate three:

First, you could close the single quotes before the variable substitution and re-open them after:

osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'$computerName'")'

This will in this form as long as $computerName contains no spaces. This is unlikely as the default computer name is something like Armin's MacBook Pro. The shell will consider that space a separator before a new argument, breaking the AppleScript command into meaningless pieces and failing the osascript command. We can avoid that by putting the substitution itself in double quotes:

osascript -e 'text returned of (display dialog "Enter Computer Name" default answer "'"$computerName"'")'

This works and is entirely legal syntax, but not very legible.

Escaping the escape characters

Another solution is to use double quotes for the entire AppleScript command, we can use variable substitution inside. But then we have to deal with the double quotes required for the AppleScript string literal. The good news here is that we can escape those with the backslash:

osascript -e "text returned of (display dialog \"Enter Computer Name\" default answer \"$computerName\")"

This doesn’t win prizes for legibility either, but I consider it an improvement over the previous approach.

Here Docs

The above approaches with work in sh, bash, and zsh. But bash and zsh have another tool available that can work here. The ‘here doc’ syntax can be used to include an entire block of AppleScript code in a bash or zsh script:

#!/bin/bash

computerName=$(scutil --get ComputerName)

newName=$(osascript <<EndOfScript
    text returned of (display dialog "Enter Computer Name" default answer "$computerName")
EndOfScript
)

echo "New name: $newName"

The syntax is a bit weird. The <<EndOfScript says: take all the text until the next appearance of EndOfScript and pipe it into the preceding command, in this case osascript.

The ‘marker’ EndOfScript is entirely arbitrary. Many people choose EOF but I prefer something a little more descriptive. Whatever label you choose the ending marker has to stand alone in its line. This is why the parenthesis ) which closes the command substition $( has to stand alone in the next line.

You can still use variable substitution in a here doc, so the variable $computerName will be substituted before the here doc is piped into osascript.

Wrangling Pythons

As I noted in my last Weekly News Summary, several open source projects for MacAdmins have completed their transition to Python 3. AutoPkg, JSSImport and outset announced Python 3 compatible versions last week and Munki already had the first Python 3 version last December.

Why?

Apple has included a version of Python 2 with Mac OS X since 10.2 (Jaguar). Python 3.0 was released in 2008 and it was not fully backwards compatible with Python 2. For this reason, Python 2 was maintained and updated alongside Python 3 for a long time. Python 2 was finally sunset on January 1, 2020. Nevertheless, presumably because of the compatibility issues, Apple has always pre-installed Python 2 with macOS and still does so in macOS 10.15 Catalina. With the announcement of Catalina, Apple also announced that in a “future version of macOS” there will be no pre-installed Python of any version.

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. (macOS 10.15 Catalina Release Notes)

This also applies to Perl and Ruby runtimes and other libraries. I will be focussing on Python because it is used more commonly for MacAdmin tools, but most of this post will apply equally to Perl and Ruby. Just mentally replace “Python” for your preferred language.

The final recommendation is what AutoPkg and Munki are following: they are bundling their own Python runtime.

How to get Python

There is a second bullet in the Catalina release notes, though:

Use of Python 2.7 isn’t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS won’t include Python 2.7. Instead, it’s recommended that you run python3 from within Terminal. (51097165)

This is great, right? Apple says there is a built-in Python 3! And it’s pre-installed? Just move all your scripts to Python 3 and you’ll be fine!

Unfortunately, not quite. The python3 binary does exist on a ‘clean’ macOS, but it is only a stub tool, that will prompt a user to download and install the Command Line Developer Tools (aka “Developer Command Line Tools” or “Command Line Tools for Xcode”). This is common for many tools that Apple considers to be of little interest to ‘normal,’ non-developer users. Another common example is git.

Dialog prompting to install the Command Line Tools
Dialog prompting to install the Command Line Tools

When you install Xcode, you will also get all the Command Line Developer Tools, including python3 and git. This is useful for developers, who may want to use Python scripts for build operation, or for individuals who just want to ‘play around’ or experiment with Python locally. For MacAdmins, it adds the extra burden of installing and maintaining either the Command Line Developer Tools or the full Xcode install.

Python Versions, a multitude of Snakes

After installing Xcode or the Command Line Developer Tools, you can check the version of python installed: (versions on macOS 10.15.3 with Xcode 11.3.1)

> python --version    
Python 2.7.16
> python3 --version    
Python 3.7.3

When you go on the download page for Python.org, you will get Python 3.8.1 (as of this writing). But, on that download page, you will also find download links for “specific versions” which include (as of this writing) versions 3.8.1, 3.7.6, 3.6.10, 3.5.9, and the deprecated 2.7.17.

The thing is, that Python isn’t merely split into two major release versions, which aren’t fully compatible with each other, but there are several minor versions of Python 3, which aren’t fully compatible with each other, but are still being maintained in parallel.

Developers (individuals, teams, and organisations) that use Python will often hold on to a specific minor (and sometimes even patch) version for a project to avoid issues and bugs that might appear when changing the run-time.

When you install the latest version of Munki, it will install a copy of the Python framework in /usr/local/munki/ and create a symbolic link to that python binary at /usr/local/munki/python. You can check its version as well:

 % /usr/local/munki/python --version
Python 3.7.4

All the Python code files for Munki will have a shebang (the first line in the code file) of

#!/usr/local/munki/python

This ensures that Munki code files use this particular instance of Python and no other copy of Python that may have been installed on the system.

The latest version of AutoPkg has a similar approach:

> /usr/local/autopkg/python --version    
Python 3.7.5

In both cases the python binary is a symbolic link. This allows the developer to change the symbolic link to point to a different Python framework. The shebangs in the all the code files point to the symbolic link, which can be changed to point to a different Python framework.

This is useful for testing and debugging. Could MacAdmins use this to point both tools to the same Python framework? Should they?

The Bridge to macOS

On top of all these different versions of Python itself, many scripts, apps, and tools written in Python rely on ‘Python modules.’ These are libraries (or frameworks) of code for a certain task, that can be downloaded and included with a Python installation to extend the functionality of Python.

The most relevant of these modules for MacAdmins is the “Python Objective-C Bridge.” This module allows Python code to access and use the native macOS Cocoa and CoreFoundation Frameworks. This not only allows for macOS native GUI applications to be written in Python (e.g. AutoDMG and Munki’s Managed Software Center [update: MSC was re-written in Swift last year]), but also allows short scripts to access system functions. This is sometimes necessary to get a data that matches what macOS applications “see” rather than what the raw unix tools see.

For example, the defaults tool can be used to read the value of property lists on disk. But those might not necessarily reflect the actual preference value an application sees, because that value might be controlled by a different plist file or configuration profile.

(Shameless self-promotion) Learn more about Property lists, Preferences and Profiles

You could build a tool with Swift or Objective-C that uses the proper frameworks to get the “real” preference value. Or you can use Python with the Objective-C bridge:

#!/usr/bin/python
from Foundation import CFPreferencesCopyAppValue
print CFPreferencesCopyAppValue("idleTime", "com.apple.screensaver")

Three simple lines of Python code. This will work with the pre-installed Python 2.7, because Apple also pre-installs the Python Objective-C bridge with that. When you try this with the Developer Tools python3 you get an error:

ModuleNotFoundError: No module named 'Foundation'

This is because the Developer Tools do not include the Objective-C bridge in the installation. You could easily add it with:

> sudo python3 -m pip install pyobjc

But again, while this command is “easy” enough for a single user on a single Mac, it is just the beginning of a Minoan labyrinth of management troubles.

Developers and MacAdmins, have to care about the version of the Python they install, as well as the list of modules and their versions, for each Python version.

It is as if the Medusa head kept growing more smaller snakes for every snake you cut off.

(Ok, I will ease off with Greek mythology metaphors.)

You can get a list of modules included with the AutoPkg and the Munki project with:

> /usr/local/munki/python -m pip list
> /usr/local/autopkg/python -m pip list

You will see that not only do Munki and AutoPkg include different versions of Python, but also a different list of modules. While Munki and AutoPkg share many modules, their versions might still differ.

Snake Herding Solutions

Apple’s advice in the Catalina Release Notes is good advice:

It’s recommended that you bundle the runtime within the app.

Rather than the MacAdmin managing a single version of Python and all the modules for every possible solution, each tool or application should provide its own copy of Python and its required modules.

If you want to build your own Python bundle installer, you can use this script from Greg Neagle.

This might seem wasteful. A full Python 3 Framework uses about 80MB of disk space, plus some extra for the modules. But it is the safest way to ensure that the tool or application gets the correct version of Python and all the modules. Anything else will quickly turn into a management nightmare.

This is the approach that Munki and AutoPkg have chosen. But what about smaller, single script solutions? For example simple Python scripts like quickpkg or prefs-tool?

Should I bundle my own Python framework with quickpkg or prefs-tool? I think that would be overkill and I am not planning to do that. I think the solution that Joseph Chilcote chose for the outset tool is a better approach for less complex Python scripts.

In this case, the project is written to run with Python 3 and generic enough to not require a specific version or extra modules. An admin who wants to use this script or tool, can change the shebang (the first line in the script) to point to either the Developer Tool python3, the python3 from the standard Python 3 installer or a custom Python version, such as the Munki python. A MacAdmin would have to ensure that the python binary in the shebang is present on the Mac when the tool runs.

You can also choose to provide your organization’s own copy Python with your chosen set of modules for all your management Python scripts and automations. You could build this with the relocatable Python tool and place it in a well-known location the clients. When updates for the Python run-time or modules are required, you can build and push them with your management system. (Thanks to Nathaniel Strauss for pointing out this needed clarifying.)

When you build such scripts and tools, it is important to document which Python versions (and module versions) you have tested the tool with.

(I still have to do that for my Python tools.)

What about /usr/bin/env python?

The env command will determine the path to the python binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the python binary is unknown.

This is useful when developers want to use the same script in different environments across different computers, user accounts, and platforms. However, this renders the actual version of python that will interpret the script completely unpredictable.

Not only is it impossible to predict which version of Python will interpret a script, but you cannot depend on any modules being installed (or their versions) either.

For MacAdmin management scripts and tools, a tighter control is necessary. You should use fixed, absolute paths in the shebang.

Conclusion

Managing Python runtimes might seem like a hopeless sisyphean task. I believe Apple made the right choice to not pre-install Python any more. Whatever version and pre-selection of module versions Apple would have chosen, it would only have been the correct combination for a few Python solutions and developers.

While it may seem wasteful to have a multitude of copies of the Python frameworks distributed through out the system, it is the easiest and most manageable solution to ensure that each tool or application works with the expected combination of run-time and modules.

Command-Control-Power Podcast featuring Scripting OS X!

I have the honor of being a guest on the latest Command-Control-Power podcast!

The wonderfully gracious hosts Joe Saponare and Sam Valencia talk with me about life as admin/consultant, my new book “Moving to zsh,” what the new shell means for Mac users and also the EraseInstall app we built at ProWarehouse and how workflows are important for admins and consultants.

I had a great time while we were recording this podcast, even though we had to schedule it “way past my usual bedtime.” I hope you enjoy listening, as well!

desktoppr v0.2 – Notarized

I have just pushed an update to my desktoppr tool which can change the desktop on macOS. No new features but the installer pkg is now notarized.

And because you need to use a newer Xcode and Swift 5 to notarize the tool, it now requires the Swift 5 Runtime support for command line tools when you install it on versions of macOS older than 10.14.4. (On those older versions, you can also continue to use desktoppr-0.1)