Launching Scripts #1: From Terminal

Scripts, no matter which language they are written in, are an incredibly important part of a MacAdmin’s tool kit.

Most posts, books and tutorials focus on how to write scripts. Admittedly, the skill to make the computer and all its software running do what you want is very important. But once you have a working script, you will have to consider how to launch the script at the right time and with the right input.

When I started to think about this, I realized macOS provides many ways to launch or schedule scripts. Some are fairly obvious, others are quite obscure and maybe even frivolous.

Over the next few weeks (i.e. when I find time), I will publish a series of posts describing various ways of launching scripts and workflows on macOS. This post starts with the most obvious: launch a script from Terminal.

The Basics: Launch from Terminal

It might seem trivially obvious that you can launch scripts and workflows from Terminal. Nevertheless, it does require some explanation and context. Launching from Terminal is the most basic form of launching a script. As such, it defines the ‘baseline’ experience of how we expect scripts to run.

Command lookup

When you enter something into a Terminal prompt and hit the return key, the shell will split the text on whitespace. The first element of the text is the command. If that text contains a / character, the shell will interpret the entire element as a path to the executable. The path can be relative or absolute.

If the first element does not contain a / then the shell will first look for functions, aliases and built-in commands, in that order. Then the shell will search for a command executable using the colon-separated paths in the PATH environment variable. It will use the first executable it finds.

If the shell fails to find something to execute, it will show an error.

This is why you need to the ./ to execute scripts in the current working directory. Even though the ./ seems redundant, it tells the shell to look for the executable in the current working directory, instead of using the PATH.

You can find this process described in more detail here. I also have a post on how to configure your PATH variable.

Shebang

Scripts are ‘just’ text files. Two things distinguish them from normal text files.

First, the executable bit is set to tell the system this file contains executable code. You do this with the chmod +x my_script.sh command. Here comes the trick, though. The text in the script file isn’t really executable code that the CPU can understand directly. It needs an ‘interpreter’ which is another program that, wells, interprets the text file and can tell the CPU what to do.

The interpreter for a script file is set in the first line of the script with the #! character combination. The #! is also called ‘hashbang’ or ‘shebang.’ The shebang is followed by an absolute path to the interpreter, e.g. /bin/sh, /bin/bash, or /usr/local/bin/python.

Note: the shebang has to be in the very first line (actually the first characters) of the script, you cannot have comments (or anything else) before it.

Note: you often see shebangs using the env command in the form #!/usr/bin/env bash. These are useful for cross-platform portability, but have trade-offs. I have an article just for that.

Arguments

The shell is only interested in the first part of the command line text you entered. The entire list of parts (or elements, split on whitespace) are passed into the command as its arguments. A shell script sees the first element, the command or path to the command as $0 and the the remaining arguments as $1, $2, and so on.

Other languages will handle this in a similar fashion, arrays or lists of arguments are common.

Environment

Shells also have environment variables. We already talked about the role of the PATH variable in the command lookup. In Terminal, you can list all variables in your current environment with the env command. Some of these environment variables can be extremely useful, like SHELL, USER, and HOME.

When you launch a script from Terminal, a new shell environment is created and all of the environment variables are copied. This is great, beacuse the script can access the environment variables, and might even change them. But the changes in the script’s sub-shell, will not affect the current shell. In other words, a script can change the PATH environment variable in its own context, without changing yours.

In other Unix-based systems, environment variables are a common means of providing data to apps and processes. In the interactive shell on macOS, environment variables are very useful for this purpose. We will see, however, that when you launch processes and scripts through other means, environment variables can be a challenge on macOS.

Note: A shell environment also contains shell options, which are also inherited to sub shells.

Input and Output

Shell scripts and tools also have input and output. When you launch a script from the the Terminal, both output streams (Standard Output and Standard Error) are connected to the Terminal, so the output will be shown in the Terminal window.

When you use a script with pipes, then its Standard Input (stdin) will be connected to the previous tool’s Standard Out (stdout). The last script’s stdout and stderr will be shown in Terminal.

One thing that is special about interactive Terminal input and output, is that it happens while the script or tool is running. That means you can get live updates on the progress.

When we launch scripts in other contexts, their input and output may be buffered. This means that the system waits for the script or tool process to complete before piping its output to the next tool or to the process that called it. This is not something you usually have to worry about, but again it is something you should have in mind when running scripts in non-Terminal contexts.

Conclusion

While glaringly obvious, launching a script from Terminal does have some intricacies. This post sets a ‘baseline’ for how scripts work on macOS. In future installments, we will re-visit some of these topics, and the differences will become relevant when we launch scripts in contexts other than an interactive shell.

In the next post, learn how to create a double-clickable file to launch a script from Finder.

Weekly News Summary for Admins — 2022-03-25

After all the announcements of the last weeks, this week feels quieter. Something must be going in on Cupertino, as we have not gotten betas for macOS 12.4/iOS 15.4 yet. Maybe the developers have to start preparing their WWDC sessions.


(Sponsor: Mosyle)

The Fusion of Apple MDM, Identity, Patching & Security.

Mosyle Fuse logo

Mosyle Fuse is the first and only product to bring a perfect blend of an Enterprise-grade MDM, an innovative solution for macOS Identity Management, automated application installation and patching, and purpose-built multi-layer endpoint security, all specially designed for Apple devices used at work at a price point that’s almost unexplainable.

Click here to learn more!


If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

News and Opinion

Books

macOS and iOS Updates

MacAdmins on Twitter

  • Mr. Macintosh: “The Mac Studio has arrived! I’ll go over a few details others might not have covered” (Thread)

Security and Privacy

🔨Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Watch

To Listen

Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

macOS Monterey 12.3 removes Python 2 – Link collection

Note: I will update this post for the next few weeks with new and updated information. If you find anything that is interesting, ping me on MacAdmins Slack or Twitter as @scriptingosx.

Last Updated: 2022-03-23

What is going on!?

Tools

Updates

  • dockutil: command line tool for managing dock items
  • quickpkg: wrapper for pkgbuild to quickly build simple packages from an installed app, a dmg or zip archive
  • Mist: A Mac command-line tool that automatically downloads macOS Installers/Firmwares
  • DownloadFullInstaller: macOS application written in SwiftUI that downloads installer pkgs for the Install macOS Big Sur application
  • SUS Inspector 2.1: Inspect Apple software update service
  • mkuser: Make user accounts for macOS with many advanced options

Replacing Python

Support

Installomator v8.0

We have published an update for Installomator. It is now at version 8.0 and has over 360 labels!

There were some bugs in the script that could make the script stall the Jamf agent, which prevented the client from checking back in with the Jamf server. This might affect other management systems as well. Please test behavior with the new version and report any issues that might remain.

The changes in detail:

  • removed leading 0 from the version because it has lost all meaning (thanks to @grahampugh for the inspiration)
  • Installomator now detects when an app is already installed, and will display notifications correctly the user based on if the app was updated or installed for the first time.
  • New variables for labels that should be installed using CLI: CLIInstaller and CLIArguments. When the installer app is named differently than the installed app, then the variable installerTool should be used to name the app that should be located in the DMG or zip. See the label adobecreativeclouddesktop to see its use.
  • buildLabel.sh has been improved to build GitHub software labels much easier. In essense if the URL contains github.com, then it will try to find if it’s the latest version or if variable archiveName is needed for finding the software. Also improved messaging throughout the script, as well as handling a situation where a pkg does not include a “Distribution” file, but a “PackageInfo”.
  • MDM script extended with caffeinate so Mac will not go to sleep during the time it takes installomator to run. Especially during setup, this can be useful.
  • Microsoft labels with updateTool variable, is updated to run msupdate --list before running the updateTool directly. Problems have been reported that the update would fail if the --list parameter for the command was not run first. This should help with the Jamf agent stalling during installation.
  • Added bunch of new labels (for a total of 364), and improved others

Most of the work for v8 was done by Søren Theilgaard, but we had many, many contributions from the community! Thanks to everyone!

Save up to 25% pkg file size with this weird Monterey trick!

macOS 12 Monterey brings with it a lot of new features, both for admins and users. You will probably be busy learning and experimenting with them right now, unless you already did that during the beta phase.

But, as usual, there are also many undocumented new changes and features hidden away in the update. I discovered one in the pkgbuild man page. pkgbuild, as the name implies, is used by developers and admins to build installer packages, or pkg files. Even when you are using a tool like munki-pkg or AutoPkg, it is probably using pkgbuild to assemble the installer package.

If you want to learn more about pkgbuild and creating installer packages, read my book: “Packaging for Apple Administrators

Large Payloads and Minimum Versions

“Discovered” is a strong word here. I stumbled over this as I was looking for a different new option for pkgbuild in Monterey. In a converstation with the ever awesome Duncan McCracken, he mentioned that the tool had gained an new option, --large-payload, which allows for individual files in the payload to be larger than 8GB.

This is a significant change to the pkg file format, so pkg installers created with this option will not work on systems older than 12.0. “This option requires the user to pass --min-os-version 12.0 or later to acknowledge this requirement.” (quote from the man page)

This comment led me to look for the description of the --min-os-version option and right between --large-payload and --min-os-version I stumbled over --compression.

There is no indication that these are new options in the man page. There is also no mention of any of these new options in the Developer Release Notes or the AppleSeed for IT release notes.

Payload Compression

The description for the --compression option reads:

--compression compression-mode
Allows control over the compression used for the package. This option does not affect the compression used for plugins or scripts. Not specifying this option will leave the chosen compression algorithm up to the operating system. Two compression-mode arguments are supported:

• legacy forces a 10.5-compatible compression algorithm for the package.

• latest enables pkgbuild to automatically select newer, more efficient compression algorithms based on what is provided to [--min-os-version <version>].

With this new option, in combination with the --min-os-version option, we can influence the compression algorithm used for the payload inside the pkg file. Other than that, we are left in the dark. What kind of compression algorithms? And which minimum macOS versions use which compression algorithms?

The man page is silent on this, so we need to experiment!

Lots of pkgs

After some less organized experimentation, I put together this one-liner:

for x in 10.{5..15} 11 12; do caffeinate time pkgbuild --component /Applications/Numbers.app --min-os-version $x --compression latest Numbers-min$x.pkg; done 

This is very, well, compressed, so I will explain it in steps:

Brace expansion

for x in 10.{5..15} 11 12; do

If I used for x in 10 11 12; do the shell would loop through the list using the values 10, 11, and 12. However, I also need eleven versions starting with 10. so I use the ‘brace expansion:’ 10.{5..15} will expand to 10.5, 10.6, 10.7, … until 10.15.

Brace expansion is rarely used but a very useful shell feature.

This for loop will loop through 10.5 through 10.15, and then 11 and 12, as well.

caffeinate

I did not want the MacBook I was testing on to fall asleep during the test. The caffeinate will prevent a Mac from sleeping. Most people use this command as a standalone command where it will prevent the Mac from sleeping indefinitely. Maybe you have used caffeinate -t 3600 to prevent sleep for a certain time.

But you can also use caffeinate together with a second command, and then caffeinate will prevent sleep for as long as the second command is running. For example:

caffeinate system_profiler

will prevent the Mac from falling asleep while system_profiler does its thing, which always seems like it takes ages.

The default mode of caffeinate will only prevent system sleep. The display may still dim, sleep, or even lock. If you want to prevent that as well, use caffeinate -di.

time

I was interested in the duration each pkgbuild run would take. The time command will give you that information. For example, when you run time system_profiler the system_profiler command will run, showing all its output, but the time command will add this at the end:

system_profiler  11.64s user 7.35s system 53% cpu 35.493 total

The first number (sometimes called the ‘real’ time) is the time that elapsed on the clock from the start to end of the process. The ‘user’ is also interesting as it gives the cpu time the process itself was actively running (and not waiting for other processes). Confusingly, the user time may be larger than the real time. This means the process was running on multiple cpus at once.

pkgbuild

And then we have the actual pkgbuild command using the --compression and the --min-os-version to build a pkg installer from the Numbers application on my system. I chose Numbers.app because it is fairly large (589MB) so the compression algorithm has something to do.

Results

I ran this on a MacBook Air M1 with 8GB of RAM. While pkgbuild was doing its work, I kept doing other tasks on the Mac. You can see that the run times vary by a few seconds. The system was experiencing memory pressure as the commands ran. This test wasn’t really very accurate, but the quality of the output, as you will see, is good enough to yield conclusions.

After running the above command I had 13 pkg files. and I created a chart with the output from the time command and the pkg file sizes.

When you chart the resulting filesize against the --min-os-version you see a distinct change in 10.10:

The filesizes will vary by a few bytes, but I presume that stems from different timestamps and a different minimum OS version set in the metadata of each package.

The same chart with the time required to create the pkg file:

When you use a --min-os-version value of 10.10 or higher, the file size drops by about 24% but the creation time (the ‘real’ value) nearly doubles. The ‘user’ time value increases nearly ten-fold and you may wonder how the user time can be so much higher than the ‘real’ time elapsed. The explanation is that the legacy compression algorithm uses only a single core, while the 10.10+ compression algorithm uses all available cores.

From 10.10 (Yosemite) on, the numbers stay fairly constant, all the way to 12.0, so my assumption is the compression algorithm stays the same.

Compression Algorithm

So which compression algorithms are actually used?

To figure this out, I first wanted to know if the file type of the pkg file file itself changed:

> file *.pkg    
Numbers-min10.5.pkg:   xar archive compressed TOC: 709, SHA-1 checksum, contains zlib compressed data
...
Numbers-min11.pkg:     xar archive compressed TOC: 706, SHA-1 checksum, contains zlib compressed data
Numbers-min12.pkg:     xar archive compressed TOC: 708, SHA-1 checksum, contains zlib compressed data
Numbers-minLegacy.pkg: xar archive compressed TOC: 708, SHA-1 checksum, contains zlib compressed data

As you can see the format of the wrapping archive of the pkg installer remains the same. This is probably necessary so that old versions of macOS can read the metadata inside the pkg.

But inside the pkg, is another compressed archive. You can see this when you run

> pkgutil --expand Numbers-min10.5.pkg 10.5Pkg
> ls 10.5Pkg
Bom         PackageInfo Payload
> file 10.5Pkg/Payload    
10.5Pkg/Payload: gzip compressed data, from Unix, original size modulo 2^32 594074112

Up to and including 10.9 the Payload is a gzip archive. From 10.10 upward the file command returns only data:

> file 10.10Pkg/Payload 
10.10Pkg/Payload: data

On an educated guess, I tried to list the contents of the 10.10 payload with the aa command which reads and writes the poorly documented ‘Apple Archive’ format:

> aa list -i 10.10Pkg/Payload    
.
./Numbers.app
./Numbers.app/Contents
./Numbers.app/Contents/_CodeSignature
./Numbers.app/Contents/_CodeSignature/CodeResources
...

(Note: the aa command is available on macOS Big Sur and higher.)

Expanding

When I saw how much more compute intensive the compression was, I was a bit concerned the decompression might be compute intensive on old hardware. To see if that would be a problem I used my 2012 Mac mini (the server model) running Catalina 10.15.7.

Catalina does not have the aa command line tool, (it was added in Big Sur) but pkgutil has an undocumented --expand-full option which will expand the pkg and the payload. So, I used that as a comparison:

> time pkgutil --expand-full Numbers-min10.10.pkg 10.10ExpandFull/

The M1 MacBook Air took 8.97 seconds for this operation, the 2012 Mac mini took 15.46. While that is slower, it is not a dramatic difference.

For comparison, expanding a legacy compressed file took 1.76 seconds (M1 MacBook Air) and 3.98 seconds (Mac mini 2021). The decompression is about four times slower using the new compression.

But what about productbuild?

For admins, component pkgs built with pkgbuild, are sufficient for most tasks, but sometimes developers require distribution packages. Developers generally prefer to build distribution packages.

For details on the differences of the package types and when you need which type, watch my MacDevOps YVR 2021 presentation: The Encyclopedia of Packages

The productbuild command line tool builds distribution packages. Since distribution pkgs can be far more complex than component packages, this tool has many more options. (Read my Packaging book for details.) But it has a similar mode to quickly build a distribution pkg from an app bundle:

> caffeinate time productbuild --component /Applications/Numbers.app NumbersDistDefault.pkg
productbuild: Adding component at /Applications/Numbers.app
productbuild: Inferred install-location of /Applications
productbuild: Wrote product to NumbersDistDefault.pkg
productbuild: Supported OS versions: [Min: 11.0, Before: None]
       28.85 real        22.43 user         3.42 sys

From the timing, we can guess that is creating a legacy compressed payload.

When we dig into the man page for productbuild on Monterey, we find a --component-compression option, which sounds promising. It has three options: legacy, auto, and default. The man page states that default behaves the same as legacy but that “may change in future releases of OS X.”

> caffeinate time productbuild --component /Applications/Numbers.app --component-compression auto NumbersDist.pkg
productbuild: Adding component at /Applications/Numbers.app
productbuild: Inferred install-location of /Applications
productbuild: Wrote product to NumbersDist.pkg
productbuild: Supported OS versions: [Min: 11.0, Before: None]
       44.87 real       207.40 user         8.70 sys

In this case the time suggests this uses the Apple Archive compression. But we didn’t have to provide a minimum OS version. The trick here is in the last output line of productbuild. There we see that productbuild automatically determined a minimum OS version from the application bundle. It reads the LSMinimumSystemVersion key from the app bundle’s Info.plist for this.

This is even more flexible than generically setting a min OS version of 10.10.

However, this will only work with the --component option of productbuild. Usually you have to build the components individually with pkgbuild and combine them with productbuild and in that workflow you will have to provide the minimum OS version for each component. Or determine it dynamically from the source, which is even more flexible.

Conclusion

We have learned that when you use the --compression latest with a --min-os-version of 10.10 or higher the pkg creation uses the Apple Archive compression for the payload, leading to smaller pkg file sizes. I did a few more tests with some other apps and the file compression improvements were between 20% and 25%.

When I set out to explore this, I did not expect a new compression algorithm to be present except maybe in the lastest macOS releases. This would have meant admins (who usually need to support at least two or three versions back) would have had to wait a few years before we could use a new compression algorithm, unless they were pushing the bleeding edge. However, a minimum macOS version of 10.10 means that a large majority of Apple Admins should be able to use this.

Most of the software deployed will have higher system requirements than OS X Yosemite. The minimum OS version for the package should be determined dynamically from the contents, pkgbuild and productbuild will then use the appropriate compression. The --component-compression auto option for productbuild has this dynamic behavior, but it should not be too complicated to add similar logic to your package creation workflows.

You might ask if a ~20-25% reduction in file size is really worth the extra effort of updating your packaging workflows. Since many management systems are now hosted in the cloud, every bit you can save in up- and download might have a noticeable price, if not in bandwidth costs, then in time saved for user downloading the pkg. The savings will be multiplied by the number of clients, which adds up quickly with large fleets.

I think that most effective application of this knowledge would be to have an option in your packaging workflow to use this better compression. For that, the packaging workflow will have to run on Monterey. AutoPkg is the best example of such a workflow, but there are other tools, like Packages.app or munki-pkg which could profit from this, as well.

Working at Jamf…

On a personal note: today is my first day working at Jamf as a Consulting Engineer.

I have been working with Jamf for more than a decade. Back then it was called Casper. Over the years, I have experienced the product and the company as a partner, as a customer and admin, and as a consultant. Working from the inside seemed like a logical next step. I will be joining a great team where I already know many people and I am looking forward to getting to know everyone else!

But don’t be afraid, this weblog and the weekly newsletter will continue as usual!

An AirTag Adventure, Part 2—Receiving an AirTag

Anthony Reimer and I had a lot of fun sending an AirTag across the Atlantic. Now we get to the experience of being on the receiving side.

“AirTag Found Moving With You”

The “Find My” network warns you when an unknown AirTag is moving with you. This is to prevent tracking people without their approval. Since Anthony had registered the AirTag on his account to track its travel, this was a perfect opportunity to test this situation.

I just dropped the AirTag in the front pocket of my backpack and went about my business. This is the backpack I use to transport odds and ends, especially groceries, so it nearly always goes where I go. At first the backpack remained immobile in my house for the afternoon and then overnight. The next morning, we took a trip to the Leiden Farmers’ Market when I got the warning that an AirTag was moving with me.

It is interesting (but makes sense) that the warning didn’t come until I actually started moving with the AirTag. The tag is not really tracking you when it is just sitting around. But once I was on the go (together with the “strange” AirTag) I was warned fairly quickly: after about 2.5 hours. My wife also got the same warning on her iPhone, which should not be surprising, since we were walking together.

The AirTag is supposed to eventually make a sound when it is separated from its owner, but it never got to that phase in our testing, or I did not hear it.

The iPhone showed me a map where my iPhone had detected the “strange” AirTag and offered the option to play a sound on the AirTag to help locate it. Presumably, when someone tracks you without your knowledge, the AirTag would be hidden somewhere nearby and the sound will help you find it.

You can tell your phone to ignore this particular AirTag, presumably after you have checked with your companions who are travelling with you or because you are carrying a borrowed, tagged item.

The app also shows the serial number of the AirTag and the last four digits of the phone number it is registered to. These digits should allow you to identify the owner, when you know them, but maintain their anonymity when you don’t.

You can also get instructions to disable the AirTag. This will show instructions to open it and remove the battery with a simple but effective animation. (AirTags open real easy, given how small they are and how solidly sealed they seem when closed. This is really impressive engineering.)

I can see a possible downside here. The disabling process requires you to actually find the AirTag. If someone manages to hide the AirTag in way that you cannot find or access it physically, you cannot disable it. This might be harder than I imagine, because shielding the AirTag in a way that muffles the sound sufficiently might also shield the Bluetooth transmissions, which prevent the tracking in the first place. More experimentation will be needed here.

Lost Mode

Now that the covert tracking had been tested, Anthony set the AirTag to lost mode on his account. It took a few minutes for that change to propagate through the network. With lost mode enabled, I could call up his contact information (the owner can choose whether to show an email or the phone number) from the AirTag on my phone, just by tapping my phone to the AirTag.

Anthony also tried to play a sound on the AirTag, which was more than 7000km away from him. This did not work. It seems that playing the sound requires a local bluetooth connection to the AirTag. Since you would likely not be able to hear the sound when you are out of bluetooth range, and could use this to ‘terrorize’ someone (intentionally or not) in the middle of the night, I think this a reasonable limitation.

Transferring the AirTag

With all our testing done it was time for Anthony to remove the AirTag from his account, so that I could add it to my account. The interface for that in the Find My application is very straightforward.

He did, however, get an error that the iPhone could not “find” the AirTag. We presume his iPhone tried to connect to the AirTag over local bluetooth to let it “know” it was removed.

After Anthony had removed the device from his account, I tried to set it up on mine. This did not immediately work. Even after waiting for a few hours, my phone would not recognize the AirTag as new.

I then followed the instructions in this support article to reset the AirTag. It’s a bit tedious as you have to remove and replace the battery five times in a row. I figured out you don’t have to actually close the lid five times, just taking out the battery and putting it back in its place is sufficient. (there are magnets in the AirTag that seem to hold the batttery in place) After the reset process, the AirTag appeared immediately for setup on my phone and I could add it to my iCloud account.

Conclusion

Overall, the user experience for both the “Moving with you” and “Lost Mode” workflows are well thought through and kept clear and simple. Apple has good support articles for reference.

Many thanks to the comittee of MacDeployment and their sponsors that provided AirTags to all the speakers. And thanks to Anthony, who was game when I suggested that sending and tracking an AirTag across the Atlantic would be the “most fun” way to get them to me. Hope you found our experiments interesting, as well!

Right now, the AirTag has returned to my backpack. This seems reasonable since it stores my wallet and keys when I leave the house. I also want to test attaching an AirTag to my bike. I believe that bike thieves will quickly catch on to AirTags, so I don’t have high hopes for it to be useful as theft prevention. But an AirTag on the bike should be very useful to find my bike again in one of the typical Dutch bike parking areas among thousands of other bikes.

Notarize a Command Line Tool with notarytool

When Apple introduced notarization with Catalina, I published a post describing how to notarize a command line tool. At WWDC this year, Apple introduced updates to this process with Xcode 13 (currently in beta). Most importantly, there is a new command line tool called notarytool.

While the previous, altool-based, workflow still works in Xcode 13, there are many advantages to the new notarytool which makes its use much simpler.

Apple has documented this tool in a WWDC21 session and some developer articles, in addition we got some great information through the twitter account of one of the engineers, and Howard Oakley has already written a post as well:

What you need

  • Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates, nor access to the Xcode beta)
  • Xcode 13 (currently available as beta from the Apple Developer portal)
  • Developer ID Certificates
  • Application Specific Password for your Developer account
  • A command line tool project in Xcode

When you are building tools for macOS, you should have most of these already. We already covered these in the previous post, but to keep things in one place, I will cover them again, here.

Apple Developer Account

You need either the paid membership in the Apple Developer Program or be invited to an Apple Developer Enterprise Program team with access to the proper certificates.

You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.

Xcode 13 (beta)

Until the full version of Xcode 13 is released, you can get Xcode 13 beta from the beta downloads page on the Apple Developer Portal.

Once it is released (usually when iOS is released) you will be able to download it from the Mac App Store, as well.

Xcode 13 requires macOS Big Sur 11.3 or higher. According to this tweet from Rosyna Keller, notarytool can be extracted and run on macOS Catalina 10.15.7 and higher.

You can run the notarytool binary through xcrun:

% xcrun notarytool --help

If you need to extract the binary you can find where is stored on disk with:

% xcrun --find notarytool
/Applications/Xcode-beta.app/Contents/Developer/usr/bin/notarytool

Developer ID Certificates

There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.

To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.

We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.

If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.

Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:

% security find-identity -p basic -v

This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.

You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"

The ten character code at the end of the name is your Developer Team ID. Make a note of it, we will need it later. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.

Application Specific Password for your Developer Account

Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.

Note: If you followed the previous post’s instructions to store an application specific password for altool in the Keychain, you can extract that and re-use it for notarytool or create a new app-specific password.

Create a new application specific password in Apple ID portal for your developer account. Give it a name including notarytool so you know what you are using this for.

You will only be shown the password when you create it.

You can use notarytool to store the credentials in a keychain item, in a format that notarytool can read later.

% xcrun notarytool store-credentials --apple-id "name@example.com" --team-id "ABCD123456"

This process stores your credentials securely in the Keychain. You reference these credentials later using a profile name.

Profile name:
notary-example.com
Password for name@example.com: 
Validating your credentials...
Success. Credentials validated.
Credentials saved to Keychain.
To use them, specify `--keychain-profile "notary-example.com"`

The --store-credentials option will prompt for a profile name. You will need this name to retrieve the information later. Then it interactively prompts for the password associated with the given Apple Developer ID. Enter the application specific password here.

The credentials will be stored in the Keychain in an item named com.apple.gke.notary.tool. But you don’t really have to worry about that since notarytool will retrieve the credentials when you add the --keychain-profile "notary-example.com" option. (You can abbreviate the --keychain-profile with -p.)

If you are using iCloud Keychain, the credentials will be stored there, so they will be available to all other Macs you are using iCloud Keychain with. If you prefer, you can store the credentials in a specific (non-iCloud) keychain file with the --keychain option.

The Team ID is usually the 10-digit code which is also the certificates. However, in some cases the Team ID is different. You can can look-up Team IDs in the “Membership” area of the developer portal or with this altool command:

% xcrun altool --list-providers -u "name@example.com" -p "@keychain:<ITEM_NAME>"

(Thanks to ‘mhp’ for sharing this.)

You can also use an App Store Connect API key as an authentication option with notarytool. You can read notarytool‘s man page for details.

A Command Line Tool Project

You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.

My sample project for this article will be named “hello.”

Preparing the Xcode Project

The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.

The preparation in Xcode 13 diverges significantly from the steps required in the previous post. If you have created the project in earlier versions of Xcode, more configuration may be necessary.

Choosing the proper signing certificates

Before you can notarize the command line tool, it needs to be signed with the correct certificates.

  1. in Xcode, select the blue project icon in the left sidebar
  2. select the black “terminal” icon with your project’s name under the “Targets” list entry
  3. make sure the ‘Signing & Certificates’ tab is selected
  4. under ‘Signing’ disable ‘Automatically manage signing’
  5. choose your Team
  6. enter a bundle identifier for the binary
  7. choose ‘Developer ID Application‘ as the Signing Certificate

Hardened Runtime

Having the “Hardened Runtime” enabled is a requirement for notarization. When you create a new project in Xcode 13, the hardened runtime will be enabled by default. When you see the “Hardened Runtime” section under the “Signing” section, it is enabled.

When you are working with a older project, and do not see the “Hardened Runtime” section, you can enable the hardened runtime by clicking on the “+Capability” button above the “Signing” section and selecting “Hardened Runtime”.

Archive and export the binary

Choose “Archive” from the “Product” menu to build and create an archive. It will appear in the “Organizer” window. When that window does not open automatically, you can access it from the “Window” menu.

To export the binary product, select the latest archive and click on the “Distribute Content” button on the right. Choose “Built Products” as the method of distribution. Click “Next.” Choose a location to save the build products to.

This will create a directory with the project name and a timestamp in the chosen location. When you look inside this directory, you will see a “Products” directory and within it the binary in a /usr/local/bin/ directory hierarchy.

/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building the archive.

Build the installer package

Command Line Tools can be signed, but not directly notarized. You can however notarize a pkg file containing the Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.

We can use the Products directory as our payload to build the installer package:

% pkgbuild --root "hello 2021-mm-dd hh-mm-ss/Products" \
           --identifier "com.example.hello" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Name (ABCD123456)" \
           hello-1.0.pkg

I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.

This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.

If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”

Notarizing the Installer Package

Now we get to the new, most interesting part. We will notarize the newly-created installer package with notarytool:

% xcrun notarytool submit hello-1.0.pkg \
                   --keychain-profile "notary-scriptingosx" \
                   --wait

This is amazingly less effort than what we needed to do previously with the altool command. We give the filename of the archive we want to submit, the keychain profile with our credentials, and the --wait option.

notarytool will upload the file, give us a submission id, and then wait for the returned status from the Notary service. You can follow the output for the details.

You will also notice that notarytool uploads the pkg much faster than the previous altool workflow.

You can also drop the --wait option. Then the tool will submit the file and exit without waiting for a response. You can then use the info or log verbs with the submission id to get the status later. The Notary service does not seem to send emails anymore when the notarization check is complete.

There is also a --webhook option mentioned in the WWDC session which will make the Notary service call back to a webhook when the notarization is done. I have not seen any documentation on the details of this, though.

Finishing touch: stapler

Before you distribute the pkg, you can and should ‘staple’ the notarization before distributing it. This extra step will download the notarization information from Apple’s servers and attach it to the pkg. This is not mandatory, but will save the Gatekeeper service on the client an extra step when it verifies the pkg.

To do this, use the eponymous stapler tool:

% xcrun stapler staple hello-1.0.pkg

You can then verify that everything works with spctl:

% spctl --assess -vv --type install hello-1.0.pkg

Automation with Xcode

These steps are much simplified compared to the previous workflow. If you only build for distribution occasionally it would not be a big burden to do these steps manually.

Nevertheless, automating these steps saves effort and removes much pontential for errors.

When I wrote the previous post, I had not been able to figure out how all the pieces could work together to automate with a Xcode ‘Run Script’ as part of the normal “Archive” process. With the new tool and some inspiration from this developer article I have gotten this to work now.

In the project’s build settings, search for “Marketing Version” and set it to the version you want to use. Remember to update this entry for future updates as well. (You can use agvtool for this, but that is a topic for a different post.)

In Xcode, choose “Edit Scheme…” from the “Scheme” submenu in the “Project” menu. In the pane that opens, make sure the commnad line tool binary is selected at the top. Then expand the “Archive” section in the list on the left and select “Post-actions” in the expanded area. Use the “+” button at the bottom of the area to add a “New Run Script Action.”

Select the binary (again) in the popup next to “Provide build settings from”. Then paste the following in the code field:

With this post-action script in place, every “Archive” action will then also create a pkg in the project folder, submit it for notarization and staple the pkg. Since Xcode doesn’t show the output of post-action scripts, the script logs its output to a notary.log file, also in the project folder. Check that for success or failures. The notarization step takes a while after the “Archive” is complete, so you may have to wait a bit.

If you don’t want to run this workflow on every Archive, you can create a new scheme with this post-action script, then you can choose the scheme, before you do the “Archive” action.

Conclusion

The new notarytool included with Xcode 13 (beta) is a huge step up from the previous altool based workflows. It is much simpler and faster. You should start testing the tool now and move your workflows when possible.