Installomator v8.0

We have published an update for Installomator. It is now at version 8.0 and has over 360 labels!

There were some bugs in the script that could make the script stall the Jamf agent, which prevented the client from checking back in with the Jamf server. This might affect other management systems as well. Please test behavior with the new version and report any issues that might remain.

The changes in detail:

  • removed leading 0 from the version because it has lost all meaning (thanks to @grahampugh for the inspiration)
  • Installomator now detects when an app is already installed, and will display notifications correctly the user based on if the app was updated or installed for the first time.
  • New variables for labels that should be installed using CLI: CLIInstaller and CLIArguments. When the installer app is named differently than the installed app, then the variable installerTool should be used to name the app that should be located in the DMG or zip. See the label adobecreativeclouddesktop to see its use.
  • buildLabel.sh has been improved to build GitHub software labels much easier. In essense if the URL contains github.com, then it will try to find if it’s the latest version or if variable archiveName is needed for finding the software. Also improved messaging throughout the script, as well as handling a situation where a pkg does not include a “Distribution” file, but a “PackageInfo”.
  • MDM script extended with caffeinate so Mac will not go to sleep during the time it takes installomator to run. Especially during setup, this can be useful.
  • Microsoft labels with updateTool variable, is updated to run msupdate --list before running the updateTool directly. Problems have been reported that the update would fail if the --list parameter for the command was not run first. This should help with the Jamf agent stalling during installation.
  • Added bunch of new labels (for a total of 364), and improved others

Most of the work for v8 was done by Søren Theilgaard, but we had many, many contributions from the community! Thanks to everyone!

Save up to 25% pkg file size with this weird Monterey trick!

macOS 12 Monterey brings with it a lot of new features, both for admins and users. You will probably be busy learning and experimenting with them right now, unless you already did that during the beta phase.

But, as usual, there are also many undocumented new changes and features hidden away in the update. I discovered one in the pkgbuild man page. pkgbuild, as the name implies, is used by developers and admins to build installer packages, or pkg files. Even when you are using a tool like munki-pkg or AutoPkg, it is probably using pkgbuild to assemble the installer package.

If you want to learn more about pkgbuild and creating installer packages, read my book: “Packaging for Apple Administrators

Large Payloads and Minimum Versions

“Discovered” is a strong word here. I stumbled over this as I was looking for a different new option for pkgbuild in Monterey. In a converstation with the ever awesome Duncan McCracken, he mentioned that the tool had gained an new option, --large-payload, which allows for individual files in the payload to be larger than 8GB.

This is a significant change to the pkg file format, so pkg installers created with this option will not work on systems older than 12.0. “This option requires the user to pass --min-os-version 12.0 or later to acknowledge this requirement.” (quote from the man page)

This comment led me to look for the description of the --min-os-version option and right between --large-payload and --min-os-version I stumbled over --compression.

There is no indication that these are new options in the man page. There is also no mention of any of these new options in the Developer Release Notes or the AppleSeed for IT release notes.

Payload Compression

The description for the --compression option reads:

--compression compression-mode
Allows control over the compression used for the package. This option does not affect the compression used for plugins or scripts. Not specifying this option will leave the chosen compression algorithm up to the operating system. Two compression-mode arguments are supported:

• legacy forces a 10.5-compatible compression algorithm for the package.

• latest enables pkgbuild to automatically select newer, more efficient compression algorithms based on what is provided to [--min-os-version <version>].

With this new option, in combination with the --min-os-version option, we can influence the compression algorithm used for the payload inside the pkg file. Other than that, we are left in the dark. What kind of compression algorithms? And which minimum macOS versions use which compression algorithms?

The man page is silent on this, so we need to experiment!

Lots of pkgs

After some less organized experimentation, I put together this one-liner:

for x in 10.{5..15} 11 12; do caffeinate time pkgbuild --component /Applications/Numbers.app --min-os-version $x --compression latest Numbers-min$x.pkg; done 

This is very, well, compressed, so I will explain it in steps:

Brace expansion

for x in 10.{5..15} 11 12; do

If I used for x in 10 11 12; do the shell would loop through the list using the values 10, 11, and 12. However, I also need eleven versions starting with 10. so I use the ‘brace expansion:’ 10.{5..15} will expand to 10.5, 10.6, 10.7, … until 10.15.

Brace expansion is rarely used but a very useful shell feature.

This for loop will loop through 10.5 through 10.15, and then 11 and 12, as well.

caffeinate

I did not want the MacBook I was testing on to fall asleep during the test. The caffeinate will prevent a Mac from sleeping. Most people use this command as a standalone command where it will prevent the Mac from sleeping indefinitely. Maybe you have used caffeinate -t 3600 to prevent sleep for a certain time.

But you can also use caffeinate together with a second command, and then caffeinate will prevent sleep for as long as the second command is running. For example:

caffeinate system_profiler

will prevent the Mac from falling asleep while system_profiler does its thing, which always seems like it takes ages.

The default mode of caffeinate will only prevent system sleep. The display may still dim, sleep, or even lock. If you want to prevent that as well, use caffeinate -di.

time

I was interested in the duration each pkgbuild run would take. The time command will give you that information. For example, when you run time system_profiler the system_profiler command will run, showing all its output, but the time command will add this at the end:

system_profiler  11.64s user 7.35s system 53% cpu 35.493 total

The first number (sometimes called the ‘real’ time) is the time that elapsed on the clock from the start to end of the process. The ‘user’ is also interesting as it gives the cpu time the process itself was actively running (and not waiting for other processes). Confusingly, the user time may be larger than the real time. This means the process was running on multiple cpus at once.

pkgbuild

And then we have the actual pkgbuild command using the --compression and the --min-os-version to build a pkg installer from the Numbers application on my system. I chose Numbers.app because it is fairly large (589MB) so the compression algorithm has something to do.

Results

I ran this on a MacBook Air M1 with 8GB of RAM. While pkgbuild was doing its work, I kept doing other tasks on the Mac. You can see that the run times vary by a few seconds. The system was experiencing memory pressure as the commands ran. This test wasn’t really very accurate, but the quality of the output, as you will see, is good enough to yield conclusions.

After running the above command I had 13 pkg files. and I created a chart with the output from the time command and the pkg file sizes.

When you chart the resulting filesize against the --min-os-version you see a distinct change in 10.10:

The filesizes will vary by a few bytes, but I presume that stems from different timestamps and a different minimum OS version set in the metadata of each package.

The same chart with the time required to create the pkg file:

When you use a --min-os-version value of 10.10 or higher, the file size drops by about 24% but the creation time (the ‘real’ value) nearly doubles. The ‘user’ time value increases nearly ten-fold and you may wonder how the user time can be so much higher than the ‘real’ time elapsed. The explanation is that the legacy compression algorithm uses only a single core, while the 10.10+ compression algorithm uses all available cores.

From 10.10 (Yosemite) on, the numbers stay fairly constant, all the way to 12.0, so my assumption is the compression algorithm stays the same.

Compression Algorithm

So which compression algorithms are actually used?

To figure this out, I first wanted to know if the file type of the pkg file file itself changed:

> file *.pkg    
Numbers-min10.5.pkg:   xar archive compressed TOC: 709, SHA-1 checksum, contains zlib compressed data
...
Numbers-min11.pkg:     xar archive compressed TOC: 706, SHA-1 checksum, contains zlib compressed data
Numbers-min12.pkg:     xar archive compressed TOC: 708, SHA-1 checksum, contains zlib compressed data
Numbers-minLegacy.pkg: xar archive compressed TOC: 708, SHA-1 checksum, contains zlib compressed data

As you can see the format of the wrapping archive of the pkg installer remains the same. This is probably necessary so that old versions of macOS can read the metadata inside the pkg.

But inside the pkg, is another compressed archive. You can see this when you run

> pkgutil --expand Numbers-min10.5.pkg 10.5Pkg
> ls 10.5Pkg
Bom         PackageInfo Payload
> file 10.5Pkg/Payload    
10.5Pkg/Payload: gzip compressed data, from Unix, original size modulo 2^32 594074112

Up to and including 10.9 the Payload is a gzip archive. From 10.10 upward the file command returns only data:

> file 10.10Pkg/Payload 
10.10Pkg/Payload: data

On an educated guess, I tried to list the contents of the 10.10 payload with the aa command which reads and writes the poorly documented ‘Apple Archive’ format:

> aa list -i 10.10Pkg/Payload    
.
./Numbers.app
./Numbers.app/Contents
./Numbers.app/Contents/_CodeSignature
./Numbers.app/Contents/_CodeSignature/CodeResources
...

(Note: the aa command is available on macOS Big Sur and higher.)

Expanding

When I saw how much more compute intensive the compression was, I was a bit concerned the decompression might be compute intensive on old hardware. To see if that would be a problem I used my 2012 Mac mini (the server model) running Catalina 10.15.7.

Catalina does not have the aa command line tool, (it was added in Big Sur) but pkgutil has an undocumented --expand-full option which will expand the pkg and the payload. So, I used that as a comparison:

> time pkgutil --expand-full Numbers-min10.10.pkg 10.10ExpandFull/

The M1 MacBook Air took 8.97 seconds for this operation, the 2012 Mac mini took 15.46. While that is slower, it is not a dramatic difference.

For comparison, expanding a legacy compressed file took 1.76 seconds (M1 MacBook Air) and 3.98 seconds (Mac mini 2021). The decompression is about four times slower using the new compression.

But what about productbuild?

For admins, component pkgs built with pkgbuild, are sufficient for most tasks, but sometimes developers require distribution packages. Developers generally prefer to build distribution packages.

For details on the differences of the package types and when you need which type, watch my MacDevOps YVR 2021 presentation: The Encyclopedia of Packages

The productbuild command line tool builds distribution packages. Since distribution pkgs can be far more complex than component packages, this tool has many more options. (Read my Packaging book for details.) But it has a similar mode to quickly build a distribution pkg from an app bundle:

> caffeinate time productbuild --component /Applications/Numbers.app NumbersDistDefault.pkg
productbuild: Adding component at /Applications/Numbers.app
productbuild: Inferred install-location of /Applications
productbuild: Wrote product to NumbersDistDefault.pkg
productbuild: Supported OS versions: [Min: 11.0, Before: None]
       28.85 real        22.43 user         3.42 sys

From the timing, we can guess that is creating a legacy compressed payload.

When we dig into the man page for productbuild on Monterey, we find a --component-compression option, which sounds promising. It has three options: legacy, auto, and default. The man page states that default behaves the same as legacy but that “may change in future releases of OS X.”

> caffeinate time productbuild --component /Applications/Numbers.app --component-compression auto NumbersDist.pkg
productbuild: Adding component at /Applications/Numbers.app
productbuild: Inferred install-location of /Applications
productbuild: Wrote product to NumbersDist.pkg
productbuild: Supported OS versions: [Min: 11.0, Before: None]
       44.87 real       207.40 user         8.70 sys

In this case the time suggests this uses the Apple Archive compression. But we didn’t have to provide a minimum OS version. The trick here is in the last output line of productbuild. There we see that productbuild automatically determined a minimum OS version from the application bundle. It reads the LSMinimumSystemVersion key from the app bundle’s Info.plist for this.

This is even more flexible than generically setting a min OS version of 10.10.

However, this will only work with the --component option of productbuild. Usually you have to build the components individually with pkgbuild and combine them with productbuild and in that workflow you will have to provide the minimum OS version for each component. Or determine it dynamically from the source, which is even more flexible.

Conclusion

We have learned that when you use the --compression latest with a --min-os-version of 10.10 or higher the pkg creation uses the Apple Archive compression for the payload, leading to smaller pkg file sizes. I did a few more tests with some other apps and the file compression improvements were between 20% and 25%.

When I set out to explore this, I did not expect a new compression algorithm to be present except maybe in the lastest macOS releases. This would have meant admins (who usually need to support at least two or three versions back) would have had to wait a few years before we could use a new compression algorithm, unless they were pushing the bleeding edge. However, a minimum macOS version of 10.10 means that a large majority of Apple Admins should be able to use this.

Most of the software deployed will have higher system requirements than OS X Yosemite. The minimum OS version for the package should be determined dynamically from the contents, pkgbuild and productbuild will then use the appropriate compression. The --component-compression auto option for productbuild has this dynamic behavior, but it should not be too complicated to add similar logic to your package creation workflows.

You might ask if a ~20-25% reduction in file size is really worth the extra effort of updating your packaging workflows. Since many management systems are now hosted in the cloud, every bit you can save in up- and download might have a noticeable price, if not in bandwidth costs, then in time saved for user downloading the pkg. The savings will be multiplied by the number of clients, which adds up quickly with large fleets.

I think that most effective application of this knowledge would be to have an option in your packaging workflow to use this better compression. For that, the packaging workflow will have to run on Monterey. AutoPkg is the best example of such a workflow, but there are other tools, like Packages.app or munki-pkg which could profit from this, as well.

Working at Jamf…

On a personal note: today is my first day working at Jamf as a Consulting Engineer.

I have been working with Jamf for more than a decade. Back then it was called Casper. Over the years, I have experienced the product and the company as a partner, as a customer and admin, and as a consultant. Working from the inside seemed like a logical next step. I will be joining a great team where I already know many people and I am looking forward to getting to know everyone else!

But don’t be afraid, this weblog and the weekly newsletter will continue as usual!

An AirTag Adventure, Part 2—Receiving an AirTag

Anthony Reimer and I had a lot of fun sending an AirTag across the Atlantic. Now we get to the experience of being on the receiving side.

“AirTag Found Moving With You”

The “Find My” network warns you when an unknown AirTag is moving with you. This is to prevent tracking people without their approval. Since Anthony had registered the AirTag on his account to track its travel, this was a perfect opportunity to test this situation.

I just dropped the AirTag in the front pocket of my backpack and went about my business. This is the backpack I use to transport odds and ends, especially groceries, so it nearly always goes where I go. At first the backpack remained immobile in my house for the afternoon and then overnight. The next morning, we took a trip to the Leiden Farmers’ Market when I got the warning that an AirTag was moving with me.

It is interesting (but makes sense) that the warning didn’t come until I actually started moving with the AirTag. The tag is not really tracking you when it is just sitting around. But once I was on the go (together with the “strange” AirTag) I was warned fairly quickly: after about 2.5 hours. My wife also got the same warning on her iPhone, which should not be surprising, since we were walking together.

The AirTag is supposed to eventually make a sound when it is separated from its owner, but it never got to that phase in our testing, or I did not hear it.

The iPhone showed me a map where my iPhone had detected the “strange” AirTag and offered the option to play a sound on the AirTag to help locate it. Presumably, when someone tracks you without your knowledge, the AirTag would be hidden somewhere nearby and the sound will help you find it.

You can tell your phone to ignore this particular AirTag, presumably after you have checked with your companions who are travelling with you or because you are carrying a borrowed, tagged item.

The app also shows the serial number of the AirTag and the last four digits of the phone number it is registered to. These digits should allow you to identify the owner, when you know them, but maintain their anonymity when you don’t.

You can also get instructions to disable the AirTag. This will show instructions to open it and remove the battery with a simple but effective animation. (AirTags open real easy, given how small they are and how solidly sealed they seem when closed. This is really impressive engineering.)

I can see a possible downside here. The disabling process requires you to actually find the AirTag. If someone manages to hide the AirTag in way that you cannot find or access it physically, you cannot disable it. This might be harder than I imagine, because shielding the AirTag in a way that muffles the sound sufficiently might also shield the Bluetooth transmissions, which prevent the tracking in the first place. More experimentation will be needed here.

Lost Mode

Now that the covert tracking had been tested, Anthony set the AirTag to lost mode on his account. It took a few minutes for that change to propagate through the network. With lost mode enabled, I could call up his contact information (the owner can choose whether to show an email or the phone number) from the AirTag on my phone, just by tapping my phone to the AirTag.

Anthony also tried to play a sound on the AirTag, which was more than 7000km away from him. This did not work. It seems that playing the sound requires a local bluetooth connection to the AirTag. Since you would likely not be able to hear the sound when you are out of bluetooth range, and could use this to ‘terrorize’ someone (intentionally or not) in the middle of the night, I think this a reasonable limitation.

Transferring the AirTag

With all our testing done it was time for Anthony to remove the AirTag from his account, so that I could add it to my account. The interface for that in the Find My application is very straightforward.

He did, however, get an error that the iPhone could not “find” the AirTag. We presume his iPhone tried to connect to the AirTag over local bluetooth to let it “know” it was removed.

After Anthony had removed the device from his account, I tried to set it up on mine. This did not immediately work. Even after waiting for a few hours, my phone would not recognize the AirTag as new.

I then followed the instructions in this support article to reset the AirTag. It’s a bit tedious as you have to remove and replace the battery five times in a row. I figured out you don’t have to actually close the lid five times, just taking out the battery and putting it back in its place is sufficient. (there are magnets in the AirTag that seem to hold the batttery in place) After the reset process, the AirTag appeared immediately for setup on my phone and I could add it to my iCloud account.

Conclusion

Overall, the user experience for both the “Moving with you” and “Lost Mode” workflows are well thought through and kept clear and simple. Apple has good support articles for reference.

Many thanks to the comittee of MacDeployment and their sponsors that provided AirTags to all the speakers. And thanks to Anthony, who was game when I suggested that sending and tracking an AirTag across the Atlantic would be the “most fun” way to get them to me. Hope you found our experiments interesting, as well!

Right now, the AirTag has returned to my backpack. This seems reasonable since it stores my wallet and keys when I leave the house. I also want to test attaching an AirTag to my bike. I believe that bike thieves will quickly catch on to AirTags, so I don’t have high hopes for it to be useful as theft prevention. But an AirTag on the bike should be very useful to find my bike again in one of the typical Dutch bike parking areas among thousands of other bikes.

Notarize a Command Line Tool with notarytool

When Apple introduced notarization with Catalina, I published a post describing how to notarize a command line tool. At WWDC this year, Apple introduced updates to this process with Xcode 13 (currently in beta). Most importantly, there is a new command line tool called notarytool.

While the previous, altool-based, workflow still works in Xcode 13, there are many advantages to the new notarytool which makes its use much simpler.

Apple has documented this tool in a WWDC21 session and some developer articles, in addition we got some great information through the twitter account of one of the engineers, and Howard Oakley has already written a post as well:

What you need

  • Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates, nor access to the Xcode beta)
  • Xcode 13 (currently available as beta from the Apple Developer portal)
  • Developer ID Certificates
  • Application Specific Password for your Developer account
  • A command line tool project in Xcode

When you are building tools for macOS, you should have most of these already. We already covered these in the previous post, but to keep things in one place, I will cover them again, here.

Apple Developer Account

You need either the paid membership in the Apple Developer Program or be invited to an Apple Developer Enterprise Program team with access to the proper certificates.

You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.

Xcode 13 (beta)

Until the full version of Xcode 13 is released, you can get Xcode 13 beta from the beta downloads page on the Apple Developer Portal.

Once it is released (usually when iOS is released) you will be able to download it from the Mac App Store, as well.

Xcode 13 requires macOS Big Sur 11.3 or higher. According to this tweet from Rosyna Keller, notarytool can be extracted and run on macOS Catalina 10.15.7 and higher.

You can run the notarytool binary through xcrun:

% xcrun notarytool --help

If you need to extract the binary you can find where is stored on disk with:

% xcrun --find notarytool
/Applications/Xcode-beta.app/Contents/Developer/usr/bin/notarytool

Developer ID Certificates

There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.

To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.

We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.

If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.

Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:

% security find-identity -p basic -v

This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.

You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"

The ten character code at the end of the name is your Developer Team ID. Make a note of it, we will need it later. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.

Application Specific Password for your Developer Account

Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.

Note: If you followed the previous post’s instructions to store an application specific password for altool in the Keychain, you can extract that and re-use it for notarytool or create a new app-specific password.

Create a new application specific password in Apple ID portal for your developer account. Give it a name including notarytool so you know what you are using this for.

You will only be shown the password when you create it.

You can use notarytool to store the credentials in a keychain item, in a format that notarytool can read later.

% xcrun notarytool store-credentials --apple-id "name@example.com" --team-id "ABCD123456"

This process stores your credentials securely in the Keychain. You reference these credentials later using a profile name.

Profile name:
notary-example.com
Password for name@example.com: 
Validating your credentials...
Success. Credentials validated.
Credentials saved to Keychain.
To use them, specify `--keychain-profile "notary-example.com"`

The --store-credentials option will prompt for a profile name. You will need this name to retrieve the information later. Then it interactively prompts for the password associated with the given Apple Developer ID. Enter the application specific password here.

The credentials will be stored in the Keychain in an item named com.apple.gke.notary.tool. But you don’t really have to worry about that since notarytool will retrieve the credentials when you add the --keychain-profile "notary-example.com" option. (You can abbreviate the --keychain-profile with -p.)

If you are using iCloud Keychain, the credentials will be stored there, so they will be available to all other Macs you are using iCloud Keychain with. If you prefer, you can store the credentials in a specific (non-iCloud) keychain file with the --keychain option.

The Team ID is usually the 10-digit code which is also the certificates. However, in some cases the Team ID is different. You can can look-up Team IDs in the “Membership” area of the developer portal or with this altool command:

% xcrun altool --list-providers -u "name@example.com" -p "@keychain:<ITEM_NAME>"

(Thanks to ‘mhp’ for sharing this.)

You can also use an App Store Connect API key as an authentication option with notarytool. You can read notarytool‘s man page for details.

A Command Line Tool Project

You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.

My sample project for this article will be named “hello.”

Preparing the Xcode Project

The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.

The preparation in Xcode 13 diverges significantly from the steps required in the previous post. If you have created the project in earlier versions of Xcode, more configuration may be necessary.

Choosing the proper signing certificates

Before you can notarize the command line tool, it needs to be signed with the correct certificates.

  1. in Xcode, select the blue project icon in the left sidebar
  2. select the black “terminal” icon with your project’s name under the “Targets” list entry
  3. make sure the ‘Signing & Certificates’ tab is selected
  4. under ‘Signing’ disable ‘Automatically manage signing’
  5. choose your Team
  6. enter a bundle identifier for the binary
  7. choose ‘Developer ID Application‘ as the Signing Certificate

Hardened Runtime

Having the “Hardened Runtime” enabled is a requirement for notarization. When you create a new project in Xcode 13, the hardened runtime will be enabled by default. When you see the “Hardened Runtime” section under the “Signing” section, it is enabled.

When you are working with a older project, and do not see the “Hardened Runtime” section, you can enable the hardened runtime by clicking on the “+Capability” button above the “Signing” section and selecting “Hardened Runtime”.

Archive and export the binary

Choose “Archive” from the “Product” menu to build and create an archive. It will appear in the “Organizer” window. When that window does not open automatically, you can access it from the “Window” menu.

To export the binary product, select the latest archive and click on the “Distribute Content” button on the right. Choose “Built Products” as the method of distribution. Click “Next.” Choose a location to save the build products to.

This will create a directory with the project name and a timestamp in the chosen location. When you look inside this directory, you will see a “Products” directory and within it the binary in a /usr/local/bin/ directory hierarchy.

/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building the archive.

Build the installer package

Command Line Tools can be signed, but not directly notarized. You can however notarize a pkg file containing the Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.

We can use the Products directory as our payload to build the installer package:

% pkgbuild --root "hello 2021-mm-dd hh-mm-ss/Products" \
           --identifier "com.example.hello" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Name (ABCD123456)" \
           hello-1.0.pkg

I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.

This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.

If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”

Notarizing the Installer Package

Now we get to the new, most interesting part. We will notarize the newly-created installer package with notarytool:

% xcrun notarytool submit hello-1.0.pkg \
                   --keychain-profile "notary-scriptingosx" \
                   --wait

This is amazingly less effort than what we needed to do previously with the altool command. We give the filename of the archive we want to submit, the keychain profile with our credentials, and the --wait option.

notarytool will upload the file, give us a submission id, and then wait for the returned status from the Notary service. You can follow the output for the details.

You will also notice that notarytool uploads the pkg much faster than the previous altool workflow.

You can also drop the --wait option. Then the tool will submit the file and exit without waiting for a response. You can then use the info or log verbs with the submission id to get the status later. The Notary service does not seem to send emails anymore when the notarization check is complete.

There is also a --webhook option mentioned in the WWDC session which will make the Notary service call back to a webhook when the notarization is done. I have not seen any documentation on the details of this, though.

Finishing touch: stapler

Before you distribute the pkg, you can and should ‘staple’ the notarization before distributing it. This extra step will download the notarization information from Apple’s servers and attach it to the pkg. This is not mandatory, but will save the Gatekeeper service on the client an extra step when it verifies the pkg.

To do this, use the eponymous stapler tool:

% xcrun stapler staple hello-1.0.pkg

You can then verify that everything works with spctl:

% spctl --assess -vv --type install hello-1.0.pkg

Automation with Xcode

These steps are much simplified compared to the previous workflow. If you only build for distribution occasionally it would not be a big burden to do these steps manually.

Nevertheless, automating these steps saves effort and removes much pontential for errors.

When I wrote the previous post, I had not been able to figure out how all the pieces could work together to automate with a Xcode ‘Run Script’ as part of the normal “Archive” process. With the new tool and some inspiration from this developer article I have gotten this to work now.

In the project’s build settings, search for “Marketing Version” and set it to the version you want to use. Remember to update this entry for future updates as well. (You can use agvtool for this, but that is a topic for a different post.)

In Xcode, choose “Edit Scheme…” from the “Scheme” submenu in the “Project” menu. In the pane that opens, make sure the commnad line tool binary is selected at the top. Then expand the “Archive” section in the list on the left and select “Post-actions” in the expanded area. Use the “+” button at the bottom of the area to add a “New Run Script Action.”

Select the binary (again) in the popup next to “Provide build settings from”. Then paste the following in the code field:

With this post-action script in place, every “Archive” action will then also create a pkg in the project folder, submit it for notarization and staple the pkg. Since Xcode doesn’t show the output of post-action scripts, the script logs its output to a notary.log file, also in the project folder. Check that for success or failures. The notarization step takes a while after the “Archive” is complete, so you may have to wait a bit.

If you don’t want to run this workflow on every Archive, you can create a new scheme with this post-action script, then you can choose the scheme, before you do the “Archive” action.

Conclusion

The new notarytool included with Xcode 13 (beta) is a huge step up from the previous altool based workflows. It is much simpler and faster. You should start testing the tool now and move your workflows when possible.

Scripting OS X — Weekly News Summary for Admins — 2021-06-18

While attending WWDC and MacDevOps YVR last week – or at least attempting to – I realized that you can get real jet lag from virtual conferences.

Many people are catching up to the news from WWDC last week with many posts reacting to and/or summarizing the news. We also a patch for iOS 12 and and new macOS 11.5/iOS 14.7 betas.


(Sponsor: Mosyle)

Mosyle Business Free Logo

Free Remote Scripting with Mosyle Business FREE

From running scripts remotely to full Mobile Device Management (MDM) for macOS, iOS and tvOS, Mosyle Business FREE provides Apple enterprise customers with the complete MDM feature set of Mosyle Business PREMIUM for up to 30 devices at no charge.

Click here to learn more!


I’d like to thank Mosyle for being the new sponsor for this news summary! I have been watching what they have been doing in the macOS MDM space and believe they are a great fit as a sponsor.


If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

News and Opinion

macOS 12 Monterey and iOS 15

macOS and iOS Updates

MacAdmins on Twitter

  • Morten Just: “Go To Folder – ⌘⇧G in Finder got its first update in ~15 years with Monterey. Spotlight-style UI, and you can search for any folder https://t.co/jfGe1Z1RAw” (video)
  • Rich Trouton: “The more I look at macOS Monterey, the more that I’m convinced it is kindred to Mac OS X Snow Leopard – an OS focused on fixing existing issues and beneficial improvements to previously introduced features. This is a Good Thing and I look forward to the fall release.”
  • Anthony Reimer: “Reminded of the model-specific features of macOS Monterey by @howardnoakley and @TidBITS, I’ve updated my Mac Hardware/Software Obsolescence Chart (yes, for the 3rd time in a week) with info on support for Universal Control and AirPlay for Mac.”
  • mikeymikey: “Almost every scripted language in macOS provides a warning if you execute it interactively. Tcl, ruby, perl, you name it.”

Security and Privacy

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

To Listen

Just for Fun

Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

WWDC 2021 – What to watch for MacAdmins

So, WWDC started yesterday and Apple will release more than 200 sessions over the next week. Of course, most of these sessions are focussed on developers building apps for the Apple platforms. As a Mac and iOS admin, which sessions are interesting?

Of course, we won’t know which sessions are good until we can watch them. But here is the list of sessions that I expect to be interesting. I will update this post all week. Let me know if you find something interesting that I missed.

Apple releases the sessions every day of the WWC week at 9am PDT/17:00CEST. I will add the day of the week after the session title. You can watch the sessions in the Developer app on your Apple device and on Apple’s developer web page.

Keep in mind that while the excitement is huge around WWDC time, you do not have to watch all the sessions this week. The sessions will remain available and you can take your time to catch up.

Keynote and Platform State of the Union

The keynote is of course press- and end-user facing and very marketing driven. Still worth watching it (if you haven’t already) for the highlights. The ‘Platform State of the Union’ is the ‘real’ developer-focussed keynote. It is interesting to watch to understand where Apple thinks the focus is going to be. This year the highlights are the new Xcode Cloud, new Swift features, Object Capture and many new frameworks.

Device Management

There is actually a dedicated category for “Device Management” in the Developer app. Some of these sound very promising:

Swift and SwiftUI

I believe Swift will be more and more important for MacAdmins to build tools.

Security and Privacy

Other Sessions

Some of the other sessions will ahve relevance for MacAdmins (and users) as well. Often these sessions will have segments with a general overview of a feature, followed by details on how to implement it in code.

AppleSeed for IT

When you can use your managed Apple ID from Apple Business Manager or Apple School Manager to log in to AppleSeed for IT. There you can download the beta systems to start testing now. AppleSeed for IT also contains more detailed release notes, which you find as a PDF under the downloads category.

Keep in mind that while you can discuss information released in the WWDC sessions in public, information that is exclusive to AppleSeed for IT and the other seed and beta programs is subject to the NDA, and should not be discussed in public forums.

Weekly News Summary for Admins — 2021-04-16

This week Apple finally sent out the invitations for an event next week. The “Spring Loaded” event will take place next week, April 20, at 10am PDT.

Release of iOS 14.5, macOS 11.3, and siblings are likely on or around that day then. We did get an eighth beta this week. Next week should be interesting.


To support this weekly news summary, please consider:

macOS Terminal and Shell Book Cover

macOS Terminal and Shell:
You have always wanted to ‘learn Terminal,‘ right? This book teaches how (and why) to use the command line on macOS. Get it on Apple Books!

(If you have already bought the book, please leave a review on the Apple Books Store. Thank you!)


In other news, Parallels and Docker now have officially released Apple silicon native solutions. Ironically, Parallels cannot (yet?) host macOS Big Sur guest systems, only Windows 10 for ARM (preview) and ARM Linux systems.

The Docker does not come as a Universal download, but as two seperate downloads, which seems a common thing in this transition. Why is it that so many apps are separate downloads in this transition?

📰News and Opinion

🔐Security and Privacy

🔨Support and HowTos

🤖Scripting and Automation

♻️Updates and Releases

📺To Watch

🎧To Listen

🎈Just for Fun

📚 Support

If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Script Identity Crisis – get the Path to the Current Script

In shell scripting, there are many situations where you want to know the path to the script file itself.

I use this frequently in scripts that build pkgs, because I want the script to access or create files relative to the script or its enclosing folder. In other words: the script sits in some project folder, and you want to get the path to that project folder.

Some scripts just use the current working directory, but since you can launch scripts from outside the current working directory, this is quite unreliable.

One example is this script from my book ‘Packaging for Apple Administrators‘ which builds an installer package for a desktop picture from resources in the same folder.

In that package build script I have the following line:

projectfolder=$(dirname "$0")

This is a useful but somewhat tricky line. It will work in most situations, but not all. To understand when it might fail, we will have to unravel it from the inside.

The $0 variable is the first element of the command line as it is entered in to the prompt and contains the command itself. In our case it will most likely be ./buildBoringDesktopPkg.sh.

The dirname command removes the last element from a path. This will return the path to the directory containing the item you give it. When we do dirname $0 after launching the script with ./buildBoringDesktopPkg.sh we will get . which is the current working directory, but also the parent directory of the script.

When you launch the build script with a different working directory, i.e. like

> Projects/BoringDesktop/buildBoringDesktopPkg.sh

The $0 argument will be Projects/BoringDesktop/buildBoringDesktopPkg.sh the dirname will return Projects/BoringDesktop/. This is all good and it will still find the resources and create the pkg file in its project folder.

In most situations the syntax used to call your script will be a relative or absolute path to the script. Something like ./buildPkg.sh or ~/Projects/Tools/my_script.sh. When the shell ‘sees’ a slash / in the command, it will assume this is the path to the script or executable and use that.

But when there is no slash / in the command, the shell will use the directories listed in the PATH environment variable to look for the command. This is the reason you can type sw_vers instead of having to write /usr/bin/sw_vers.

This is also the reason you have to type ./my_script.sh to run a script you are working on. Your script is not in the PATH, but the command contains a slash and tells the shell exactly where to look (in the current directory). When you just type my_script.sh the shell will look in the PATH directories, not find your script, and error out. (Unless your current directory happens to be in the PATH.)

Note: Learn the details about PATH in my new book: “macOS Terminal and Shell

So, there are a lot of pieces that are really confusing here. Let’s create a script that will help us visualize this:

#!/bin/sh

echo "pwd: $(pwd)"
echo "\$0: $0"
echo "dirname \$0: $(dirname $0)"

This script shows the values of the current working directory, the $0 argument, and the value returned by dirname $0. Save this script as script_path_test.sh in your Desktop folder.

> cd Desktop                      
> ./script_path_test.sh     
pwd: /Users/armin/Desktop
$0: ./script_path_test.sh
dirname $0: .
> cd ~                  
> Desktop/script_path_test.sh     
pwd: /Users/armin
$0: Desktop/script_path_test.sh
dirname $0: Desktop

So far, so good. While some of these paths look a little odd, they all return the expected values.

But now imagine we like our script so much, that we use it a lot. To save on typing the path to the script all the time, we add a symbolic link to it to /usr/local/bin. This puts our script in the default PATH (for an interactive shell) and we can now execute it without needing to type the path to it:

> sudo ln -s ~/Desktop/script_path_test.sh /usr/local/bin/script_path_test.sh
> script_path_test.sh
pwd: /Users/armin
$0: /usr/local/bin/script_path_test.sh
dirname $0: /usr/local/bin

So, this is less ideal. When you enter just script_path_test.sh the shell finds the symbolic link in /usr/local/bin and runs it. The $0 argument points to the symbolic link instead of the actual script and hence dirname $0 points to /usr/local/bin which is not where the resources for our script are.

(Remember to delete this symbolic link from /usr/local/bin/ when you are done with this.)

We need a tool that resolves symbolic links.

There are quite a few solutions to this, many of which can be found in this Stack Overflow post.

GNU-based systems have a realpath command line tool, which does exactly this. macOS does not have this tool. You can add GNU tools to macOS with many of the popular package managers, but that introduces a dependency and seems overkill.

Python has a function that does this. Add this line to the script:

echo "python realpath: $(python -c "import os; print(os.path.realpath('$0'))")"

This works fine, however, the built-in Python for macOS is deprecated. We don’t want to introduce this dependency, when it might go away in a future macOS update.

Then I found this post where I learned that zsh has a parameter expansion modifier to return the absolute path while resolving symbolic links.

Let’s modify our script to use zsh:

#!/bin/zsh

echo "pwd: $(pwd)"
echo "\$0: $0"
echo "dirname \$0: $(dirname $0)"

echo "\${0:A}: ${0:A}"
echo "dirname \${0:A}: $(dirname ${0:A})"

and then we can run our script:

> script_path_test.sh
pwd: /Users/armin
$0: /usr/local/bin/script_path_test.sh
dirname $0: /usr/local/bin
${0:A}: /Users/armin/Desktop/script_path_test.sh
dirname ${0:A}: /Users/armin/Desktop

Zsh to the rescue. We have a solution! Zsh is pre-installed on every macOS and since Apple chose it as their default shell since Catalina, it will be around for a while longer.

But what if you have sh or bash scripts that you cannot just quickly convert to zsh? The good news here is that we can create a zsh one-liner that does this, even from a bash or sh script, very similar to the python one-liner above.

echo "zsh absolute path: $(zsh -c 'echo ${0:A}' "$0")"
echo "zsh dirname absolute: $(dirname $(zsh -c 'echo ${0:A}' "$0"))"

So, in conclusion:

  • dirname $0 will give you the enclosing folder of the script in most situations
  • with zsh you can just use dirname ${0:A} to be even safer
  • when you have to use sh or bash, you can use a zsh one-liner: dirname $(zsh -c 'echo ${0:A}' "$0").

Happy Scripting!

(Remember to delete the symbolic link from /usr/local/bin/!)

Ten Years of Scripting OS X

The oldest post on this weblog happened ten years ago. It makes me especially proud, that the first post has aged quite well. (The second post, not so much.)

This weblog was (quite obviously) inspired by the weblogs of Greg Neagle, Rich Trouton, Ben Toms, and many others. I wanted to share and give back to the MacAdmins community. I have always enjoyed scripting, in the sense of tinkering and combining tools in useful, or sometimes just funny ways and sharing that seemed like an obvious thing to do.

I wasn’t really planning on honing my writing skills for writing books later. As many of these personal weblogs go, I posted quite irregularly for the first few years. Still, I was often quite excited when a post got dozens of views.

As it turns out, a weblog is also a decent marketing platform for a self-published book (or two, or three, or four). When you are writing a book, there are always a few things that don’t quite “fit in” and that then get turned into a blog post. Once I got into the habit of posting regularly, the viewer numbers and the books sales increased. At some point, my ranking in the weird black magic that is the Google algorithm reached some critical point and the views from searches started pouring in.

Having posts being read and re-shared does give you a bit of a thrill. And I wanted to share that feeling. I started the weekly news summary to have a place where all the great work of fellow MacAdmins (and related experts) is gathered. I got requests to have an email newsletter fairly quickly. We passed 1000 subscribers on the email newsletter in October, and many more read it on the website.

Last week, according to the Jetpack metrics, this site had its millionth unique visitor overall and its 500.000th unique visitor in this calendar year. Yes, that means traffic is more than doubling year over year.

Back when a post would get hundreds of views that was exciting. The traffic now is still exciting, but also a bit humbling. So, thank you all for being here and reading and sharing my posts and books.

There was never a big strategy or plan. But I am very happy how everything worked out. Over the years, writing for the weblog gave me confidence to write and self-publish a book. The books and posts lead to conference presentations, which lead to more posts and books and the newsletter. And the best part is: I got to meet and befriend some pretty great people from all over the world.

And true to the spirit of how the last ten years turned out, I have no plans or strategies for major changes in the near future. I will keep the things that work and try new things as they occur to me. That doesn’t mean that there won’t be a few minor changes happening soon, though.

On to the next ten years!