MDOYVR 22 Talk: The Encyclopedia of macOS Automation

Last week I had the pleasure and honor of participating and presenting at MacDevOps YVR. The videos for the sessions are now appearing on YouTube.

There is a page for my talk “The Encyclopedia of macOS Automation,” in which I discuss the options for scripting and automation on macOS, with extra links and notes. You can go directly to the video here.

The talks this year were graphic recorded by the amazing Ashton Rodenhiser (website, twitter). The graphic at the top of this post was made by her while I was presenting.

As always, I had a lot of fun at this conference. Many thanks to the organizers and all the other speakers. Until next year!

Scripting macOS, part 7: Download and Install Firefox

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Follow this blog or the Twitter account for updates on the book’s progress!

Download and Install Firefox

To further illustrate the progress from the idea of a workflow to a working script, let us look at another, more involved example.

To download and install the latest version of Firefox a user has to go to the Firefox website and download the latest version, which will come as a disk image (dmg) file. Then the user needs locate the dmg in the ~/Downloads folder and open it to mount the virtual disk image. Finally, they need to copy the Firefox application from the virtual disk to the Applications folder.

When we want to automate the task ‘Download and Install Firefox,’ we have the following steps:

  • download latest Firefox disk image
  • mount downloaded disk image
  • copy Firefox application to /Applications
  • unmount disk image

From this list of steps, we can build the first ‘frame’ of our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

This breaks the workflow into smaller pieces, that we will now tackle individually.

Download from the Command Line

You can use the curl command to download data in the command line. The curl command is very complex and has many options. We will only discuss the few options that we require for our task here. As always, you can find a detailed description of the curl command and its options in the curl man page.

The URI to download the latest Firefox is
https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US

However, when you try to curl this URI, you only get the following:

> curl "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US"
<a href="https://download-installer.cdn.mozilla.net/pub/firefox/releases/86.0.1/mac/en-US/Firefox%2086.0.1.dmg">Found</a>.

This is a re-direction, that is commonly used to have a single URI, that is redirected to different final URIs, so that when the software updates, the same URI always returns the latest version.

We can tell curl to follow these redirections with the --location option.

By default, the curl command will output the download to standard out. To save the download to a file, we can use the --output option with a file name.

> curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" --output Firefox.dmg

This command will download the latest Firefox disk image to a file named Firefox.dmg in your current working directory. We can use this as our first step:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

Note: Like many other command line tools, curl has short and long options. The short options for –location and –output are -L and -o.
Short options are convenient in the interactive shell, as they save typing and reduce the potential for typos. But they are much less readable, and you usually have to look up their function in the documentation. For that reason, I recommend using the long, descriptive options in scripts.

Working with Disk Images

The command line tool to work with disk image (dmg) files on macOS is hdiutil. This is also a very powerful command with many verbs and options. You can find all the detail in the hdiutil man page.

To mount a disk image, use the attach verb:

> hdituil attach Firefox.dmg

This will output some information and mount the virtual disk. The last line ends with the path to the mounted virtual disk /Volumes/Firefox.

By default, you can see the mounted volume in Finder. We do not really need the disk image to appear in Finder while the script is running. We can suppress this behavior with the -nobrowse option.

Since we are only going to read from the disk image, we can tell hdiutil to mount the dmg in readonly mode with the -readonly option. This speeds things up a bit.

> hdiutil attach Firefox.dmg -nobrowse -readonly

You can unmount or eject the virtual disk with

> hdiutil detach -force /Volumes/Firefox

The -force option will unmount the disk image, even when another process is still using it.

Thehdiutil command covers two of our steps, so we can fill them in:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications

# unmount disk image
hdiutil detach /Volumes/Firefox -force

Copying the Application

When you manually install Firefox the disk image shows you a nice graphic that reminds you to drag the app to the Applications folder. Once the disk image is mounted, the cp command can be used to do this in the shell:

> cp -R /Volumes/Firefox/Firefox.app /Applications/

This provides the last missing step in our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications
echo "copying Firefox to /Applications"
cp -R /Volumes/Firefox/Firefox.app /Applications/

# unmount disk image
hdiutil detach /Volumes/Firefox/ -force

You can now test the script. If Firefox is running, you want to quit it before you run the script. You may also want to delete the existing copy of Firefox from the Applications folder, to be sure that your script is doing the work.

Lists of Commands—Conclusion

We have been able to automate a fairly complex workflow with a script of four commands.

To be perfectly honest, this script (as well as all the others we have built so far) is not complete yet.

A ‘proper’ script needs to be able to react to errors that occur. In our example, imagine the download fails. The script should be able to detect the failure before it overwrites the installed, functional Firefox application.

We will get to this kind of error handling later.

Nevertheless, this script is already useful in its current form. You can try to adapt this script to work with some other software you can download as a disk image.

You can also add extra commands that

  • delete the downloaded disk image at the end
  • open the newly installed Firefox app after installation
  • quit or kill the Firefox process before copying the new version

In the book “Scripting macOS”, you will learn more scripting techniques, and we will re-visit some of these sample scripts and keep improving them.

Follow this blog or the Twitter account for updates on the book’s progress!

Note: After using different variations of these kinds of workflows, I did put together a more generic script to download and install various kinds of software, called ‘Installomator.’ You can see the script at its open source repository on GitHub.

Download Full Installer

A while back I wrote up a blog post on deploying the Install macOS Big Sur application. As one of the solutions, I posted a script (based on Greg Neagle’s installinstallmacos.py) which listed the pkgs from Apple’s software update catalogs so you could download them.

During and after WWDC, I wanted to see if I could build a SwiftUI app. I thought that building a user interface for this task would be a nice practice project.

Ironically, since I want the app to work on Big Sur, I could not use any of the new Swift and SwiftUI features Apple introduced this year. Even so, since I had not used SwiftUI to build a Big Sur application, most of the features Apple introduced last year were still new to me.

It was often unexpected to me which parts turned out to be challenging and which parts were really easy to implement. For example, implementing a preferences window, turned out to be super-easy, but it took me two false-starts to find the correct approach. Communicating with the preferences system of macOS is also very easy, but so poorly documented that you are always second guessing if what you are doing is right.

Apple’s documentation for Swift and SwiftUI on this has definite highlights, but is very sparse overall. I am still not sure if some of the decisions I made while putting this together were “good” choices.

Nevertheless, it works! I think it might be a nice tool to have, so I put it on GitHub. You can just download the app from the release page and use it, or clone the repo and take a look at the code.

Constructive feedback is always welcome! I am still learning this as I go along, too.

Installomator Updated: v0.5

It has been a while, mainly because I was busy with other things, but there finally is a new release version of Installomator!

The reason work has progressed—quite significantly—even though I was distracted is that Søren Theilgaard and Isaac Ordonez have joined the project as conributors. All of the work from 0.4 to 0.5 was from one of them. We ahve some great plans to move this tool forward, as well.

Many of these new app labels have been provided from others, either through GitHub issues, pull requests, or through comments in the #installomator channel on MacAdmins Slack. Thanks to all who contributed.

What’s new in v0.5:

  • Major update and now with help from @Theile and @Isaac
  • Added additional BLOCKING_PROCESS_ACTION handlings
  • Added additional NOTIFY=all. Usuful if used in Self Service, as the user will be notified before download, before install as well as when it is done.
  • Added variable LOGO for icons in dialogs, use LOGO=appstore (or jamf or mosyleb or mosylem or addigy). It’s also possible to set it to a direct path to a specific icon. Default is appstore.
  • Added variable INSTALL that can be set to INSTALL=force if software needs to be installed even though latest version is already installed (it will be a reinstall).
  • Version control now included. The variable appNewVersion in a label can be used to tell what the latest version from the web is. If this is not given, version checking is done after download.
  • For a label that only installs a pkg without an app in it, a variable packageID can be used for version checking.
  • Labels now sorted alphabetically, except for the Microsoft ones (that are at the end of the list). A bunch of new labels added, and lots of them have either been changed or improved (with appNewVersion og packageID).
  • If an app is asked to be closed down, it will now be opened again after the update.
  • If your MDM cannot call a script with parameters, the label can be set in the top of the script.
  • If your MDM is not Jamf Pro, and you need the script to be installed locally on your managed machines, then take a look at Theiles fork. This fork can be called from the MDM using a small script.
  • Script buildCaseStatement.sh to help with creating labels have been improved.
  • Fixed a bug in a variable name that prevented updateTool to be used
  • added type variable for value "updateronly" if the label should only run an updater tool.

And if you are counting, there are now more than 260 application labels in Installomator. However, that number is a bit inflated, because several vendors have multiple downloads for Intel and Apple Silicon apps.

Get the script and find the instructions on the GitHub repo.

If you have any feedback or questions, please join us in the #installomator channel on MacAdmins Slack.

Thanks again to all those who contributed!

(Installomator Icon credit: Mischa van der Bent)

EraseInstall application retired

It makes me very sad that the EraseInstall application has been retired.

We built this tool three years ago, mostly because we wanted to learn how to build an app like this on macOS. We think it worked out well. We learned a lot, and are glad the application was useful to some.

Since then, all the people involved in the EraseInstall Project have moved on to other jobs or other responsibilities. Unfortunately, this leaves us with no time or resources to maintain or improve EraseInstall.

The repository and the application will remain available in its current state. There will be no more updates. If someone feels they can take up the project and continue it, please do!

If you are looking for a similar solution, we recommend Graham Pugh’s eraseinstall script.

Thank you, all, again!

Team EraseInstall: Mischa van der Bent, Arnold Nefkens, and Armin Briegel

Wrangling Pythons

As I noted in my last Weekly News Summary, several open source projects for MacAdmins have completed their transition to Python 3. AutoPkg, JSSImport and outset announced Python 3 compatible versions last week and Munki already had the first Python 3 version last December.

Why?

Apple has included a version of Python 2 with Mac OS X since 10.2 (Jaguar). Python 3.0 was released in 2008 and it was not fully backwards compatible with Python 2. For this reason, Python 2 was maintained and updated alongside Python 3 for a long time. Python 2 was finally sunset on January 1, 2020. Nevertheless, presumably because of the compatibility issues, Apple has always pre-installed Python 2 with macOS and still does so in macOS 10.15 Catalina. With the announcement of Catalina, Apple also announced that in a “future version of macOS” there will be no pre-installed Python of any version.

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. (macOS 10.15 Catalina Release Notes)

This also applies to Perl and Ruby runtimes and other libraries. I will be focussing on Python because it is used more commonly for MacAdmin tools, but most of this post will apply equally to Perl and Ruby. Just mentally replace “Python” for your preferred language.

The final recommendation is what AutoPkg and Munki are following: they are bundling their own Python runtime.

How to get Python

There is a second bullet in the Catalina release notes, though:

Use of Python 2.7 isn’t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS won’t include Python 2.7. Instead, it’s recommended that you run python3 from within Terminal. (51097165)

This is great, right? Apple says there is a built-in Python 3! And it’s pre-installed? Just move all your scripts to Python 3 and you’ll be fine!

Unfortunately, not quite. The python3 binary does exist on a ‘clean’ macOS, but it is only a stub tool, that will prompt a user to download and install the Command Line Developer Tools (aka “Developer Command Line Tools” or “Command Line Tools for Xcode”). This is common for many tools that Apple considers to be of little interest to ‘normal,’ non-developer users. Another common example is git.

Dialog prompting to install the Command Line Tools
Dialog prompting to install the Command Line Tools

When you install Xcode, you will also get all the Command Line Developer Tools, including python3 and git. This is useful for developers, who may want to use Python scripts for build operation, or for individuals who just want to ‘play around’ or experiment with Python locally. For MacAdmins, it adds the extra burden of installing and maintaining either the Command Line Developer Tools or the full Xcode install.

Python Versions, a multitude of Snakes

After installing Xcode or the Command Line Developer Tools, you can check the version of python installed: (versions on macOS 10.15.3 with Xcode 11.3.1)

> python --version    
Python 2.7.16
> python3 --version    
Python 3.7.3

When you go on the download page for Python.org, you will get Python 3.8.1 (as of this writing). But, on that download page, you will also find download links for “specific versions” which include (as of this writing) versions 3.8.1, 3.7.6, 3.6.10, 3.5.9, and the deprecated 2.7.17.

The thing is, that Python isn’t merely split into two major release versions, which aren’t fully compatible with each other, but there are several minor versions of Python 3, which aren’t fully compatible with each other, but are still being maintained in parallel.

Developers (individuals, teams, and organisations) that use Python will often hold on to a specific minor (and sometimes even patch) version for a project to avoid issues and bugs that might appear when changing the run-time.

When you install the latest version of Munki, it will install a copy of the Python framework in /usr/local/munki/ and create a symbolic link to that python binary at /usr/local/munki/python. You can check its version as well:

 % /usr/local/munki/python --version
Python 3.7.4

All the Python code files for Munki will have a shebang (the first line in the code file) of

#!/usr/local/munki/python

This ensures that Munki code files use this particular instance of Python and no other copy of Python that may have been installed on the system.

The latest version of AutoPkg has a similar approach:

> /usr/local/autopkg/python --version    
Python 3.7.5

In both cases the python binary is a symbolic link. This allows the developer to change the symbolic link to point to a different Python framework. The shebangs in the all the code files point to the symbolic link, which can be changed to point to a different Python framework.

This is useful for testing and debugging. Could MacAdmins use this to point both tools to the same Python framework? Should they?

The Bridge to macOS

On top of all these different versions of Python itself, many scripts, apps, and tools written in Python rely on ‘Python modules.’ These are libraries (or frameworks) of code for a certain task, that can be downloaded and included with a Python installation to extend the functionality of Python.

The most relevant of these modules for MacAdmins is the “Python Objective-C Bridge.” This module allows Python code to access and use the native macOS Cocoa and CoreFoundation Frameworks. This not only allows for macOS native GUI applications to be written in Python (e.g. AutoDMG and Munki’s Managed Software Center [update: MSC was re-written in Swift last year]), but also allows short scripts to access system functions. This is sometimes necessary to get a data that matches what macOS applications “see” rather than what the raw unix tools see.

For example, the defaults tool can be used to read the value of property lists on disk. But those might not necessarily reflect the actual preference value an application sees, because that value might be controlled by a different plist file or configuration profile.

(Shameless self-promotion) Learn more about Property lists, Preferences and Profiles

You could build a tool with Swift or Objective-C that uses the proper frameworks to get the “real” preference value. Or you can use Python with the Objective-C bridge:

#!/usr/bin/python
from Foundation import CFPreferencesCopyAppValue
print CFPreferencesCopyAppValue("idleTime", "com.apple.screensaver")

Three simple lines of Python code. This will work with the pre-installed Python 2.7, because Apple also pre-installs the Python Objective-C bridge with that. When you try this with the Developer Tools python3 you get an error:

ModuleNotFoundError: No module named 'Foundation'

This is because the Developer Tools do not include the Objective-C bridge in the installation. You could easily add it with:

> sudo python3 -m pip install pyobjc

But again, while this command is “easy” enough for a single user on a single Mac, it is just the beginning of a Minoan labyrinth of management troubles.

Developers and MacAdmins, have to care about the version of the Python they install, as well as the list of modules and their versions, for each Python version.

It is as if the Medusa head kept growing more smaller snakes for every snake you cut off.

(Ok, I will ease off with Greek mythology metaphors.)

You can get a list of modules included with the AutoPkg and the Munki project with:

> /usr/local/munki/python -m pip list
> /usr/local/autopkg/python -m pip list

You will see that not only do Munki and AutoPkg include different versions of Python, but also a different list of modules. While Munki and AutoPkg share many modules, their versions might still differ.

Snake Herding Solutions

Apple’s advice in the Catalina Release Notes is good advice:

It’s recommended that you bundle the runtime within the app.

Rather than the MacAdmin managing a single version of Python and all the modules for every possible solution, each tool or application should provide its own copy of Python and its required modules.

If you want to build your own Python bundle installer, you can use this script from Greg Neagle.

This might seem wasteful. A full Python 3 Framework uses about 80MB of disk space, plus some extra for the modules. But it is the safest way to ensure that the tool or application gets the correct version of Python and all the modules. Anything else will quickly turn into a management nightmare.

This is the approach that Munki and AutoPkg have chosen. But what about smaller, single script solutions? For example simple Python scripts like quickpkg or prefs-tool?

Should I bundle my own Python framework with quickpkg or prefs-tool? I think that would be overkill and I am not planning to do that. I think the solution that Joseph Chilcote chose for the outset tool is a better approach for less complex Python scripts.

In this case, the project is written to run with Python 3 and generic enough to not require a specific version or extra modules. An admin who wants to use this script or tool, can change the shebang (the first line in the script) to point to either the Developer Tool python3, the python3 from the standard Python 3 installer or a custom Python version, such as the Munki python. A MacAdmin would have to ensure that the python binary in the shebang is present on the Mac when the tool runs.

You can also choose to provide your organization’s own copy Python with your chosen set of modules for all your management Python scripts and automations. You could build this with the relocatable Python tool and place it in a well-known location the clients. When updates for the Python run-time or modules are required, you can build and push them with your management system. (Thanks to Nathaniel Strauss for pointing out this needed clarifying.)

When you build such scripts and tools, it is important to document which Python versions (and module versions) you have tested the tool with.

(I still have to do that for my Python tools.)

What about /usr/bin/env python?

The env command will determine the path to the python binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the python binary is unknown.

This is useful when developers want to use the same script in different environments across different computers, user accounts, and platforms. However, this renders the actual version of python that will interpret the script completely unpredictable.

Not only is it impossible to predict which version of Python will interpret a script, but you cannot depend on any modules being installed (or their versions) either.

For MacAdmin management scripts and tools, a tighter control is necessary. You should use fixed, absolute paths in the shebang.

Conclusion

Managing Python runtimes might seem like a hopeless sisyphean task. I believe Apple made the right choice to not pre-install Python any more. Whatever version and pre-selection of module versions Apple would have chosen, it would only have been the correct combination for a few Python solutions and developers.

While it may seem wasteful to have a multitude of copies of the Python frameworks distributed through out the system, it is the easiest and most manageable solution to ensure that each tool or application works with the expected combination of run-time and modules.

Build a macOS Application to Run a Shell Command with Xcode and SwiftUI

A few years ago, I published a post that described how to build a Mac application in Swift that would run a shell command. Surprisingly, this post still gets a lot of traffic. Xcode and Swift have progressed over the last three years and that old example is mostly useless now. It is time for an update!

In this post, I will describe how to build a simple Mac app which runs a shell command using SwiftUI.

SwiftUI is the new framework from Apple to build user interfaces across platforms. Applications built with Swift UI will require 10.15 Catalina and higher. As you will see, SwiftUI does simplify a lot of the work of creating the user interface. I believe that learning SwiftUI now will be a good investment into the future.

I have written this post with Xcode 11.2.1 on macOS Catalina 10.15.1. That means we are using Swift 5.1. You can download Xcode from the Mac App Store or Apple’s developer page.

In this post, we will be using the say command to make the Mac speak. You can test the command by opening the Terminal and entering

> say "Hello World"

The say command is a simple and fun stand in for other command line tools. It also provides instant feedback, so you know whether it is working or not. That said, when you want to provide text-to-speech functionality in a Swift app, you should probably use the text-to-speech frameworks, rather than sending out a shell command.

Nevertheless, for Mac Admin tasks, there are some that are only possible, or at least much easier, with shell tools.

First: Swift UI hello world

Before we get to more complicated things, let’s honor the classics and build a “Hello, World” application using Swift UI.

Open Xcode and from the project picker select “New Project.”

In the template chooser select ‘macOS’ and ‘Application’. Then click ‘Next.’

In the next pane, enter a name for the project: “SayThis.” Verify the other data and make sure the choice for ‘User Interface’ is ‘SwiftUI’. Then click ‘Next.’

The window with the new project will have four vertical panes. You can use the controller element in the top right of the toolbar to hide the right most “Inspector” pane as we will not need it.

Click it the “Play” button in the top left to build and run the template code, a window should open which show the place holder text “Hello World!”

When you return to Xcode the preview pane on the right should now be active and also display the “Hello World!” text. If it does not there should be a “Resume” button at the top of the preview pane (also called “Canvas”) which you can click to make Xcode resume live updating the preview. There are many situations that will stop Xcode from continuously updating the preview and you can click this button to resume.

The left column shows a list of files in this project. Select the ContentView.swift file. This file contains the Swift UI code that sets up the interface. You will see the code in the center pane. The relevant part of the code is:

struct ContentView: View {
    var body: some View {
        Text("Hello, World!")
            .frame(maxWidth: .infinity, maxHeight: .infinity)
    }
}

The ContentView contains a body which contains a Text element with the text Hello World!. at the end of the Text object, you a ‘modifier’ that sets the frame of the Text object to use all available space.

Hello, me!

In the code pane, change World to something else:

Text("Hello, Armin!")

You will see that the text in the preview pane updates as you change the code.

Note: there are several reasons Xcode might stop updating and when it does, you will have to hit the ‘Resume’ button above the preview pane.

The preview pane or canvas allows you to edit as well. When you (command) ⌘-click on the “Hello, …” text in the canvas, you will get an “Action menu.”

The first item in the action menu: “Show SwiftUI Inspector” will an inspector with the attributes or “modifiers” of the text object. Note that, even though there is no visual indication, the inspector view can be scrolled to reveal more options.

Change the font of the text to Body. The preview in the canvas will update, as well as the code. The code will now look like:

Text("Hello, Armin!")
    .font(.body)
    .frame(maxWidth: .infinity, maxHeight: .infinity)

I have broken the modifiers into their own lines for clarity.

Stacking it up… or down… or sideways…

The SwiftUI body can only contain one object. But there are SwiftUI objects that you can use to group multiple objects together. Command-Click on the text and choose “Embed in VStack” from the action menu.

The code will change to:

 VStack {
    Text("Hello, Armin!")
        .font(.body)
        .frame(maxWidth: .infinity, maxHeight: .infinity)
}

The VStack stands for ‘vertical stack.’ The VStack and its sibling the HStack (horizontal) can contain multiple objects. So, you can add another Text over our “Hello” text:

VStack {
    Text("SayThis")
        .font(.largeTitle)
    Text("Hello, Armin!")
        .font(.body)
}.frame(maxWidth: .infinity, maxHeight: .infinity)

This adds the "SayThis" text above the "Hello, ..." text in a larger font. I have also moved the .frame(...) modifier to the VStack because that leads to a nicer output. Feel free to apply the .frame modifer to different objects to see how that affects the layout.

The text is a little bit close together, so we can add some .padding() modifiers:

VStack {
    Text("SayThis")
        .font(.largeTitle)
        .padding()
    Text("Hello, Armin!")
        .font(.body)
        .padding()
}.frame(maxWidth: .infinity, maxHeight: .infinity)

You can choose to change the properties using the inspector UI or in the code. Either way the changes should be reflected in the canvas and the code.

Adding some interaction…

Now we want to add some interaction. Eventually, we want the user to be able to enter some text, which will be sent to the say command. Before we get, we will add a field, to enter some text.

To add a TextField where the user can enter text to the layout, click on the + icon in the top right of the tool bar. A window will appear with SwiftUI objects. Enter TextField in the search area to find the TextField object and drag it between the two text items.

You will want to add a .padding() modifier to the text field as well.

We also need a local variable in our ContentView to store the value of the TextField. Add this variable declaration under the struct ContentView definition and before the body declaration:

@State var message = "Hello, World!"

This adds and initializes a variable named message. The @State marker tells SwiftUI this ‘State’ variable will be used with the view. The variable is part of the ‘state’ of the view. Changes to the variable will immediately update the UI and vice versa.

Use the message variable for the TextField and Text objects:

@State var message = "Hello, World!"

var body: some View {
    VStack {
        Text("SayThis")
            .font(.largeTitle)
            .padding()
        TextField("Message", text: $message)
            .padding()
        Text(message)
            .font(.body)
            .padding()
    }.frame(maxWidth: .infinity, maxHeight: .infinity)
}

When you look at this interface in the preview pane, you will see that the contents of the message variable are reflected in the TextField and Text. To test the interactivity, you need to either ‘build and run’ the application or hit the ‘Play’ button in the preview. Then you can change the text in the text field and immediately see the changes displayed in the Text below.

Declaring the message variable as a @State will automatically set up all the notifications for these seamless updates.

Buttons and Actions

While updating another object is quite cool, it is not what we set out to do. Let’s get back on track.

  • Remove the last Text(message) and all its modifiers.
  • Command-click on the TextField and use the action menu to embed it in an HStack (horizontal stack)
  • Drag a Button from the object Library (you can open this with the ‘+’ button in the top right of the window) next to the text field

The code you generated should look like this:

HStack {
    TextField("Message", text: $message)
        .padding()
    Button(action: {}) {
        Text("Button")
    }
}

You can change the label of the button by changing the Text object inside of it. Change "Button" to "Say". You also want to add a .padding() modifier to the button.

The preview should now look like this:

The action of the Button is a code block that will be run when the button is clicked. Add the code between the action closure brackets:

Button(action: {
    let executableURL = URL(fileURLWithPath: "/usr/bin/say")
    try! Process.run(executableURL,
                     arguments: [self.message],
                     terminationHandler: nil)
}) {
    Text("Say")
}.padding(.trailing)

This uses the run() convenience method of the Process class. You need to provide the full path to the command line tool. You can determine the full path in Terminal with the which command:

> which say
/usr/bin/say

The arguments to the say command need to be provided as an Array of Strings. The process is launched asynchronously, so the main thread continues immediately. We can also provide a code block that will be executed when the Process terminates. For now, we have no code that needs to be run on termination, so we set the terminationHandler to nil.

Note: using the try! statement to crash when this method throws an exception is lazy. I use it here to simplify the code drastically. Process.run() will throw an error when the file at exectableURL does not exist or is not executable. We can be fairly certain the /usr/bin/say executable exists, so the try! statement is justifiable. You should probably add proper error prevention and handling in production code.

Updating the Interface

The say command is executed asynchronously. This means that it will be running in the background and our user interface remains responsive. However, you may want to provide some feedback to the user while the process is running. One way of doing that, is to disable the ‘Say’ button while the process is running. Then you can re-enable it when the process is done.

Add a second @State variable to track whether our process is running:

@State var isRunning = false

Add a .disabled(isRunning) modifier to the Button. This will disable the button when the value of the isRunning variable changes to true.

Then we add a line in the Button action code to set that variable to true. We also add a code block to the terminationHandler which sets its value back to false:

 Button(action: {
    let executableURL = URL(fileURLWithPath: "/usr/bin/say")
    self.isRunning = true
    try! Process.run(executableURL,
                     arguments: [self.message],
                     terminationHandler: { _ in self.isRunning = false })
}) {
    Text("Say")
}.disabled(isRunning)
    .padding(.trailing)

Now, when you press the button, it will disable until the say process is done speaking.

Sample Code

For your convenience, here is the finished code for the ContentView.swift class:

//
//  ContentView.swift
//  SayThis
//

import SwiftUI

struct ContentView: View {
    @State var message = "Hello, World!"
    @State var isRunning = false
    
    var body: some View {
        VStack {
            Text("SayThis")
                .font(.largeTitle)
                .padding()
            HStack {
                TextField("Message", text: $message)
                    .padding(.leading)
                Button(action: {
                    let executableURL = URL(fileURLWithPath: "/usr/bin/say")
                    self.isRunning = true
                    try! Process.run(executableURL,
                                     arguments: [self.message],
                                     terminationHandler: { _ in self.isRunning = false })
                }) {
                    Text("Say")
                }.disabled(isRunning)
                    .padding(.trailing)
            }
        }.frame(maxWidth: .infinity, maxHeight: .infinity)
    }
}


struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

More Tutorials

If you want to learn more about SwiftUI and how it works, Apple has some excellent Tutorials on their developer page:

Notarize a Command Line Tool

There is an updated version of this post for the new tools in Xcode 13.

The upcoming macOS 10.15 Catalina will require more apps and tools to be notarized. Apple has somewhat loosened the requirements at last minute, but these changed limitations are only temporary, to give developers more time to adapt.

Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.

Howard Oakley has written up his experiences and that post was extremely helpful. But there were a few omissions and some steps that aren’t really necessary, so I decided to make my own write-up.

And yes, there is a script at the end…

Note: these instructions are for macOS 10.14.6 Mojave, Xcode 10.3 and Swift 5.0. It is very likely that the details will change over time.

Update 2019-09-24: Tested with Xcode 11 and it still works (the screen layout has changed for some of the options)

What do you need?

  • Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates)
  • Xcode 10.3 or 11
  • Developer ID Certificates (Application and Install)
  • Application Specific Password for your Developer account
  • a Command Line Tool Project that you want to sign and notarize

That’s a longish list. If you are already building command line tools in Xcode, you should have most of these covered already. We will walk through the list step-by-step:

Apple Developer Program Account

You need either the paid membership in the Apple Developer Program or be invited to an Apple Developer Enterprise Program team with access to the proper certificates.

You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.

Xcode

You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.

Developer ID Certificates

There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.

To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.

We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.

If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.

Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:

% security find-identity -p basic -v

This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.

You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"

The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.

Application Specific Password for your Developer Account

Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.

Create a new application specific password in Apple ID portal for your developer account.

You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:

  • Keychain Item Name: Developer-altool
  • Account Name: your developer account email
  • Password: the application-specific password you just created

This will create a developer specific password item that we can access safely from the tools.

If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.

A Command Line Tool Project

You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.

My sample project for this article will be named “hello.”

Preparing the Xcode Project

The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.

Choosing the proper signing certificates

Before you can notarize the command line tool, it needs to be signed with the correct certificates.

  1. in Xcode, select the blue project icon in the left sidebar
  2. select the black “terminal” icon with your project’s name under the “Targets” list entry
  3. make sure the ‘General’ tab is selected
  4. under ‘Signing’ disable ‘Automatically manage signing’
  5. under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
  6. under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Setting the certificates
Setting the certificates

Enable Hardened Runtime

Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.

  1. from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
  2. click on ‘All’ to show all available settings
  3. enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
  4. set the value in the project column (blue icon) to YES
Enable Hardened Runtime
Enable Hardened Runtime

Change the Install Build Location

If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:

  1. in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
  2. double click on the value in the Project column (blue icon), this will open a popup window
  3. change the value to $SRCROOT/build/pkgroot
Change the Installation Build location
Change the Installation Build location

If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)

Build the Binary

You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:

  1. open Terminal and change directory to the project folder
  2. % xcodebuild clean install

This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.

After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.

/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.

Build the pkg

Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.

We can use the pkgroot directory as our payload to build the installer package:

% pkgbuild --root build/pkgroot \
           --identifier "com.example.hello" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Armin Briegel (ABCD123456)" \
           build/hello-1.0.pkg

I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.

This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.

If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”

Notarizing the Installer Package

Xcode has a command line tool altool which you can use to upload your tool for notarization:

xcrun altool --notarize-app \
             --primary-bundle-id "com.example.com" \
             --username "username@example.com" \
             --password "@keychain:Developer-altool" \
             --asc-provider "ABCD123456" \
             --file "build/hello-1.0.pkg"

The username is your developer account email.

The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.

The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)

This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:

xcrun altool --notarization-info "Your-Request-UUID" \
             --username "username@example.com" \                                    
             --password "@keychain:Developer-altool"   

Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.

Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.

Completing the Process

You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.

If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.

You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:

% xcrun stapler staple build/hello-1.0.pkg

You can also use stapler to verify the process went well:

% xcrun stapler validate build/hello-1.0.pkg

But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:

% spctl --assess -vvv --type install build/hello-1.0.pkg

Automating the Process

Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.

The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.

You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.

Links and Videos

These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.

Going forward

Notarization is a key part of Apple’s security strategy going in macOS.

As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.

If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.

On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.

Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.

Swift 5 for MacAdmins

macOS 10.14.4 also includes Swift 5. The main new feature of Swift 5 is that Swift is ABI stable.

Simply said, ABI stability allows swift binaries to use a Swift library on the system instead of having to bundle the libraries with them. This will, of course reduce the size of the binaries.

For example, my desktoppr tool compiles to 6.5MB with Swift 4.2 and 56KB with Swift 5.

However, command line tools built with Xcode 10.2 now rely on the Swift library to be available on the system. macOS 10.14.4 and future versions will include the libraries, but older macOS versions did not. There is no option in Xcode 10.2 to keep the old behavior of bundling the libraries in the tool.

This means that when you re-build a tool in Xcode 10.2 with Swift it will not run on older macOS version:

$ sw_vers -productVersion
10.13.6
$ ./desktoppr 
dyld: Library not loaded: @rpath/libswiftAppKit.dylib
  Referenced from: /Users/armin/Desktop/desktoppr
  Reason: image not found
Abort trap: 6

Apple provides an installer for the Swift libraries for “earlier versions of macOS.” (The package installer declares a minimum OS version of 10.9.)

The libraries are installed in /usr/lib/swift, where your binaries can find them:

$ ./desktoppr 
/Library/Desktop Pictures/BoringBlueDesktop.png

Note: these libraries are used by compiled binaries. Installing the runtime libriaries will not allow you to run swift script files, i.e. text files starting with the #!/usr/bin/swift shebang. You still need to install Xcode or the Developer Command Line tools for that.

So far, only command line tools written and compiled in Xcode 10.2 will require the libraries. Application bundles will continue to include their own libraries.

As Swift gets updated, you will need to update the installed libraries as well. You can get the installed version of the Runtime with pkgutil:

$ pkgutil --info com.apple.pkg.SwiftRuntimeForCommandLineTools
package-id: com.apple.pkg.SwiftRuntimeForCommandLineTools
version: 10.2.0.0.1.1552586384
volume: /
location: /
install-time: 1553789052
groups: com.apple.FindSystemFiles.pkg-group 

However, on a Mac with 10.14.4 the swift libraries are present but not installed by the same installer package, they are part of the entire system:

$ pkgutil --file-info /usr/lib/swift/libswiftFoundation.dylib
volume: /
path: /usr/lib/swift/libswiftFoundation.dylib

pkgid: com.apple.pkg.update.os.10.14.4.18E226
pkg-version: 1.0.0.0.1.1553145155
install-time: 1553765876
uid: 0
gid: 0
mode: 755

If/When Apple updates the runtime libraries, this might be a challenge to track and update properly.

Note: While I used my tool desktoppr as an example, I have not updated the version available for download to Swift 5 yet. So, that is still built with the “old” Swift and Xcode and should work everywhere without the runtime libraries.

Nevertheless, it should be prudent for MacAdmins to install the Swift 5 runtime libraries on their fleet. At the very least be aware that these errors can occur and how to fix them.