Swift Command Line Tools and Argument Parser — Part 1

When building tools in Swift, I usually start with a command line tool. This allows me to ignore the complexity of creating a user interface while figuring out the underlying APIs and data models.

Technically, command line tools have a user interface, as well. They print output to pipes, standard out or standard error for data, progress, status or other information. They should provide information using the exit status. They take input from standard in, pipes, environment variables, or from command line arguments.

While there are many subtleties to consider with all of these, these “interfaces” are still less complex to handle than a full user interface built with AppKit, UIKit or SwiftUI.

Swift provides the functionality to deal with files, outputs, pipes with APIs. This post will not cover those. This post will focus on getting arguments from the command line.

Note: this tutorial was written using Swift 5.10, Xcode 15.4, swift-argument-parser 1.5 on macOS 14.6.1. The details and user interfaces may change with different versions, but the fundamentals should remain.

CommandLine.arguments

The built-in way to get the arguments for your process is with CommandLine.arguments which returns an array of String objects. CommandLine.arguments is quite basic, but can be suitable for simple use cases.

Create a project directory named CLArgs and use swift package init to create a swift package to build an executable:

> mkdir CLArgs
> cd CLArgs
> swift package init --type executable

This will create a skeleton project, you will find a basic “Hello, world” code in Sources/main.swift. Replace the print statement there with:

import Foundation
let arguments = CommandLine.arguments.dropFirst()
guard let name = arguments.first
else {
  print("command requires an argument!")
  exit(1)
}
print("Hello, \(name)")

Note: You can use your favorite text editor or IDE to edit SPM projects. You can also use Xcode. When you run xed . in the Swift package directory, Xcode will open the Swift package in a project view. You can edit, build and run the package in Xcode or use Xcode for editing and build and run it from the command line.

In terminal, build and run the project with

> swift run CLArgs Armin

This tells the swift to build the CLArgs target defined in Package.swift and run it with the argument Armin. You should see this output:

> swift run CLArgs Armin
Building for debugging...
[1/1] Write swift-version-39B54973F684ADAB.txt
Build of product 'CLArgs' complete! (0.11s)
Hello, Armin

Let’s look at the code in detail.

let arguments = CommandLine.arguments.dropFirst()

CommandLine.arguments returns an array of strings. By convention, the first argument (arguments[0]) contains the path to the executable. In most situations, you will not be interested in this first argument. One straightforward way to deal with this is to ‘drop’ the first element of the array right away.

guard let name = arguments.first
else {
  print("command requires an argument!")
  exit(1)
}

We get the first element of the arguments array. Additional arguments are simply ignored. When no arguments are provided, this will return nil and guard statement will trigger, where we print an error message and exit the code with a non-zero value, signaling a failure.

print("Hello, \(name)")

The actual point of this sample code: print a greeting with the name.

In this simplest of examples, we spend a majority of the code on preparing the arguments and verifying that they meet our requirements.

CommandLine.arguments will serve you well for simple needs and quick command line tools. However, you will quickly notice that a robust command line tool needs to verify the existence of certain arguments, whether the value matches certain criteria, and print error messages and usage directions when the arguments don’t match the expectations. Many command line tools also have flags and options with short and long forms that need to be processed.

This turns into a lot of code very quickly.

Swift Argument Parser

Enter Swift Argument Parser. A package that provides “straightforward, type-safe argument parsing for Swift.”

You could modify the Package.swift file in our CLArgs project to import Swift Argument Parser but there is an even easier way to start. Back out of the CLArgs project directory and create a new one:

> cd ..
> mkdir SwiftArg
> cd SwiftArg
> swift package init --type tool

When you inspect the Package.swift file in this new project, you will see that it is already linked to the Swift Argument Parser package. Sources/SwiftArgs.swift contains another “Hello, world” template code, but using Swift Argument Parser.

import ArgumentParser
@main
struct SwiftArgs: ParsableCommand {
  mutating func run() throws {
    print("Hello, world!")
  }
}

The struct here implements the ParsableCommand protocol which allows us to use all the nice functionality from the ArgumentParser library. It is also marked with the @main tag, which tells the compiler to run the main() function in this when the binary is launched. The main() function is implemented by ParsableCommand which, well, parses the arguments and then launches the run() function.

Swift Package Manager vs Xcode projects

You can open and edit Swift Package Manager projects in Xcode with the xed . command. Recent Xcode versions know how to work with SPM projects without needing to create an Xcode project file. Xcode will use the configurations in the Package.swift file. This is useful when you like to work in Xcode, but want the project to remain transferable to other editors or IDEs.

There is a weird quirk. When you build and/or run the project from within Xcode it will use the default Xcode build directory (default is ~/Library/Developer/Xcode/DerivedData/). This is different from the location that the swift build or swift run commands in Terminal use (.build in the package directory). This can lead to longer build times and confusion.

You can also use swift-argument-parser and other packages within Xcode projects. This can be necessary if you are building the command line as a target within a larger Xcode project. Maybe you want to use some of Xcode’s advanced features for managing projects, like build phases and Archives. Or maybe you just prefer working in Xcode.

To create a command line with ArgumentParser in Xcode, create a new Project and select the ‘Command Line Tool’ template for macOS. Once the new project is created, select ‘Add Package Dependencies…’ from the File menu. Locate ‘swift-argument-parser’ in the ‘Apple Swift Packages’ collection or just enter the URL in the search field and click ‘Add Package…’ (twice)

Then, you have to delete the main.swift file from the template and create a new SwiftArgs.swift with this code:

import Foundation
import ArgumentParser
@main
struct SwiftArgs: ParsableCommand {
  mutating func run() throws {
    print("Hello, world!")
  }
}

This is the same as the template code created with the swift package init --type tool from above.

When testing and running the command line tool in Xcode will will want to pass arguments into the binary. You can do so by editing the scheme. Choose Product > Scheme > Edit Scheme… from the menu or click on the target icon (the one with the command line icon in the center of the menu bar) and select Edit Scheme… Make sure you are on the ‘Run’ section in that dialog and select the ‘Arguments’ tab. Here you can add, remove, enable or disable the arguments that Xcode passes into your tool when you run it from Xcode.

Continually changing the arguments in the scheme editor can be tedious. You can also use ‘Show Build Folder in Finder’ from the ‘Product’ menu, open the Products/Debug folder in Terminal by dragging that folder on to the Terminal icon in the Dock and run the built command from there with ./SwiftArgs

Whichever way you prefer to create and work with your project, the rest of this tutorial will work the same way.

Using Swift Argument Parser

Right now, we are just running print("Hello, world!"), which is quite underwhelming. Let’s step this up just a little bit:

@main
struct SwiftArgs: ParsableCommand
  @Argument var name: String
  func run() {
    print("Hello, \(name)")
  }
}

First we create a property called name of type Stringwith the @Argument property wrapper. This tells the ArgumentParser library, that we want this variable filled with an argument from the command line. When the run() function is called, we can “just use” the name property, like any other Swift property.

When you run this, something interesting happens: we get an error!

> swift run SwiftArgs
Error: Missing expected argument '<name>'
USAGE: swift-args <name>
ARGUMENTS:
  <name>
OPTIONS:
  -h, --help              Show help information.

When you check the exit code of the command with echo $? you see it return an error code of 64. This means it was missing arguments or got malformed arguments. As should be good practice for command line tools, our tool did print a help message, describing what it was expecting. Here we see that our SwiftArgs command expects a single argument giving a name.

Run the command again, but with an argument:

> swift run SwiftArgs Armin
Hello, Armin

Now everything works as expected. When our tool launches, ArgumentParser grabs the argument, places it in our name property and executes the run() function in our struct that implements ParsableCommand. Since ArgumentParser errors out with the help message, when an argument is missing or too many arguments are present, we can be certain that the name variable is populated when our code runs.

Command Configuration

There is a small detail that is bugging me, though. The help message generated by ArgumentParser deduced that the name of binary should be swift-args instead of SwiftArgs, but the binary name is SwiftArgs, which is the name of the directory we initialized the project in. This is because of different naming standards for Swift types and command line tools. You can change the name of the executable created in the Package.swift file in line 15 under .executableTarget.

We could change the name to something completely different here, say apdemo for ‘Argument Parser Demo. When you apply that change inPackage.swift` it changes the name of the binary, but the auto-generated help message does not pick that up. It still use the auto-generated name.

> swift run apdemo --help
USAGE: swift-args <name>
ARGUMENTS:
  <name>
OPTIONS:
  -h, --help              Show help information.

(Isn’t it neat that ArgumentParser automatically implements --help and -h flags?)

We could change the name of our struct, which will work in simple situations. But you will have a situation where the struct name will not match what you want for the executable name. There is a way to tell ArgumentParser exactly what we want, though.

Insert this code below the struct SwiftArgs line and above the @Argument:

  static let configuration = CommandConfiguration(
    commandName: "apdemo"
  )

When you now look at the generated help again, the command name matches:

OVERVIEW: apdemo - swift-argument-parser tutorial tool
USAGE: apdemo <name>
ARGUMENTS:
  <name>
OPTIONS:
  -h, --help              Show help information.

There is more information we can provide in the command configuration. Extend the CommandConfiguration initializer like this:

  static let configuration = CommandConfiguration(
    commandName: "apdemo",
    abstract: "apdemo - swift-argument-parser tutorial tool",
    version: "0.1"
  )

and run the command to get the help message again.

OVERVIEW: apdemo - swift-argument-parser tutorial tool
USAGE: apdemo <name>
ARGUMENTS:
  <name>
OPTIONS:
  --version               Show the version.
  -h, --help              Show help information.

The abstract appears as an ‘overview’ above the help message and we now see a new option --version. When you run the tool with that option, you will not be surprised to see the 0.1 provided in the configuration, but it is useful nonetheless.

There are more fields you can provide in the CommandConfiguration: discussion allows you to provide a long form description of the command. usage allows you to override the auto-generated usage text. There are some more that we will explore later.

More Arguments

You can add more @Arguments and they will be filled in order from the arguments provided at the command line. Add another property with the @Argument wrapper:

  @Argument var name: String
  @Argument var age: Int
  func run() {
    print("Hello, \(name)!")
    print("You are \(age) years old.")
  }

When you run the tool without any arguments, you can inspect the updated help message. The usage and arguments area now shows both expected arguments. When you run the tool with a single argument, you get an abbreviated help, showing only the missing argument. When you provide a name and a number as command line arguments, everything works as expected.

But what if you provide two strings?

> swift run apdemo Armin lalala            
Error: The value 'lalala' is invalid for '<age>'
Help:  <age>  
Usage: apdemo <name> <age>
  See 'apdemo --help' for more information.

We declared the age property as an Int, so ArgumentParser expects an integer number for the second argument. When the second argument cannot be parsed into an integer, it shows the error.

Change the type of the age property to a double and run it again with a decimal for the age.

Some Help, please?

name and age might be enough to tell a user of your command line tool what to enter. But I think we should provide a bit more explanation. You can attach a help message to the argument:

  @Argument(help: "a name")
  var name: String
  @Argument(help: "age (integer number)")
  var age: Int

I have broken the property declarations into two lines each for clarity and changed the age back to an Int for simplicity. The help messages will appear next to the argument names in the long and short help messages.

Off to a good start

We have just started to scratch the surface of what swift-argument-parser can do for us. In the next part, we will cover options and flags.

Build a notarized package with a Swift Package Manager executable

One of the most popular articles on this blog is “Notarize a Command Line Tool with notarytool.” In that post I introduced a workflow for Xcode to build and notarize a custom installer package for a command line tool. This workflow also works with apps and other projects that require a customized installer package building workflow. I use it in many of my own projects.

But Xcode is not the only way to build Swift binaries. Especially for command line tools, you can also use Swift Package Manager. This provides a mostly command line based interface to building and organizing your project, which you might prefer if you want to use an IDE that is not Xcode, or have Swift projects that need to run cross-platform.

I also have an older article on building a command line tool with Swift Package Manager. But then, I did not create an installer package or notarize the resulting binary.

Placing the binary in an installer package file is the best way to distribute a binary as you can control where in the file system the binary is installed. Notarizing the pkg file is necessary when you are distributing a command line tool, since it enables installations without scary dialogs or handling quarantine flags.

Also, some of the behavior of Swift Package Manager (SPM) and Xcode have changed since the previous posts. So, this article will introduce an updated workflow using Swift Package Manager tools and how to sign, package and notarize a command line tool for distribution.

Note on nomenclature: Swift Package Manager projects are called ‘packages.’ On macOS, installer files (with the pkg file extension) are also called ‘packages.’ We will be using SPM to build a notarized installation package (a pkg file) from a Swift package project. This is confusing. There is not much I can do about that other than using ‘installer package’ and ‘Swift package project’ to help distinguish.

Prerequisites

I wrote this article using Xcode 14.3.1 and Swift 5.8.1. It should also work with somewhat older or newer versions of Xcode and Swift, but I have not tested any explicitly.

Since I said earlier that using Swift Package Manager allows us to not use Xcode and maybe even build a cross-platform project, you may be wondering why we need Xcode. While we don’t need Xcode for our project, it is one way of installing all the tools we need, most importantly the swift and notarytool binaries. You get those from Developer Command Line tools, as well. We will also see that we can combine Xcode with the command line Swift Package Manager workflow, which I find a very useful setup.

To submit a binary to Apple’s notarization process you will need a Personal or Enterprise Apple Developer account, and access to the Developer ID Application and Developer ID Installer certificates from that account. A free Apple Developer account does not provide those certificates, but they are necessary for notarization

You can follow the instructions in the Xcode article on how to get the certificates and how to configure notarytool with an application specific password. If you had already done this previously you should be able to re-use all of that here. When you reach the ‘Preparing the Xcode Project’ section in that article, you can stop and continue here. Apple also has some documentation on how to configure notarytool.

The sample code we will be using will only work on macOS as it uses CoreFoundation functions. Installer packages and notarization are features of macOS, too, so this is not really a problem here. You can use this workflow to build macOS specific signed binaries and notarized installation pkg files from a cross-platform Swift package project. This will work as long as you keep in mind that the tools to sign, package and notarize only exist and/or work on macOS.

The sample code

We will build the same simple sample tool as in the last article. The prf command (short for ‘pref’ or ‘preference’) reads a default setting’s effective value using the CFPreferencesCopyAppValue function.

The macOS defaults command will read preferences, but only from the user level, or from a specified file path. This ignores one of the main features of macOS’ preferences system as it will not show if a value is being managed by a different preference level, such as the global domain, a file in /Library/Preferences, or (most importantly for MacAdmins) a configuration profile.

You can learn all about preferences and profiles in my book “Property Lists, Preferences and Profiles for Apple Administrators.”

We will build a really simple command line tool, named prf which shows the effective value of a setting, no matter where the value comes from. You could make this tool far more elaborate, but we will keep it simple, since the code is not the actual topic for this article.

We will also be using the Swift Argument Parser package to parse command line arguments and provide a help message. We could build this simple tool without using Argument Parser, but using an external package module is one of the strengths of using Swift Package Manager.

Create the Swift Package project

With all the preparations done, it is time to create our Swift package. We will do all the work in the shell, so open Terminal or your other favorite terminal app and navigate to the directory where you want to create the project.

> cd ~/Projects

Then create a new directory with the name swift-prf. This will contain all the files from the Swift package project. Change directory into that new directory. All following commands will assume this project directory is the current working directory.

> mkdir swift-prf
> cd swift-prf

Then run the swift tool to setup the template structure for our command line tool or ‘executable.’

> swift package init --type executable 
Creating executable package: swift-prf
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/main.swift

You can inspect the hierarchy of files that the init tool created in the Finder (open .) or in your preferred editor or IDE.

.gitignore
Package.swift
Sources
    main.swift

`

You can open this package project in Xcode. In older versions of Xcode you had to run a special swift package command to generate the Xcode project, but now, Xcode can open Swift package projects directly. Use xed (the ‘Xcode text editor invocation tool’) to open the current directory in Xcode.

> xed .

There is a pre-filled .gitignore (which will be hidden in Finder and probably your IDE), a Package.swift, and a Sources directory with a single main.swift inside. If you want to use git (or another version control) system, now is the time to initialize with git init.

Build the project with swift build and/or run it with swift run. Not surprisingly, the template prints Hello, world!.

> swift build
Building for debugging...
[3/3] Linking swift-prf
Build complete! (0.92s)
> swift run  
Building for debugging...
Build complete! (0.11s)
Hello, world!

After building, there will also be a .build directory (also hidden in Finder, unless you toggle the visibility of invisible files using shift-command-.) which contains all the interim files. In the debug folder, you can find the swift-prf executable. You can run it directly:

> .build/debug/swift-prf
Hello, world!

You can clean all the generated pieces from the .build directory with swift package clean. This will leave some empty folders behind but remove all the interim and final products. This means the next build is going to take much longer, but this can be helpful after reconfiguring the Package.swift file or when the compiler gets confused.

Sidenote: when you use Xcode to edit your Swift package project, and choose Build or Run from the Xcode interface, then it will build and run in a different location (~/Library/Developer/Xcode/DerivedData/swift-prf-<random-letters>/Build). You need to be aware of this when you alternate between Xcode and the command line.

Configuring the Package

The Package.swift file contains the configuration for a Swift package project. You can see that the executable package template has a single target named swift-prf that builds from the files in Sources.

To change the name of the executable file, change the value of the name: of the .executableTarget to just prf. There is another name: earlier in the file, that sets the name of the entire project, you can leave that being swift-prf. They do not have to match.

Then build the project in the command line and run it directly:

> swift build
Building for debugging...
[3/3] Linking prf
Build complete! (0.51s)
> .build/debug/prf          
Hello, world!

We want to add the Swift Argument Parser package to our project as a dependency, so we can use its functionality in our code. For that, we will have to add a ‘dependency’ to the project and then to the target, as well. Modify the Package.swift file to match this:

// swift-tools-version: 5.8
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
  name: "swift-prf",
  products: [
    .executable(name: "prf", targets: ["prf"]),
  ],
  dependencies: [
    .package(url: "https://github.com/apple/swift-argument-parser", from: "1.2.0"),
  ],
  targets: [
    .executableTarget(
      name: "prf",
      dependencies: [.product(name: "ArgumentParser", package: "swift-argument-parser")],
      path: "Sources")
  ]
)

This means that our project uses the package available at the given URL, and our target is going to use the specific product (or module or framework) named ArgumentParser from that package. Some packages have several products combined out of several targets.

You can find more information on the format of the Package.swift file in this overview, and the full documentation.

The next time you build after this change, it will download the repository, build and link to toward your executable. That might take a while. The next build should be much faster again. Also, a Package.resolved file will appear in the project. This file caches the current versions of the included packages protecting you from unexpected changes when a package repo dependency updates. You can force Swift Package Manager to update the file with swift package update.

Sprinkle some code

Now that we have the Swift package project prepared, we can add the code to actually do something.

First, let’s keep the ‘Hello, world!’ for a moment, but put it in the right struct to use ArgumentParser. Change main.swift to:

import Foundation
import ArgumentParser
@main
struct PRF: ParsableCommand {
  func run() {
    print("Hello, world!")
  }
}

This should build and run fine from the command line with swift build and swift run. However, when you open this now in Xcode, you will see an error: 'main' attribute cannot be used in a module that contains top-level code

This comes from a long-running issue in Swift. In older versions of Swift it appears on the command line, as well. The work-around is easy though. It only seems to appear when the @main designator is the main.swift file. We can rename our main file to PRF.swift.

You may want to close the Xcode project window before you do this because this can confuse Xcode. If you manage to get Xcode into a confused state where the project in Xcode does not match what is on disk any more, quit Xcode and delete the .swiftpm/xcode directory, which is where Xcode keeps its generated files.

> mv Sources/main.swift Sources/PRF.swift

Now the project should build and run the same with the Swift Package Manager tools and in Xcode.

Now we can add the ‘full’ code for our tool. Keep in mind that the goal of this tutorial is not to learn how to write complex swift code for command line tools, but to learn the infrastructure requires to create and distribute them, so this code is intentionally simple and basic.

import Foundation
import ArgumentParser
@main
struct PRF: ParsableCommand {
  static var configuration = CommandConfiguration(
    commandName: "prf",
    abstract: "read effective preference value",
    version: "1.0"
  )
  @Argument(help: "the preference domain, e.g. 'com.apple.dock'")
  var domain: String
  @Argument(help: "the preference key, e.g. 'orientation'")
  var key: String
  func run() {
    let plist = CFPreferencesCopyAppValue(key as CFString, domain as CFString)
    print(plist?.description ?? "<no value>")
  }
}

When you compare that to the code from the last article, there are a few differences. We are using the @main attribute to designate the main entry point for the code (this was added in Swift 5.3) and I have added some help text to the tool and argument declarations.

When you use Swift Argument Parser, you should study the documentation on adding help to [commands](I have added some help text to the tool and argument declarations. ) and flags, arguments and options. (To be honest, you should read the entire documentation, a lot has changed since the last article.)

When you now run the tool:

> swift run  
Building for debugging...
[3/3] Linking prf
Build complete! (0.54s)
Error: Missing expected argument '<domain>'
OVERVIEW: read effective preference value
USAGE: prf <domain> <key>
ARGUMENTS:
  <domain>                the preference domain, e.g. 'com.apple.dock'
  <key>                   the preference key, e.g. 'orientation'
OPTIONS:
  --version               Show the version.
  -h, --help              Show help information.

We get the help text generated by Swift Argument Parser with the extra information we provided in the code.

If you want to provide the arguments to the swift run you have to add the executable name, as well:

> swift run prf com.apple.dock orientation       
Building for debugging...
Build complete! (0.11s)
left

Or you can run the executable directly from the .build/debug directory. (This will not automatically re-build the command like swift run does.

> .build/debug/prf com.apple.dock orientation
left

Since we provided a version in the CommandConfiguration, ArgumentParser automatically generates a --version option:

> .build/debug/prf --version       
1.0

Now that we have a simple but working tool, we can tackle the main part: we will package and notarize the tool for distribution.

Preparing the binary

When you run swift build or swift run it will compile the tool in a form that is optimized for debugging. This is not the form you want to distribute the binary in. Also, we want to compile the release binary as a ‘universal’ binary, which means it will contain the code for both Intel and Apple silicon, no matter which CPU architecture we are building this on.

The command to build a universal release binary is

> swift build --configuration release --arch arm64 --arch x86_64

When that command is finished, you will find the universal binary file in .build/apple/Products/Release/prf. we can check that it contains the Intel (x86_64) and Apple silicon (arm64) with the lipo tool:

> lipo -info .build/apple/Products/Release/prf
Architectures in the fat file: .build/apple/Products/Release/prf are: x86_64 arm64 

For comparison, the debug version of the binary only contains the platform you are currently on:

> lipo -info .build/debug/prf
Non-fat file: .build/debug/prf is architecture: arm64

Apple’s notarization process requires submitted binaries to fulfill a few restrictions. They need a timestamped signature with a valid Developer ID and have the ‘hardened runtime’ enabled.

Xcode will always sign code it generates, but the swift command line tool does not. We will have to sign it ourselves using the codesign tool. You will need the full name of your “Developer ID Application” certificate for this. (Don’t confuse it with the “Developer ID Installer” certificate, which we will need later.)

You can list the available certs with

> security find-identity -p basic -v

and copy the entire name (including the quotes) of your certificate. Then run codesign:

> codesign --sign "Developer ID Application: Your Name (ABCDEFGHJK)" --options runtime  --timestamp .build/apple/Products/Release/prf

You can verify the code signature with

> codesign --display --verbose .build/apple/Products/Release/prf

Build the installation package

Now that we have prepared the binary for distribution, we can wrap it in an package installer file.

To cover all deployment scenarios, we will create a signed ‘product archive.’ You can watch my MacDevOps presentation “The Encyclopedia of Packages” for all the gory details.

First, create a directory that will contain all the files we want put in the pkg. Then we copy the binary there.

> mkdir .build/pkgroot
> cp .build/apple/Products/Release/prf .build/pkgroot/

Then build a component pkg from the pkgroot:

> pkgbuild --root .build/pkgroot --identifier com.scriptingosx.prf --version 1.0 --install-location /usr/local/bin/ prf.pkg

The --identifier uses the common reverse domain notation. This is what the installer system on macOS uses to determine whether an installation is an upgrade, so you really need to pay attention to keep using the same identifier across different versions of the tool. The --version value should change on every update.

The --install-location determines where the contents of the payload (i.e. the contents of the pkgroot directory) get installed to. /usr/local/bin/ is a useful default for macOS, but you can choose other locations here.

Next, we need to wrap the component pkg inside a distribution package.

> productbuild --package prf.pkg --identifier com.scriptingosx.prf --version 1.0 --sign "Developer ID Installer: Your Name (ABCDEFGHJK)" prf-1.0.pkg

It is important that you use the “Developer ID Installer” certificate here. The --identifier and --version are optional with productbuild but this data required for some (admittedly rare) deployment scenarios, and we want to cover them all.

You can inspect the installer pkg file with a package inspection tool such as the amazing Suspicious Package. The package file should as a signed “Product Archive.”

We don’t need the component pkg anymore, and it’s presence might be confusing, so let’s remove it:

> rm prf.pkg

Note: If you want to learn more about building installation packages, check out my book “Packaging for Apple Administrators”

Notarization

We are nearly there, just two more steps.

It is important to notarize pkgs that will be installed by a user, because otherwise they will get a scary warning that Apple can’t verify the pkg for malicious software.

notarytool submits the installer package to Apple’s Notarization process and returns the results. Use the keychain profile name you set up, following the instructions in the previous article or the instructions from the Apple Developer page.

> xcrun notarytool submit prf-1.0.pkg --keychain-profile notary-example.com --wait

This will print a lot of logging, most of which is self-explanatory. The process might stall at the “waiting” step for a while, depending on how busy Apple’s servers are. You should eventually get status: Accepted.

If you got a different status, or if you are curious, you can get more detail about the process, including rejection reasons, with notarytool log. You will need the ‘Submission ID’ from the submission output:

xcrun notarytool log <submission-uuid> --keychain-profile notary-example.com

As the final step, you should ‘staple’ the notarization ticket to the pkg. This means that the (signed) notarization information is attached to the pkg-file, saving a round trip to Apple’s servers to verify the notarization status when a system evaluates the downloaded installer package file.

xcrun stapler staple prf-1.0.pkg
Processing: /Users/armin/Desktop/swift-prf/prf-1.0.pkg
Processing: /Users/armin/Desktop/swift-prf/prf-1.0.pkg
The staple and validate action worked!

And with that, we have a signed and notarized installation pkg file! You can verify this with spctl:

> spctl --assess --verbose -t install prf-1.0.pkg 
prf-1.0.pkg: accepted
source=Notarized Developer ID

Automation

While it is instructive to do this process manually, it is also quite complex and error-prone. If you have been following this blog for any time, you will know that I don’t stop at detailed step-by-step instructions with explanations.

You can find a script to automate all of these steps here. The enclosing repository includes the entire project (all three files) for your reference.

There is a section at the beginning with variables to modify with the information specific to your environment and project, such as your developer ID information and the name of the credential profile for notarytool. Then there are a few variables, such as the product name, and the installation package identifier.

Run the pkgAndNotarize.sh script from the root of the Swift package project directory.

./pkgAndNotarize.sh

The script creates the installer pkg file in the .build directory. The last line of output is the path to the final, signed, notarized and stapled pkg file.

The script mostly follows the process described above, with a few extras. For example, the script determines the version dynamically by running the tool with the --version option. It also uses the modern compression options I described in this post.

If any of the steps in the script fail, you can determine what exactly failed from the output, error message and error code.

(I belief that this could probably be a makefile, but I have no experience with that (yet). I guess I will need to ‘make’ time for this…)

Conclusion

Apple provides developers and MacAdmins with amazing platforms and tools to build all kinds of powerful apps, tools and automations. But then they don’t really document any of the processes or best practices at all. The amount of searching, guesswork, and frustrating trial and error necessary to piece all of this together for a workflow like this one is quite the shocking condemnation of Apple’s documentation.

There are glimmers of hope. The documentation for the notarization process and notarytool are exemplary.

But they only cover one piece of this puzzle. A developer building a tool has to still figure out how to

  • sign all the binaries properly
  • assemble the binaries and resources necessary into an installation package payload
  • how (and when, and when not) to use pre- and postinstall scripts
  • which kind of package installer to build and which tools to use
  • securely manage the Developer ID certificates (this is especially challenging for developer teams)
  • automate this workflow with Xcode or Swift Package Manager or a CI/CD system

MacAdmins often complain about poorly built installer pkgs, and often for good reasons. But to be fair, there are no best practices and little to no documentation for this from Apple. How are developers supposed to know all of this? Most MacAdmins can define what an installer package should do and not do, but wouldn’t be able to explain to a developer how to build such an installer package, let alone integrate that into their build automations. And most developers don’t even know a MacAdmin to ask about this.

Apple requires that developers create signed and notarized archives for software distribution. And I agree wholeheartedly with their motivations and goals here. But when impose requirements for distribution, you have to make the process of creating the installers the correct way easy, or at least well documented, whether you use Xcode or a different tool set, whether you want to distribute a simple self-contained app, a single command line tool, or a complex suite of tools and resources.

Apple has their work cut out to improve this. Official best practices and sample workflows for installer creation and distribution that consider and respect the requirements of MacAdmins for deployment, have been disgracefully lacking for a long time. The more requirements and security Apple piles on to application and tool distribution, the more desperately they need to provide documentation, best practices and guidance.

Until that happens, you have my paltry scripts.

Swift UI Tutorial – Part 3

The big conferences expect you to provide a topic and a brief description several months in advance. This can be challenging as you have to pick something that you think will still be interesting in eight months or so, even if there is a WWDC and the larger part of the beta phase of a new macOS and iOS version in between. It also has to be able to hold my attention for about eight months, which is not an easy requirement.

I usually try to meet that challenge by choosing something that is complex enough that it has a large likelihood of remaining relevant. For this year’s JNUC, I chose “Use Swift with the Jamf API.” This would very certainly remain relevant, as both Swift and the Jamf Pro API were certain to remain in existence. I was also not expecting too great changes in either and if there were smaller changes—both Swift and the Jamf API update regularly—I was confident I would be able to handle them.

It also covered another goal I have for my presentation: it was something I didn’t really know at the time and wanted to learn more about. I had dabbled with SwiftUI before, most prominently for my 2021 MacSysAdmin Online presentation “Let’s Swift Again” and I had also tried myself with using the Jamf API from Swift. But the new concurrency features of Swift 5.5 looked like they could make the code much more interesting.

My hunch was correct. But, even though the new concurrency features simplified the code, using Swift to retrieve and work with objects from the Jamf API still remained quite a bit more complex than doing so with curl in a shell script. A JNUC session was supposed to fit into 30 minutes.

So, I hatched a crazy plan. To remain in the 30 minutes, I would just superficially introduce the most important concepts in the sessions, and then publish a tutorial and some sample code that would explain the details. Ideally, the tutorial would publish around the same time as JNUC.

Even early in putting together the session slides and sample code, I realized, there is far too much to explain for a single post. So it would become a series of posts. No problem, I have done that before. I suggested the series to Jamf marketing and they were happy to go with it, so I was committed.

I did manage to get the first part done and published in time for JNUC. Then the work that piled up over being away for the conference struck and it took a while to get part two out. Part three was published yesterday. The project is starting to take form and is diving into some really essential, but also exciting features of Swift.

I am polishing part 4 right now and will send it to the great people who run the Jamf blog for editing and more polishing soon-ish. I am working on the sequels, where we finally, actually will get into the SwiftUI part of the tutorial. I expect there to be seven parts in total, though this project may have more surprises for me yet.

I think this worked out well, even though it certainly turned out to be far more complex and far more work than I had originally anticipated. I have certainly learned a lot along the way, so that goal was achieved! I hope you will enjoy this series as much as I did writing it. (Some people like it.)

Next year, maybe I will go for something less complex… maybe…

MDOYVR 22 Talk: The Encyclopedia of macOS Automation

Last week I had the pleasure and honor of participating and presenting at MacDevOps YVR. The videos for the sessions are now appearing on YouTube.

There is a page for my talk “The Encyclopedia of macOS Automation,” in which I discuss the options for scripting and automation on macOS, with extra links and notes. You can go directly to the video here.

The talks this year were graphic recorded by the amazing Ashton Rodenhiser (website, twitter). The graphic at the top of this post was made by her while I was presenting.

As always, I had a lot of fun at this conference. Many thanks to the organizers and all the other speakers. Until next year!

Scripting macOS, part 7: Download and Install Firefox

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Follow this blog or the Twitter account for updates on the book’s progress!

Download and Install Firefox

To further illustrate the progress from the idea of a workflow to a working script, let us look at another, more involved example.

To download and install the latest version of Firefox a user has to go to the Firefox website and download the latest version, which will come as a disk image (dmg) file. Then the user needs locate the dmg in the ~/Downloads folder and open it to mount the virtual disk image. Finally, they need to copy the Firefox application from the virtual disk to the Applications folder.

When we want to automate the task ‘Download and Install Firefox,’ we have the following steps:

  • download latest Firefox disk image
  • mount downloaded disk image
  • copy Firefox application to /Applications
  • unmount disk image

From this list of steps, we can build the first ‘frame’ of our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

This breaks the workflow into smaller pieces, that we will now tackle individually.

Download from the Command Line

You can use the curl command to download data in the command line. The curl command is very complex and has many options. We will only discuss the few options that we require for our task here. As always, you can find a detailed description of the curl command and its options in the curl man page.

The URI to download the latest Firefox is
https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US

However, when you try to curl this URI, you only get the following:

> curl "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US"
<a href="https://download-installer.cdn.mozilla.net/pub/firefox/releases/86.0.1/mac/en-US/Firefox%2086.0.1.dmg">Found</a>.

This is a re-direction, that is commonly used to have a single URI, that is redirected to different final URIs, so that when the software updates, the same URI always returns the latest version.

We can tell curl to follow these redirections with the --location option.

By default, the curl command will output the download to standard out. To save the download to a file, we can use the --output option with a file name.

> curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" --output Firefox.dmg

This command will download the latest Firefox disk image to a file named Firefox.dmg in your current working directory. We can use this as our first step:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

Note: Like many other command line tools, curl has short and long options. The short options for –location and –output are -L and -o.
Short options are convenient in the interactive shell, as they save typing and reduce the potential for typos. But they are much less readable, and you usually have to look up their function in the documentation. For that reason, I recommend using the long, descriptive options in scripts.

Working with Disk Images

The command line tool to work with disk image (dmg) files on macOS is hdiutil. This is also a very powerful command with many verbs and options. You can find all the detail in the hdiutil man page.

To mount a disk image, use the attach verb:

> hdituil attach Firefox.dmg

This will output some information and mount the virtual disk. The last line ends with the path to the mounted virtual disk /Volumes/Firefox.

By default, you can see the mounted volume in Finder. We do not really need the disk image to appear in Finder while the script is running. We can suppress this behavior with the -nobrowse option.

Since we are only going to read from the disk image, we can tell hdiutil to mount the dmg in readonly mode with the -readonly option. This speeds things up a bit.

> hdiutil attach Firefox.dmg -nobrowse -readonly

You can unmount or eject the virtual disk with

> hdiutil detach -force /Volumes/Firefox

The -force option will unmount the disk image, even when another process is still using it.

Thehdiutil command covers two of our steps, so we can fill them in:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications

# unmount disk image
hdiutil detach /Volumes/Firefox -force

Copying the Application

When you manually install Firefox the disk image shows you a nice graphic that reminds you to drag the app to the Applications folder. Once the disk image is mounted, the cp command can be used to do this in the shell:

> cp -R /Volumes/Firefox/Firefox.app /Applications/

This provides the last missing step in our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications
echo "copying Firefox to /Applications"
cp -R /Volumes/Firefox/Firefox.app /Applications/

# unmount disk image
hdiutil detach /Volumes/Firefox/ -force

You can now test the script. If Firefox is running, you want to quit it before you run the script. You may also want to delete the existing copy of Firefox from the Applications folder, to be sure that your script is doing the work.

Lists of Commands—Conclusion

We have been able to automate a fairly complex workflow with a script of four commands.

To be perfectly honest, this script (as well as all the others we have built so far) is not complete yet.

A ‘proper’ script needs to be able to react to errors that occur. In our example, imagine the download fails. The script should be able to detect the failure before it overwrites the installed, functional Firefox application.

We will get to this kind of error handling later.

Nevertheless, this script is already useful in its current form. You can try to adapt this script to work with some other software you can download as a disk image.

You can also add extra commands that

  • delete the downloaded disk image at the end
  • open the newly installed Firefox app after installation
  • quit or kill the Firefox process before copying the new version

In the book “Scripting macOS”, you will learn more scripting techniques, and we will re-visit some of these sample scripts and keep improving them.

Follow this blog or the Twitter account for updates on the book’s progress!

Note: After using different variations of these kinds of workflows, I did put together a more generic script to download and install various kinds of software, called ‘Installomator.’ You can see the script at its open source repository on GitHub.

Download Full Installer

A while back I wrote up a blog post on deploying the Install macOS Big Sur application. As one of the solutions, I posted a script (based on Greg Neagle’s installinstallmacos.py) which listed the pkgs from Apple’s software update catalogs so you could download them.

During and after WWDC, I wanted to see if I could build a SwiftUI app. I thought that building a user interface for this task would be a nice practice project.

Ironically, since I want the app to work on Big Sur, I could not use any of the new Swift and SwiftUI features Apple introduced this year. Even so, since I had not used SwiftUI to build a Big Sur application, most of the features Apple introduced last year were still new to me.

It was often unexpected to me which parts turned out to be challenging and which parts were really easy to implement. For example, implementing a preferences window, turned out to be super-easy, but it took me two false-starts to find the correct approach. Communicating with the preferences system of macOS is also very easy, but so poorly documented that you are always second guessing if what you are doing is right.

Apple’s documentation for Swift and SwiftUI on this has definite highlights, but is very sparse overall. I am still not sure if some of the decisions I made while putting this together were “good” choices.

Nevertheless, it works! I think it might be a nice tool to have, so I put it on GitHub. You can just download the app from the release page and use it, or clone the repo and take a look at the code.

Constructive feedback is always welcome! I am still learning this as I go along, too.

Installomator Updated: v0.5

It has been a while, mainly because I was busy with other things, but there finally is a new release version of Installomator!

The reason work has progressed—quite significantly—even though I was distracted is that Søren Theilgaard and Isaac Ordonez have joined the project as conributors. All of the work from 0.4 to 0.5 was from one of them. We ahve some great plans to move this tool forward, as well.

Many of these new app labels have been provided from others, either through GitHub issues, pull requests, or through comments in the #installomator channel on MacAdmins Slack. Thanks to all who contributed.

What’s new in v0.5:

  • Major update and now with help from @Theile and @Isaac
  • Added additional BLOCKING_PROCESS_ACTION handlings
  • Added additional NOTIFY=all. Usuful if used in Self Service, as the user will be notified before download, before install as well as when it is done.
  • Added variable LOGO for icons in dialogs, use LOGO=appstore (or jamf or mosyleb or mosylem or addigy). It’s also possible to set it to a direct path to a specific icon. Default is appstore.
  • Added variable INSTALL that can be set to INSTALL=force if software needs to be installed even though latest version is already installed (it will be a reinstall).
  • Version control now included. The variable appNewVersion in a label can be used to tell what the latest version from the web is. If this is not given, version checking is done after download.
  • For a label that only installs a pkg without an app in it, a variable packageID can be used for version checking.
  • Labels now sorted alphabetically, except for the Microsoft ones (that are at the end of the list). A bunch of new labels added, and lots of them have either been changed or improved (with appNewVersion og packageID).
  • If an app is asked to be closed down, it will now be opened again after the update.
  • If your MDM cannot call a script with parameters, the label can be set in the top of the script.
  • If your MDM is not Jamf Pro, and you need the script to be installed locally on your managed machines, then take a look at Theiles fork. This fork can be called from the MDM using a small script.
  • Script buildCaseStatement.sh to help with creating labels have been improved.
  • Fixed a bug in a variable name that prevented updateTool to be used
  • added type variable for value "updateronly" if the label should only run an updater tool.

And if you are counting, there are now more than 260 application labels in Installomator. However, that number is a bit inflated, because several vendors have multiple downloads for Intel and Apple Silicon apps.

Get the script and find the instructions on the GitHub repo.

If you have any feedback or questions, please join us in the #installomator channel on MacAdmins Slack.

Thanks again to all those who contributed!

(Installomator Icon credit: Mischa van der Bent)

EraseInstall application retired

It makes me very sad that the EraseInstall application has been retired.

We built this tool three years ago, mostly because we wanted to learn how to build an app like this on macOS. We think it worked out well. We learned a lot, and are glad the application was useful to some.

Since then, all the people involved in the EraseInstall Project have moved on to other jobs or other responsibilities. Unfortunately, this leaves us with no time or resources to maintain or improve EraseInstall.

The repository and the application will remain available in its current state. There will be no more updates. If someone feels they can take up the project and continue it, please do!

If you are looking for a similar solution, we recommend Graham Pugh’s eraseinstall script.

Thank you, all, again!

Team EraseInstall: Mischa van der Bent, Arnold Nefkens, and Armin Briegel

Wrangling Pythons

As I noted in my last Weekly News Summary, several open source projects for MacAdmins have completed their transition to Python 3. AutoPkg, JSSImport and outset announced Python 3 compatible versions last week and Munki already had the first Python 3 version last December.

Why?

Apple has included a version of Python 2 with Mac OS X since 10.2 (Jaguar). Python 3.0 was released in 2008 and it was not fully backwards compatible with Python 2. For this reason, Python 2 was maintained and updated alongside Python 3 for a long time. Python 2 was finally sunset on January 1, 2020. Nevertheless, presumably because of the compatibility issues, Apple has always pre-installed Python 2 with macOS and still does so in macOS 10.15 Catalina. With the announcement of Catalina, Apple also announced that in a “future version of macOS” there will be no pre-installed Python of any version.

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. (macOS 10.15 Catalina Release Notes)

This also applies to Perl and Ruby runtimes and other libraries. I will be focussing on Python because it is used more commonly for MacAdmin tools, but most of this post will apply equally to Perl and Ruby. Just mentally replace “Python” for your preferred language.

The final recommendation is what AutoPkg and Munki are following: they are bundling their own Python runtime.

How to get Python

There is a second bullet in the Catalina release notes, though:

Use of Python 2.7 isn’t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS won’t include Python 2.7. Instead, it’s recommended that you run python3 from within Terminal. (51097165)

This is great, right? Apple says there is a built-in Python 3! And it’s pre-installed? Just move all your scripts to Python 3 and you’ll be fine!

Unfortunately, not quite. The python3 binary does exist on a ‘clean’ macOS, but it is only a stub tool, that will prompt a user to download and install the Command Line Developer Tools (aka “Developer Command Line Tools” or “Command Line Tools for Xcode”). This is common for many tools that Apple considers to be of little interest to ‘normal,’ non-developer users. Another common example is git.

Dialog prompting to install the Command Line Tools
Dialog prompting to install the Command Line Tools

When you install Xcode, you will also get all the Command Line Developer Tools, including python3 and git. This is useful for developers, who may want to use Python scripts for build operation, or for individuals who just want to ‘play around’ or experiment with Python locally. For MacAdmins, it adds the extra burden of installing and maintaining either the Command Line Developer Tools or the full Xcode install.

Python Versions, a multitude of Snakes

After installing Xcode or the Command Line Developer Tools, you can check the version of python installed: (versions on macOS 10.15.3 with Xcode 11.3.1)

> python --version    
Python 2.7.16
> python3 --version    
Python 3.7.3

When you go on the download page for Python.org, you will get Python 3.8.1 (as of this writing). But, on that download page, you will also find download links for “specific versions” which include (as of this writing) versions 3.8.1, 3.7.6, 3.6.10, 3.5.9, and the deprecated 2.7.17.

The thing is, that Python isn’t merely split into two major release versions, which aren’t fully compatible with each other, but there are several minor versions of Python 3, which aren’t fully compatible with each other, but are still being maintained in parallel.

Developers (individuals, teams, and organisations) that use Python will often hold on to a specific minor (and sometimes even patch) version for a project to avoid issues and bugs that might appear when changing the run-time.

When you install the latest version of Munki, it will install a copy of the Python framework in /usr/local/munki/ and create a symbolic link to that python binary at /usr/local/munki/python. You can check its version as well:

 % /usr/local/munki/python --version
Python 3.7.4

All the Python code files for Munki will have a shebang (the first line in the code file) of

#!/usr/local/munki/python

This ensures that Munki code files use this particular instance of Python and no other copy of Python that may have been installed on the system.

The latest version of AutoPkg has a similar approach:

> /usr/local/autopkg/python --version    
Python 3.7.5

In both cases the python binary is a symbolic link. This allows the developer to change the symbolic link to point to a different Python framework. The shebangs in the all the code files point to the symbolic link, which can be changed to point to a different Python framework.

This is useful for testing and debugging. Could MacAdmins use this to point both tools to the same Python framework? Should they?

The Bridge to macOS

On top of all these different versions of Python itself, many scripts, apps, and tools written in Python rely on ‘Python modules.’ These are libraries (or frameworks) of code for a certain task, that can be downloaded and included with a Python installation to extend the functionality of Python.

The most relevant of these modules for MacAdmins is the “Python Objective-C Bridge.” This module allows Python code to access and use the native macOS Cocoa and CoreFoundation Frameworks. This not only allows for macOS native GUI applications to be written in Python (e.g. AutoDMG and Munki’s Managed Software Center [update: MSC was re-written in Swift last year]), but also allows short scripts to access system functions. This is sometimes necessary to get a data that matches what macOS applications “see” rather than what the raw unix tools see.

For example, the defaults tool can be used to read the value of property lists on disk. But those might not necessarily reflect the actual preference value an application sees, because that value might be controlled by a different plist file or configuration profile.

(Shameless self-promotion) Learn more about Property lists, Preferences and Profiles

You could build a tool with Swift or Objective-C that uses the proper frameworks to get the “real” preference value. Or you can use Python with the Objective-C bridge:

#!/usr/bin/python
from Foundation import CFPreferencesCopyAppValue
print CFPreferencesCopyAppValue("idleTime", "com.apple.screensaver")

Three simple lines of Python code. This will work with the pre-installed Python 2.7, because Apple also pre-installs the Python Objective-C bridge with that. When you try this with the Developer Tools python3 you get an error:

ModuleNotFoundError: No module named 'Foundation'

This is because the Developer Tools do not include the Objective-C bridge in the installation. You could easily add it with:

> sudo python3 -m pip install pyobjc

But again, while this command is “easy” enough for a single user on a single Mac, it is just the beginning of a Minoan labyrinth of management troubles.

Developers and MacAdmins, have to care about the version of the Python they install, as well as the list of modules and their versions, for each Python version.

It is as if the Medusa head kept growing more smaller snakes for every snake you cut off.

(Ok, I will ease off with Greek mythology metaphors.)

You can get a list of modules included with the AutoPkg and the Munki project with:

> /usr/local/munki/python -m pip list
> /usr/local/autopkg/python -m pip list

You will see that not only do Munki and AutoPkg include different versions of Python, but also a different list of modules. While Munki and AutoPkg share many modules, their versions might still differ.

Snake Herding Solutions

Apple’s advice in the Catalina Release Notes is good advice:

It’s recommended that you bundle the runtime within the app.

Rather than the MacAdmin managing a single version of Python and all the modules for every possible solution, each tool or application should provide its own copy of Python and its required modules.

If you want to build your own Python bundle installer, you can use this script from Greg Neagle.

This might seem wasteful. A full Python 3 Framework uses about 80MB of disk space, plus some extra for the modules. But it is the safest way to ensure that the tool or application gets the correct version of Python and all the modules. Anything else will quickly turn into a management nightmare.

This is the approach that Munki and AutoPkg have chosen. But what about smaller, single script solutions? For example simple Python scripts like quickpkg or prefs-tool?

Should I bundle my own Python framework with quickpkg or prefs-tool? I think that would be overkill and I am not planning to do that. I think the solution that Joseph Chilcote chose for the outset tool is a better approach for less complex Python scripts.

In this case, the project is written to run with Python 3 and generic enough to not require a specific version or extra modules. An admin who wants to use this script or tool, can change the shebang (the first line in the script) to point to either the Developer Tool python3, the python3 from the standard Python 3 installer or a custom Python version, such as the Munki python. A MacAdmin would have to ensure that the python binary in the shebang is present on the Mac when the tool runs.

You can also choose to provide your organization’s own copy Python with your chosen set of modules for all your management Python scripts and automations. You could build this with the relocatable Python tool and place it in a well-known location the clients. When updates for the Python run-time or modules are required, you can build and push them with your management system. (Thanks to Nathaniel Strauss for pointing out this needed clarifying.)

When you build such scripts and tools, it is important to document which Python versions (and module versions) you have tested the tool with.

(I still have to do that for my Python tools.)

What about /usr/bin/env python?

The env command will determine the path to the python binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the python binary is unknown.

This is useful when developers want to use the same script in different environments across different computers, user accounts, and platforms. However, this renders the actual version of python that will interpret the script completely unpredictable.

Not only is it impossible to predict which version of Python will interpret a script, but you cannot depend on any modules being installed (or their versions) either.

For MacAdmin management scripts and tools, a tighter control is necessary. You should use fixed, absolute paths in the shebang.

Conclusion

Managing Python runtimes might seem like a hopeless sisyphean task. I believe Apple made the right choice to not pre-install Python any more. Whatever version and pre-selection of module versions Apple would have chosen, it would only have been the correct combination for a few Python solutions and developers.

While it may seem wasteful to have a multitude of copies of the Python frameworks distributed through out the system, it is the easiest and most manageable solution to ensure that each tool or application works with the expected combination of run-time and modules.