Building Simple Component Packages

Building packages can turn complex very quickly. We will start with a few simple examples first and build up from there.

You should read the first two posts in this series, first:

Boring Wallpaper

For our first simple example, imagine your organization insists that every Mac use the same wallpaper (or desktop picture). The first step for an admin is to provide the image file on each Mac.

Apple changed the naming for the background image in macOS Ventura from ‘desktop picture’ to ‘wallpaper’ in macOS Sonoma to match the naming across all their platforms. As we will see, the old name still appears in some places.

Technically, an image file used as a wallpaper image could be stored anywhere on the disk. When you place image files in /Library/Desktop Pictures/ (still the old name) a user will see them as a choice among the pictures in the ‘Wallpaper’ pane in System Settings. They will have to scroll all the way to the right in the ‘Dynamic Wallpapers’ or ‘Pictures’ section but the images you add in that folder will appear there.

The /Library/Desktop Pictures folder does not exist on a ‘clean’ installation of macOS, but the installation package will create it for us.

All the resources for building the package have to be provided in a certain folder structure. It is easiest to contain all of that in a single project folder.

I will provide the instructions to create and build the projects in the command line as they are easier to represent than descriptions of how to achieve something in the graphical user interface. It does not really matter whether you create a folder from Terminal or in Finder. Do what you are comfortable with. However, Getting comfortable with command line tools in Terminal is an important step towards automated package creation workflows.

My book “macOS Terminal and Shell” can teach you the fundamentals. Buy it on Ko-Fi or Apple Books!

I often find it helpful to have a Finder window for the current working directory open next to Terminal. You can open the current working directory in Finder with the open . command.

Create a folder for the project named BoringWallpaper in a location of your choosing.

> mkdir BoringWallpaper
> cd BoringWallpaper

You can download the sample image file I will be using here. Of course you can use another wallpaper image of your choice.

The BoringWallpaper folder will be our project folder. Create another folder inside BoringWallpaper named payload. Inside the payload folder, we will re-create the path to where we want the file be installed in the file system, in this case /Library/Desktop Pictures.

Since the Desktop Pictures folder name contains a space, we need to quote the path in the shell and scripts. You can also escape the space with a backslash \. The effect will be the same. In general, I recommend using tab-completion in the command line for file paths which will take care of special characters.

We can create the entire folder structure at once with mkdir -p:

> mkdir -p "payload/Library/Desktop Pictures"

Then copy the first desktop picture BoringBlueDesktop.png into payload.

> cp /path/to/BoringBlueDesktop.png "payload/Library/Desktop Pictures"

The payload folder will gather all the files we want the package to install.

Your folder structure inside the BoringWallpaper project folder should now look like this:

📁 payload
    📁 Library
        📁 Desktop Pictures
            📄 BoringBlueDesktop.png

The payload folder represents the root of the target volume during installation.This will usually be the root of the system volume /. We recreated the folder structure where we want the file to be installed in the file system. The installer will create intermediate folders that do not yet exist during installation.

In this example, the /Library folder will already exist, but the Desktop Pictures subfolder will not yet exist on a clean system, so it will be created and the image file will be placed inside.

When you run the installer on a system where the Desktop Pictures subfolder already exists, the image will be placed inside that. Should a file with the same name already exist in that location, it will be overwritten. Other files that might be in that folder will generally not be affected.

Introducing pkgbuild

macOS provides the pkgbuild command line tool to create installer package components. Make sure your current working directory is BoringWallpaper and run

> pkgbuild --root payload --install-location / --identifier com.example.BoringWallpaper --version 1 BoringWallpaper.pkg

The --root option designates the root of our payload, so we pass the payload folder.

The identifier should be a string modeled on the reverse DNS of your organization, e.g. com.scriptingosx.itservices.BoringWallpaper. The exact form of the identifier does not really matter as long as it is unique.

We will use com.example.BoringWallpaper for the identifier. For the exercise, you can use those or replace them with your own. When you build packages for production or deployment they should use your organization’s reverse DNS format.

This should create a BoringWallpaper.pkg file in your project folder.

You should now inspect the resulting pkg file with the tools from earlier:

> lsbom $(pkgutil --bom BoringWallpaper.pkg )
.    40755   0/0
./Library    40755   0/0
./Library/Desktop Pictures    40755   0/0
./Library/Desktop Pictures/BoringBlueDesktop.png    100644  0/0 17393342618871431

(You may see a ._BoringBlueDesktop.png file appear here. That is a resource fork file. Preview.app sometimes creates these for file previews. You can safely ignore those.)

There are two relevant things to notice here: the payload contains the intermediate folders /Library and /Library/Desktop Pictures, which means they will be created, should they not exist on the system yet. This is generally what we want to happen, but a good thing to keep in mind.

Also notice that pkgbuild set the owner and group ID for the folders and the image file to 0/0 or root:wheel. This is the default ownership for files installed by packages, which ensures that non-admin users cannot change, delete or overwrite the files. This is a useful default, but there are options to have more granular control.

pkgbuild will always preserve the file mode or access privileges. When you change the file mode in of the file in the payload folder before you run pkgbuild, the command line tool will use that for the payload and Bom. In this case, the 644 or r-wr--r-- file mode works quite well, but for a test, let’s change the mode to 444 (removing the write access for the owner) and re-run the pkgbuild command.

> chmod u-w payload/BoringBlueDesktop.png 
> pkgbuild --root payload --install-location / --identifier com.example.BoringWallpaper --version 1 BoringWallpaper.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to BoringWallpaper.pkg
> lsbom $(pkgutil --bom BoringWallpaper.pkg )
.    40755   0/0
./Library    40755   0/0
./Library/Desktop Pictures    40755   0/0
./Library/Desktop Pictures/BoringBlueDesktop.png    100644  0/0 17393342618871431

Note that running the pkgbuild command again overwrote the previously generated pkg file with warning. This is generally not a problem, but something you need to be aware of.

We will want to change the image file in the following steps, so add the writable flag back:

> chmod u+w "payload/Library/Desktop Pictures/BoringBlueDesktop.png"

Handling extended attributes

In recent versions of macOS, pkgbuild will preserve extended file attributes in the payload.

This is a change in behavior to earlier versions of macOS, where you had to use the undocumented --preserve-xattr option to preserve extended attributes.

Most extended attributes contain metadata for Finder and Spotlight. For example, when you open the image file in Preview, you will get a com.apple.lastuseddate#PS extended attribute. You can use the -@ option of the ls command to see extended attributes:

> open "payload/Library/Desktop Pictures/BoringBlueDesktop.png"
> ls -l@ "payload/Library/Desktop Pictures"
total 3400
-rw-r--r--@ 1 armin  staff  1739334 Aug  1 14:03 BoringBlueDesktop.png
    com.apple.lastuseddate#PS        16 

You generally do not want to have extended attributes be part of your package payload. This is especially true of quarantine flags!

There are some exceptions. For example, signed shell scripts store the signature information in an extended attribute. In these cases you will have to carefully build your package creation workflow to ensure only the desired extended attributes are preserved in the package and installed to the target file system.

You can remove extended attributes recursively with the xattr command:

> xattr -cr payload

Then rebuild the package:

> pkgbuild --root payload --install-location / --identifier com.example.BoringWallpaper --version 1 BoringWallpaper.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to BoringWallpaper.pkg

Creating a Build Script

The command line tools to create installer package files have a large number of options. Even in our simple example, pkgbuild requires several options and arguments. Each one needs to be entered correctly so the installer process does the right thing. An error in the identifier or a version number will result in unexpected behavior that may be very hard to track down. In addition, there are steps like running xattr -c that need to be performed before creating the package.

To avoid errors and simplify the process, we will create a shell script which runs the required commands with the correct options. The script will always repeat the commands with the proper arguments in the correct order, reducing the potential for errors. Updating the version is as simple as changing the variable in the script.

In your favored text editor create a file named buildBoringWallpaperPkg.sh and with the following code:

#!/bin/sh

pkgname="BoringWallpaper"
version="1.0"
install_location="/"
identifier="com.example.${pkgname}"

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

projectfolder=$(dirname "$0")
payloadfolder="${projectfolder}/payload"

# recursively clear all extended attributes
xattr -cr "${payloadfolder}"

# build the component
pkgbuild --root "${payloadfolder}" \
         --identifier "${identifier}" \
         --version "${version}" \
         --install-location "${install_location}" \
         "${projectfolder}/${pkgname}-${version}.pkg"

The script first stores all the required pieces of information in variables. This way you can quickly find and change data in one place and do not have to search through the entire script.

The identifier variable is composed from our com.example reverse-DNS prefix and the pkgname variable set earlier.

Then the script sets the shell PATH variable, which is always a prudent step.

In the next line, the script determines the folder enclosing the script, by reading the $0 argument which contains the path to the script itself and applying the dirname command which will return the enclosing folder. This way we can use the projectfolder variable later to write the resulting pkg file into a fixed location (the project folder), instead of the current working directory.

Finally the pkgbuild command is assembled from all the variables.

The backslash \ in a shell script allows a command to continue in the next line. Instead of a command in a single very long line we can have one line per argument. This makes the script easier to read and update.

In Terminal, set the script file’s executable bit with

> chmod +x buildBoringWallpaperPkg.sh

Delete the original BoringWallpaper.pkg and run the build script.

> rm BoringWallpaper.pkg
> ./buildBoringWallpaperPkg.sh
pkgbuild: Inferring bundle components from contents of ./payload
pkgbuild: Wrote package to ./BoringWallpaper-1.0.pkg 

This will create a new package file named BoringWallpaper-1.0.pkg.

Now you do not have to remember every option exactly but instead can run ./buildBoringDesktopPkg.sh. If you need to change options like the version number, it is easy to do by changing a variable. Package creation is easy to repeat and if you use a version control system (e.g. git) changes to the script are tracked.

You have literally codified the package creation process. If you are working in a team, you can point to the script and say: “This is how we create this package!” and everyone who has access to the script can recreate the package creation workflow. They can also read the script and understand what is going to happen. A script like this does not replace the need for documentation, but is better than no documentation at all.

To be precise, the build script and the files and folder structure in the payload are required together for the package creation workflow. They should be kept and archived together.

Ideally together with documentation describing:

  • motivation/reason for building this package
  • where and how the package is intended to be used
  • whether the software installed requires certain configurations that are not provided by the pkg, such as licenses or default settings through a script or a configuration profile and where and how to obtain and set those
  • macOS and platforms (Intel or Apple silicon) the package was built on
  • macOS versions and platforms the package was tested on
  • where to obtain the resources in the payload, should they get lost or need to be updated
  • person(s) or team responsible for this package project
  • version history or change log
  • an archive of older versions of the pkg file
  • uninstall process or script
  • any other relevant links and history, for example problems and issues that lead to certain design choices

For developers, scripts like this can be part of an automated release or CI/CD workflow.

You do not have to include the version number in the package name, but it helps in many situations. It also helps when you build an archive or history of installers. You never know when you will need an older version of an installer. When vendors/developers provide the version number in their file names, it helps admins and users identify new or outdated versions.

You should (again) inspect the new package file you just created in Suspicious Package and using pkgutil.

Testing the Package

Finally, you can install this package on your test machine.

For this simple package, you can also use the Mac you are currently working on. With more complex packages, especially when we get into installation scripts and LaunchD configuration, a virtual machine or separate testing device is strongly recommended.

When you run the package in the Installer application, note that the dialogs are the defaults and very terse. System administrators will rarely build packages that are meant to be installed with the user interface, so this is not a problem. Most administrative package files will be installed by management systems in the background and never show the UI.

Developers can customize the user interface for installer packages with distribution files, we will get to in a future post.

You can also use the installer command to install the package:

> sudo installer -pkg BoringWallpaper-1.0.pkg -tgt / -verbose
installer: Package name is BoringWallpaper-1.0
installer: Installing at base path /
installer: The install was successful.

After installing, go to /Library/Desktop Pictures and look for the BoringBlueDesktop.png image file, then open System Settings and go “Wallpaper.” You will have to scroll down to the “Pictures” section and all the way to the right, but the picture will appear there!

You can also open Terminal and run the pkgutil commands to inspect what was installed: (replace com.example.* with your identifier)

> pkgutil --pkgs="com.example.*"         
com.example.BoringWallpaper

> pkgutil --info com.example.BoringWallpaper
package-id: com.example.BoringWallpaper
version: 1.0
volume: /
location: 
install-time: 1754058568

> pkgutil --files com.example.BoringWallpaper
BoringBlueDesktop.png

> pkgutil --file-info /Library/Desktop\ Pictures/BoringBlueDesktop.png                        
volume: /
path: /Library/Desktop Pictures/BoringBlueDesktop.png

pkgid: com.example.BoringWallpaper
pkg-version: 1.0
install-time: 1754058568
uid: 0
gid: 0
mode: 100644

If you created an installer package that attempted to install the image file in /System/Library/Desktop Pictures, it would fail. This directory is protected by two technologies, System Integrity Protection and the read-only sealed System volume. Figuring out the proper location to install management files is an important, but sometimes complicated task.

Removing the installed files

If you tested this package on your main Mac, you can remove the installed files with the following commands:

> sudo rm /Library/Desktop\ Pictures/BoringBlueDesktop.png
> sudo rmdir /Library/Desktop\ Pictures
> sudo pkgutil --forget com.example.BoringWallpaper

Note the rmdir command will error when there are other files remaining in it. This is intentional here, since we only want to remove the folder if it is empty and not remove files other than those we installed. The /Library folder is part of the base macOS installation, so we are not going to touch it.

Similar to the build script, it can be useful to maintain the un-install script while developing the package, especially for more complex installs. An un-install script for this project might look like this:

#!/bin/sh

# uninstall Boring Wallpaper

# reverts the installation of com.example.BoringWallpaper


# remove the file
rm -vf "/Library/Desktop Pictures/BoringBlueDesktop.png"

# remove folder
# fails when there are other files remaining inside
# we do not want to affect files we did not install
rmdir -v "/Library/Desktop Pictures"

# forget the pkg receipt
pkgutil --forget com.example.BoringWallpaper

Since the installed file and folder are owned by root, you need to run the entire script with root privileges or sudo:

> sudo ./uninstallBoringWallpaper.sh

Note that many device management services offer the option to run scripts on the Mac clients and they generally run the scripts with root privileges. Consult your management service’s documentation for details.

You have to remember to update the uninstall script when you change the payload and other settings in the package. This will be useful for your testing.

Developers or vendors can also provide the uninstall script to customers in case they need to uninstall the software. Administrators could use the uninstall script in a self service portal to allow end users to remove software they no longer require.

Note that software might create other files while it is running that are not part of the installer package. Use your judgment whether they need to be removed as part of the uninstall script. Some files might contain user data that should be preserved, even when the software is deleted.

Installing Packages

What is a package?

If you have been using a Mac, you will have encountered an installation package file or pkg.

Package files are used to install software and configuration files on your Mac.

Package files come in different flavors and formats, but they can be summarized to these relevant components:

  • a payload
  • installation scripts

A package file may have only a payload, only scripts, or both.

The payload is an archive of all the files that the package will install on the system. The package also contains a “bill of materials” (BOM) which lists where each file should be installed and what the file privileges or mode should be.

Installation scripts can be executed before and after the payload is installed.

Additionally, packages contain some metadata, which provides extra information about the package and its contents. They can also contain images and text files, such as license agreements that can customize the default Installer app interface.

Installer application

The most common way of installing a package file and start its installation process, is to open it with a double-click. This opens the default application for the pkg extension: Installer. The Installer app can be found in /System/Library/CoreServices/. However, you rarely need to open it directly. It is usually started indirectly by opening a package file.

After launch, the Installer app presents different panels to the user. The exact order and content of the panels depends on what the developer of the package configured. At the very least, it will show:

  • a short introduction
  • prompt to authenticate for administrative privileges
  • a progress bar of the installation
  • whether the installation succeeded or failed

A package may also show:

  • a custom background image
  • a detailed introduction
  • a license agreement that needs to be accepted
  • alternative installation location
  • options to select certain subsets of files and apps (components)
  • more custom steps implemented by the developer

Installer app can also list the contents of a package file without installing it first. You can choose ‘Show Files…’ (⌘I) from the ‘File’ menu to get a list of files in the payload. This list can be very extensive and there is a search field to filter the list.

If you want to see more than just the progress bar during the installation, you can select “Installer Log” from the “Window” menu (⌘L) and then increase the output to “Show all Errors and Logs” (⌘3).

You can also review the installation log afterwards, by opening the Console application from Utilities and choosing “install.log” under “Log Reports.” You can also look at /var/log/install.log directly.

This log is notoriously verbose. There will be many entries for each installation and some related system services like software update will log here, too.

Security

Packages can place files and executables in privileged locations in the file system. When you open a package file with the installer application, it will usually prompt for administrator privileges.

In the early days of Mac OS X, package installers had no limitations at all. However, these days, digital security and privacy are important features and criteria for platforms and Apple has implemented several features in macOS which set strong limits on what third-party package installers can do.

Most importantly, System Integrity Protection (SIP, introduced in OS X Yosemite 10.11) and the Signed System Volume (introduced in macOS Catalina 10.15) prevent third-party packages from affecting system files.

Even with these protections in place, packages still provide many options for abuse. Packages are very useful to install legitimate software but can also be used to install and persist malicious tools.

File Quarantine

File quarantine does not directly limit the abilities of package installers, but it is important to understand how it works together with other security features in macOS.

Quarantine flags were introduced in Mac OS X as early as version 10.4 (Tiger) but were not actually used for much until later in 10.5 (Leopard).

When a file arrives on your Mac from an external source, such as a download in a web browser, an email attachment, or an external drive, the process that downloads or copies generally adds a quarantine flag. The quarantine flag is added in an extended attribute to the file.

Note: the examples here reference desktoppr-0.5-218.pkg, the installer for a open source command line tool I wrote. You can download its installer pkg from the GitHub repository. The version and build number might be different.

After the download, you can see some of the metadata added to the download the Finder info window. Select the downloaded pkg file and choose ‘Get Info’ (⌘I) from the context or ‘File’ Menu. In the Info window that appears, you need to expand the ‘More Info’ section, where you will see the URL it was downloaded from. There might be more than one URL if the browser was redirected.

You can get more information in the command line using the xattr command line tool: (xattr stands for ‘extended attribute’, it is often pronounced like ‘shatter’)

> xattr desktoppr-0.5-218.pkg         
com.apple.metadata:kMDItemDownloadedDate
com.apple.metadata:kMDItemWhereFroms
com.apple.quarantine

The quarantine flag has the attribute name com.apple.quarantine. You can also use xattr to show the contents of the extended attribute:

> xattr -p com.apple.quarantine desktoppr-0.5-218.pkg
0083;6842fcbe;Safari;A2964F09-ACDF-430F-8CFF-48BD75C464CD

The contents are (in order, separated by semi-colons):

  • a hexadecimal flag number: always 0083
  • a hexadecimal timestamp: e.g. 6842fcbe
  • the process that created the file: e.g. Safari
  • a universal identifier (UUID)

You can convert the hexadecimal timestamp into a human readable time with

> date -jf %s $((16#6842fcbe)) 
Fri Jun  6 16:35:42 CEST 2025

There are two more extended attributes attached to the downloaded file that are interesting. The awkwardly named com.apple.metadata:kMDItemDownloadedDate and com.apple.metadata:kMDItemWhereFroms contain the download date and the web addresses the file was downloaded from respectively. When you look at them withxattr, you see the data is stored in a binary property list format.

> xattr -p com.apple.metadata:kMDItemDownloadedDate desktoppr-0.5-218.pkg
bplist00?3A????r

To show this in a human readable format, you have to pipe the output through xxd and plutil:

> xattr -x -p com.apple.metadata:kMDItemDownloadedDate desktoppr-0.5-218.pkg | xxd -r -p | plutil -p -
[
  0 => 2025-06-06 14:35:42 +0000
]

The name of the extended attribute gives us a hint that the information is also accessible through the file’s metadata, which is a bit easier:

> mdls -name kMDItemDownloadedDate desktoppr-0.5-218.pkg
kMDItemDownloadedDate = (
    "2025-06-06 14:35:42 +0000"
)

The named kMDItemWhereFroms attribute contains the URLs the file was downloaded from. It might be more than one URL because of redirections.

The URLs for GitHub downloads tend to be very long. After manually downloading Firefox, I see these URLs:

> mdls -name kMDItemWhereFroms Firefox\ 139.0.1.dmg 
kMDItemWhereFroms = (
    "https://download-installer.cdn.mozilla.net/pub/firefox/releases/139.0.1/mac/en-US/Firefox%20139.0.1.dmg",
    "https://www.mozilla.org/"
)

It is important to remember that not all processes add quarantine flags to downloaded or copied files. As a general rule, applications with a user interface add quarantine files and command line tools and background processes do not. But there may be exceptions either way.

For example, when you download a file using the curl command line tool, it will have no quarantine flag or other metadata extended attributes. This is might be exploited by malicious software.

The quarantine flag serves as a signal to the system that this file or archive needs to be checked before opening or running. The part of the system that performs this check is called Gatekeeper.

Gatekeeper

Gatekeeper was introduced to macOS in OS X Lion (10.7) in 2011. Gatekeeper will verify the integrity, signature and notarization status of an app or executable before opening it.

A package installer file can be:

  • not signed at all
  • signed, but not notarized
  • signed and notarized

Gatekeeper works somewhat differently for installer packages compared to applications that you download in disk image (dmg) or other archive formats (e.g. zip). When you copy an application out of a disk image into the Applications folder or unarchive it from an archive, the system transfers the quarantine flag and metadata to the application bundle on the local disk. When you then open the app for the first time, the presence of the quarantine flag signals Gatekeeper to verify the integrity of the application using the signature and verify the notarization status. You can see the dialogs the system might present in this Apple support document.

The signature verifies the integrity and the source of an application or installer package. With an intact signature, you can be certain the package file has not been modified since it was signed. If the signature uses an Apple Developer ID, you can be fairly certain that the package was signed by that developer, or someone from that organization. There have been cases where Apple Developer ID certificates were stolen, but Apple will usually invalidate those fairly quickly.

Notarization is an extra step where the developer uploads the signed package file to Apple’s notarization servers. Apple then scans the package for certain security features and whether known malware signatures are present. Apple then adds the package file’s hash to their notarization database. The developer can also ‘staple a ticket’ to the package file, so that Gatekeeper doesn’t have to reach out to Apple’s servers on the check.

When the Gatekeeper verification of the signature and notarization status succeeds, the user gets a prompt to confirm that they want to launch the application they just downloaded.

When either verification fails, most commonly because the app is not notarized, the user gets a different, quite scary, prompt, stating that the system cannot verify the package is free of malware. You will get the same dialog when the package is not signed at all.

While generally, all software developers should sign and notarize their packages, this is not always the case. Open source projects, often have neither the financial nor logistical means to obtain an Apple Developer account, which provides the required Apple Developer ID certificates.

Bypassing Gatekeeper

Before you choose to install a package file which is not validly signed or notarized, you should first be certain the source is trustworthy. Then you should probably inspect the package using the tools in this post to verify that it only installs what it is supposed to, before actually installing it.

After attempting to open the pkg file with the Installer app by double-clicking and receiving the dialog that it could not be verified, click ‘Done.’ Then navigate to the ‘Privacy & Security’ pane in System Settings. Under the ‘Security’ section, you will see a message that the package ‘was blocked to protect your Mac’ with a button to ‘Open Anyway.’

This extra option will disappear after a few minutes.

You can also remove the quarantine flag using the command line.

> xattr -d com.apple.quarantine desktoppr-0.1.pkg

Without the quarantine flag, the Gatekeeper verification will not be triggered when the file is opened.

Packages installed by a device management service are not checked by Gatekeeper and do not need to be notarized. With some services, the packages may need to be signed, but not necessarily with an Apple Developer ID. Consult the documentation of your device management service for details.

Installer command line tool

You can also install package files from the command line using the installer tool.

To install a package file on the current system volume, use the installer command like this

> sudo installer -package desktoppr-0.5-218.pkg -target /

There are shorter flags for both these options:

> sudo installer -pkg desktoppr-0.5-218.pkg -tgt /

Installing a package with the installer command will not enforce a restart or logout, whether the package requires one or not. You will have to perform or schedule the reboot manually.

Installing with installer will also not trigger GateKeeper checks, whether the quarantine flag is set or not.

The -verbose flag will increase the output of the installer tool which can help when you need to analyze problems. The installer process will also log to /var/log/install.log so you can also monitor or review the installation log in the Console app.

New tool updated: utiluti v1.1

A few weeks back I introduced a new tool: utiluti, which sets default apps for url schemes and file types.

Not standing still, I have updated the tool with new verbs to list url schemes and file types that a given app can handle, as well as inspect and set default apps set for specific files.

You can see the Readme and download the latest release on the GitHub repo.

New tool: utiluti sets default apps

A while back I wrote a post on the Jamf Tech Thoughts blog about managing the default browser on macOS. In that post I introduced a script using JXA to set the default application for a given url scheme. (like http, mailto, ssh etc.) The beauty of using JXA/osascript is that it doesn’t require the installation of an extra tool.

However, there was a follow-up comment asking about default apps for file types, i.e. which app will open PDF files or files with the .sh file extension. Unfortunately, Apple has not bridged those AppKit APIs to AppleScript/JXA, which means it is not possible to use them in a script without dependencies.

Back then, I started working on a command line tool which uses those APIs. I didn’t really plan to publish it, since there were established tools, like duti, cdef and SwiftDefaultApp that provided the functionality. It was a chance to experiment and learn more about Swift Argument Parser. Then life and work happened and other projects required more attention.

A recent discussion on the Mac Admins Slack reminded me of this. Also, none of the above mentioned tools have been updated in the past years. As far as I can tell, none of them have been compiled for the Apple silicon platform. They don’t provide installation pkgs either, which complicates their distribution in a managed deployment.

So, I dusted off the project, cleaned it up a bit, and added a ReadMe file and a signed and notarized installation pkg. The tool is called utiluti (I am a bit proud of that name).

You can use utiluti to set the default app for an url scheme:

$ utiluti url set mailto com.microsoft.Outlook
set com.microsoft.Outlook for mailto

or to set the default app to open a uniform type identifier (UTI):

$ utiluti type set public.plain-text com.barebones.bbedit
set com.barebones.bbedit for public.plain-text

There are bunch of other options, you can read the details in the ReadMe or in the command line with utiluti help.

The functionality is quite basic, but please provide feedback if there are features you’d like to have added.

Swift Command Line Tools and Argument Parser — Part 1

When building tools in Swift, I usually start with a command line tool. This allows me to ignore the complexity of creating a user interface while figuring out the underlying APIs and data models.

Technically, command line tools have a user interface, as well. They print output to pipes, standard out or standard error for data, progress, status or other information. They should provide information using the exit status. They take input from standard in, pipes, environment variables, or from command line arguments.

While there are many subtleties to consider with all of these, these “interfaces” are still less complex to handle than a full user interface built with AppKit, UIKit or SwiftUI.

Swift provides the functionality to deal with files, outputs, pipes with APIs. This post will not cover those. This post will focus on getting arguments from the command line.

Note: this tutorial was written using Swift 5.10, Xcode 15.4, swift-argument-parser 1.5 on macOS 14.6.1. The details and user interfaces may change with different versions, but the fundamentals should remain.

CommandLine.arguments

The built-in way to get the arguments for your process is with CommandLine.arguments which returns an array of String objects. CommandLine.arguments is quite basic, but can be suitable for simple use cases.

Create a project directory named CLArgs and use swift package init to create a swift package to build an executable:

> mkdir CLArgs
> cd CLArgs
> swift package init --type executable

This will create a skeleton project, you will find a basic “Hello, world” code in Sources/main.swift. Replace the print statement there with:

import Foundation
let arguments = CommandLine.arguments.dropFirst()
guard let name = arguments.first
else {
  print("command requires an argument!")
  exit(1)
}
print("Hello, \(name)")

Note: You can use your favorite text editor or IDE to edit SPM projects. You can also use Xcode. When you run xed . in the Swift package directory, Xcode will open the Swift package in a project view. You can edit, build and run the package in Xcode or use Xcode for editing and build and run it from the command line.

In terminal, build and run the project with

> swift run CLArgs Armin

This tells the swift to build the CLArgs target defined in Package.swift and run it with the argument Armin. You should see this output:

> swift run CLArgs Armin
Building for debugging...
[1/1] Write swift-version-39B54973F684ADAB.txt
Build of product 'CLArgs' complete! (0.11s)
Hello, Armin

Let’s look at the code in detail.

let arguments = CommandLine.arguments.dropFirst()

CommandLine.arguments returns an array of strings. By convention, the first argument (arguments[0]) contains the path to the executable. In most situations, you will not be interested in this first argument. One straightforward way to deal with this is to ‘drop’ the first element of the array right away.

guard let name = arguments.first
else {
  print("command requires an argument!")
  exit(1)
}

We get the first element of the arguments array. Additional arguments are simply ignored. When no arguments are provided, this will return nil and guard statement will trigger, where we print an error message and exit the code with a non-zero value, signaling a failure.

print("Hello, \(name)")

The actual point of this sample code: print a greeting with the name.

In this simplest of examples, we spend a majority of the code on preparing the arguments and verifying that they meet our requirements.

CommandLine.arguments will serve you well for simple needs and quick command line tools. However, you will quickly notice that a robust command line tool needs to verify the existence of certain arguments, whether the value matches certain criteria, and print error messages and usage directions when the arguments don’t match the expectations. Many command line tools also have flags and options with short and long forms that need to be processed.

This turns into a lot of code very quickly.

Swift Argument Parser

Enter Swift Argument Parser. A package that provides “straightforward, type-safe argument parsing for Swift.”

You could modify the Package.swift file in our CLArgs project to import Swift Argument Parser but there is an even easier way to start. Back out of the CLArgs project directory and create a new one:

> cd ..
> mkdir SwiftArg
> cd SwiftArg
> swift package init --type tool

When you inspect the Package.swift file in this new project, you will see that it is already linked to the Swift Argument Parser package. Sources/SwiftArgs.swift contains another “Hello, world” template code, but using Swift Argument Parser.

import ArgumentParser
@main
struct SwiftArgs: ParsableCommand {
  mutating func run() throws {
    print("Hello, world!")
  }
}

The struct here implements the ParsableCommand protocol which allows us to use all the nice functionality from the ArgumentParser library. It is also marked with the @main tag, which tells the compiler to run the main() function in this when the binary is launched. The main() function is implemented by ParsableCommand which, well, parses the arguments and then launches the run() function.

Swift Package Manager vs Xcode projects

You can open and edit Swift Package Manager projects in Xcode with the xed . command. Recent Xcode versions know how to work with SPM projects without needing to create an Xcode project file. Xcode will use the configurations in the Package.swift file. This is useful when you like to work in Xcode, but want the project to remain transferable to other editors or IDEs.

There is a weird quirk. When you build and/or run the project from within Xcode it will use the default Xcode build directory (default is ~/Library/Developer/Xcode/DerivedData/). This is different from the location that the swift build or swift run commands in Terminal use (.build in the package directory). This can lead to longer build times and confusion.

You can also use swift-argument-parser and other packages within Xcode projects. This can be necessary if you are building the command line as a target within a larger Xcode project. Maybe you want to use some of Xcode’s advanced features for managing projects, like build phases and Archives. Or maybe you just prefer working in Xcode.

To create a command line with ArgumentParser in Xcode, create a new Project and select the ‘Command Line Tool’ template for macOS. Once the new project is created, select ‘Add Package Dependencies…’ from the File menu. Locate ‘swift-argument-parser’ in the ‘Apple Swift Packages’ collection or just enter the URL in the search field and click ‘Add Package…’ (twice)

Then, you have to delete the main.swift file from the template and create a new SwiftArgs.swift with this code:

import Foundation
import ArgumentParser
@main
struct SwiftArgs: ParsableCommand {
  mutating func run() throws {
    print("Hello, world!")
  }
}

This is the same as the template code created with the swift package init --type tool from above.

When testing and running the command line tool in Xcode will will want to pass arguments into the binary. You can do so by editing the scheme. Choose Product > Scheme > Edit Scheme… from the menu or click on the target icon (the one with the command line icon in the center of the menu bar) and select Edit Scheme… Make sure you are on the ‘Run’ section in that dialog and select the ‘Arguments’ tab. Here you can add, remove, enable or disable the arguments that Xcode passes into your tool when you run it from Xcode.

Continually changing the arguments in the scheme editor can be tedious. You can also use ‘Show Build Folder in Finder’ from the ‘Product’ menu, open the Products/Debug folder in Terminal by dragging that folder on to the Terminal icon in the Dock and run the built command from there with ./SwiftArgs

Whichever way you prefer to create and work with your project, the rest of this tutorial will work the same way.

Using Swift Argument Parser

Right now, we are just running print("Hello, world!"), which is quite underwhelming. Let’s step this up just a little bit:

@main
struct SwiftArgs: ParsableCommand
  @Argument var name: String
  func run() {
    print("Hello, \(name)")
  }
}

First we create a property called name of type Stringwith the @Argument property wrapper. This tells the ArgumentParser library, that we want this variable filled with an argument from the command line. When the run() function is called, we can “just use” the name property, like any other Swift property.

When you run this, something interesting happens: we get an error!

> swift run SwiftArgs
Error: Missing expected argument '<name>'
USAGE: swift-args <name>
ARGUMENTS:
  <name>
OPTIONS:
  -h, --help              Show help information.

When you check the exit code of the command with echo $? you see it return an error code of 64. This means it was missing arguments or got malformed arguments. As should be good practice for command line tools, our tool did print a help message, describing what it was expecting. Here we see that our SwiftArgs command expects a single argument giving a name.

Run the command again, but with an argument:

> swift run SwiftArgs Armin
Hello, Armin

Now everything works as expected. When our tool launches, ArgumentParser grabs the argument, places it in our name property and executes the run() function in our struct that implements ParsableCommand. Since ArgumentParser errors out with the help message, when an argument is missing or too many arguments are present, we can be certain that the name variable is populated when our code runs.

Command Configuration

There is a small detail that is bugging me, though. The help message generated by ArgumentParser deduced that the name of binary should be swift-args instead of SwiftArgs, but the binary name is SwiftArgs, which is the name of the directory we initialized the project in. This is because of different naming standards for Swift types and command line tools. You can change the name of the executable created in the Package.swift file in line 15 under .executableTarget.

We could change the name to something completely different here, say apdemo for ‘Argument Parser Demo. When you apply that change inPackage.swift` it changes the name of the binary, but the auto-generated help message does not pick that up. It still use the auto-generated name.

> swift run apdemo --help
USAGE: swift-args <name>
ARGUMENTS:
  <name>
OPTIONS:
  -h, --help              Show help information.

(Isn’t it neat that ArgumentParser automatically implements --help and -h flags?)

We could change the name of our struct, which will work in simple situations. But you will have a situation where the struct name will not match what you want for the executable name. There is a way to tell ArgumentParser exactly what we want, though.

Insert this code below the struct SwiftArgs line and above the @Argument:

  static let configuration = CommandConfiguration(
    commandName: "apdemo"
  )

When you now look at the generated help again, the command name matches:

OVERVIEW: apdemo - swift-argument-parser tutorial tool
USAGE: apdemo <name>
ARGUMENTS:
  <name>
OPTIONS:
  -h, --help              Show help information.

There is more information we can provide in the command configuration. Extend the CommandConfiguration initializer like this:

  static let configuration = CommandConfiguration(
    commandName: "apdemo",
    abstract: "apdemo - swift-argument-parser tutorial tool",
    version: "0.1"
  )

and run the command to get the help message again.

OVERVIEW: apdemo - swift-argument-parser tutorial tool
USAGE: apdemo <name>
ARGUMENTS:
  <name>
OPTIONS:
  --version               Show the version.
  -h, --help              Show help information.

The abstract appears as an ‘overview’ above the help message and we now see a new option --version. When you run the tool with that option, you will not be surprised to see the 0.1 provided in the configuration, but it is useful nonetheless.

There are more fields you can provide in the CommandConfiguration: discussion allows you to provide a long form description of the command. usage allows you to override the auto-generated usage text. There are some more that we will explore later.

More Arguments

You can add more @Arguments and they will be filled in order from the arguments provided at the command line. Add another property with the @Argument wrapper:

  @Argument var name: String
  @Argument var age: Int
  func run() {
    print("Hello, \(name)!")
    print("You are \(age) years old.")
  }

When you run the tool without any arguments, you can inspect the updated help message. The usage and arguments area now shows both expected arguments. When you run the tool with a single argument, you get an abbreviated help, showing only the missing argument. When you provide a name and a number as command line arguments, everything works as expected.

But what if you provide two strings?

> swift run apdemo Armin lalala            
Error: The value 'lalala' is invalid for '<age>'
Help:  <age>  
Usage: apdemo <name> <age>
  See 'apdemo --help' for more information.

We declared the age property as an Int, so ArgumentParser expects an integer number for the second argument. When the second argument cannot be parsed into an integer, it shows the error.

Change the type of the age property to a double and run it again with a decimal for the age.

Some Help, please?

name and age might be enough to tell a user of your command line tool what to enter. But I think we should provide a bit more explanation. You can attach a help message to the argument:

  @Argument(help: "a name")
  var name: String
  @Argument(help: "age (integer number)")
  var age: Int

I have broken the property declarations into two lines each for clarity and changed the age back to an Int for simplicity. The help messages will appear next to the argument names in the long and short help messages.

Off to a good start

We have just started to scratch the surface of what swift-argument-parser can do for us. In the next part, we will cover options and flags.

Building a LaunchD Installer pkg for desktoppr (and other tools)

In my last post introducing desktoppr 0.5, I said that a Mac admin could build a customized pkg with a customized LaunchAgent property list, that would run desktoppr manage at login and every few hours in the background to re-set the wallpaper when the configuration profile is updated.

I admit I glossed over the process of building a package that installs and launches a LaunchAgent (or LaunchDaemon). Since that is not a trivial configuration, I will describe the details in this post.

LaunchAgent? LaunchDaemon? LaunchD?

On macOS, launchd is the system-wide process that controls and launches every other process running on the system. When you look at the hierarchical process list in Activity Monitor, you will always see launchd running with the process ID 1 as the single child process of kernel_task and all other processes are a child of launchd.

More practically, users and admins can use launchd to control processes that need to be available at all times or be launched in certain intervals or at certain events.

launchd has two types of processes: LaunchDaemons and LaunchAgents. Since the terms daemons (or demons) and agents do not have clear definitions outside of launchd, a bit of explanation is required.

LaunchDaemons

LaunchDaemons are processes that run with system privileges (“as root”). Processes that belong to your management system and security suites may need to run periodically or on certain events in the background and require root level access to the system to do their jobs. Other processes or services need to listen for incoming network traffic on privileged ports.

Note that background process are often called ‘agents’ regardless of whether they run with root privileges or as a user. The launchd terminology is more precise here.

Apple has done a lot of work over the past few year to provide Endpoint Security or Network Extension frameworks that mitigate the need for third party root daemons, but there are still relevant use cases, especially when you are a managing Macs.

A LaunchDaemon is configured by providing a property list file in /Library/LaunchDaemons. This configuration plist will be automatically loaded at reboot, or can be loaded using the launchctl command.

LaunchAgents

LaunchAgents are processes that run with user privileges. Generally, if something needs to read or write to anything in the user home folder, it should be a LaunchAgent. Over the recent years, Apple has put restrictions on what data process can access in user space, which has made LaunchAgents more of a pain to manage, but they still have their use cases.

A LaunchAgent is configured by providing a property list file in /Library/LaunchAgents on the system level or in ~/Library/LaunchAgents in a user’s home folder. Configuration property lists that are in a user home folder, will only run for that user, while LaunchAgents that are configured in the system wide /Library/LaunchAgents will be loaded for all users on the system.

A LaunchAgent configuration plist will be automatically loaded at login, or can be loaded using the launchctl command.

Since desktoppr affects a user setting (the desktop/wallpaper), it has to run as the user, so we need to build and install a LaunchAgent. Creating and installing a LaunchDaemon is pretty much the same process, though.

Ingredients

To build an installation package to install and run a LaunchAgent or LaunchDaemon, you need three ingredients:

  • the binary or script which will be launched
  • a launchd configuration property list
  • installation scripts to unload and load the configuration

For desktoppr, we already have the binary (downloadable as a zip from the project release page) and a sample configuration property list file.

The sample launchd configuration looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.scriptingosx.desktopprmanage</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/desktoppr</string>
        <string>manage</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>StartInterval</key>
    <integer>10800</integer>
</dict>
</plist>

The label is required and is used to uniquely identify the LaunchAgent or Daemon with launchd and the launchctl command. Reverse domain notification is common. The filename of the configuration property list should match the label.

The ProgramArguments key provides an array with the command and arguments that will be launched. The first item in the array should provide the full path to the binary or script. Then you provide arguments, one string item per argument. Since the individual arguments are separated by the string tags, you do not need to escape spaces or other special characters in the arguments.

It is worth noting that launchd will launch the binary or script directly, so syntax and substitutions that are done in shells will not work. This includes variable and command substitution like $HOME, $USER, or $(date) as well as pipes and output redirection. You get a single command with a list of static arguments. (You can use the StandardOutPath and StandardErrorPath keys in the launchd config plist to redirect output.)

For the desktoppr LaunchAgent, that suits us just fine, as we are only passing the manage argument which tells desktoppr to get the details from a configuration profile (sample here).

The next key, RunAtLoad tells launchd, to run the binary with the arguments immediately when the configuration file is loaded. Since configuration files in /Library/LaunchAgents and ~/Library/LaunchAgents are automatically loaded when a user logs in, this generally means desktoppr manage will be run when a user logs in, which suits us well.

The last key StartInterval tells launchd to re-launch the process after a certain time (given in seconds, the 10800 seconds in our sample file translate to three hours). Should the system be sleeping at the time, it will not run the LaunchAgent at that time, or when the system wakes up, but wait until the next interval period comes around.

There are other keys that control when the process gets launched, such as StartCalendarInterval or WatchPath. You can read details in the man page for launchd.plist or on the excellent launchd.info page. There are also apps, like Peter Borg’s Lingon that provide a user interface for creating these plist files.

Move or copy the plist file into the correct folder. The plist files in /Library/LaunchDaemons and /Library/LaunchAgents must be owned by root. They must not be writable by group or other. The file mode 644 (rw-r--r--) is recommended. Similarly, the binary or script needs to be owned by root and cannot be writable by group or other. The file mode 755 (rwxr-xr-x) is recommended. Wrong file privileges will result in a Load/Bootstrap failed: 5: Input/output error.

Make sure that the binary or script does not have a quarantine flag attached. You can check with ls -al@ or xattr -p com.apple.quarantine /path/to/file and remove a quarantine flag with xattr -d com.apple.quarantine /path/to/file. When the binary or script has the quarantine flag, the configuration file will load fine but the actual execution will fail quietly.

Loading the LaunchAgent or LaunchDaemon

The easiest way to load a LaunchDaemon is to restart the Mac. The easiest way to load a LaunchAgent is to log out and login. This is usually not practical.

The launchctl command manages LaunchDaemons and LaunchAgents and we can use this to load LaunchAgents and LaunchDaemons on demand.

To load our desktoppr agent, use

> launchctl load /Library/LaunchAgents/com.scriptingosx.desktopprmanage.plist

People are already readying their pitchforks here. “Hold on,” they say. “Apple has marked load and unload as ‘legacy.’ You shouldn’t use them!”

While it is true that load and unload (and a bunch of other commands) are labeled as ‘legacy’ in the launchctl man page, this does not mean they are deprecated and should be avoided. The difference between the legacy and the ‘new’ commands is that the legacy commands pick up whether a process should be run in user or root context (the ‘target domain’) from the context that launchctl runs in, whereas the new bootstrap and bootout commands need the target domain to be stated explicitly.

This makes the modern commands more precise, but often more wordy to use. You will see that the legacy commands are simpler to use in the interactive terminal, while the ‘modern’ commands are more useful in scripts. The equivalent ‘modern’ command is:

> launchctl bootstrap gui/501 /Library/LaunchAgents/com.scriptingosx.desktopprmanage.plist

where 501 is the user ID for your user account. This is 501 by default for the first user on on a macOS system, but you should verify what your account’s user ID actually is with id -u.

See what I mean with “more wordy to use”? That doesn’t mean they are bad or should be avoided, but that we can be picky about when to use which format. You could argue that using the modern commands throughout would be more consistent, but you still find the ‘legacy’ command is lots of documentation, so I feel a Mac admin needs to be aware of both.

Note: the ‘modern’ commands were introduced in OS X Yosemite 10.10 in 2014.

To load a LaunchDaemon, you need to run launchctl with root privileges. In the interactive shell, that means with sudo:

> sudo launchctl load /Library/LaunchDaemons/com.example.daemon.plist

or

> sudo launchctl bootstrap system/ /Library/LaunchDaemons/com.example.daemon.plist

To stop a LaunchAgent from being launched going forward, unload the configuration:

> launchctl unload /Library/LaunchAgents/com.scriptingosx.desktopprmanage.plist

or

> launchctl bootout gui/501/com.scriptingosx.desktopprmanage

The bootout command uses the label instead of the file path.

Putting it all together in an installer

Note: I will be showing how to build and installer package for our LaunchAgent using command line tools. These instructions should have all the information you need, even if you prefer using apps such as Whitebox Packages or Composer.

If you are not comfortable with using Terminal on macOS yet, please consider my book “macOS Terminal and Shell.” If you want to learn more about building installation packages, consider my book “Packaging for Apple Administrators.”

Download the zip file for desktoppr from the releases page. Download the sample launchd property list file and modify the launch criteria (StartInterval or StartCalendarInterval) to your requirements.

Open Terminal, change directory to a location where you want to create the project directory and create a project directory:

> mkdir DesktopprManagePkg
> cd DesktopprManagePkg

Then create a payload and a scripts directory in this project folder:

> mkdir payload
> mkdir scripts

Unzip the desktoppr binary from the downloaded zip archive into the right folder hierarchy in the payload folder:

> mkdir -p payload/usr/local/bin/
> ditto -x -k ~/Downloads/desktoppr-0.5-218.zip payload/usr/local/bin/

Remove the quarantine flag from the expanded desktoppr binary and test it by getting the version and the current wallpaper/desktop picture:

> xattr -d com.apple.quarantine payload/usr/local/bin/desktoppr
> payload/usr/local/bin/desktoppr version
0.5
> payload/usr/local/bin/desktoppr
/System/Library/CoreServices/DefaultDesktop.heic

Create the /Library/LaunchAgents directory in the payload and copy or move the LaunchAgent configuration plist there:

> mkdir -p payload/Library/LaunchAgents/
> cp ~/Downloads/com.scriptingosx.desktopprmanage.plist payload/Library/LaunchAgents/

When the installation package runs, the files in the payload will be moved to the respective locations on the target drive. The pkgbuild tool will set the owner of the files to root when building the package, but we should verify that the file mode is correct:

> stat -f %Sp payload/usr/local/bin/desktoppr
-rwxr-xr-x
> stat -f %Sp payload/Library/LaunchAgents/com.scriptingosx.desktopprmanage.plist
-rw-r--r--

If the privileges don’t match the desired mode, set them with chmod

> chmod 755 payload/usr/local/bin/desktoppr
> chmod 644 payload/Library/LaunchAgents/com.scriptingosx.desktopprmanage.plist

and run the stat commands from above again to verify.

If you built the installer package now, it will install the files in the right locations and the LaunchAgent will load and run desktoppr manage on next login. This will set the desktop/wallpaper according to the information in the configuration profile. You should install and update the configuration profile using your management system.

Loading the LaunchAgent or LaunchDaemon

However, as I said earlier, requiring or waiting for a logout or reboot is not ideal, we would like to have the LaunchAgent load immediately when the package is installed. We can achieve this by adding a postinstall script to the installation package. This script will be executed after the payload files have been installed.

With your favored text editor, create a file named postinstall (no file extension) with this content in the scripts sub directory.

#!/bin/sh

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

label="com.scriptingosx.desktopprmanage"
launchdPlist="/Library/LaunchAgents/${label}.plist"

# do not load when target volume ($3) is not current startup volume
if [ "$3" != "/" ]; then
    echo "not installing on startup volume, exiting"
    exit 0
fi

# for LaunchAgent, check if user is logged in
# see https://scriptingosx.com/2019/09/get-current-user-in-shell-scripts-on-macos/

currentUser=$(echo "show State:/Users/ConsoleUser" | scutil | awk '/Name :/ { print $3 }')

# don't load at loginwindow 
if [ "$currentUser" = "loginwindow" ] || [ "$currentUser" = "_mbsetupuser" ]; then
    echo "no user logged in, exiting"
    exit 0
fi

uid=$(id -u "$currentUser")

echo "loading $label"
launchctl bootstrap "gui/$uid" "$launchdPlist"

Set the proper file privileges on the postinstall file with

> chmod 755 scripts/postinstall

First the script sets the PATH. Then some variables for the label and bath to the label. You can change these to adapt the script to other LaunchAgents or LaunchDaemons.

Then the scripts checks the $3 argument. The installer system passes the target volume to the postinstall script in this argument. While it is a not common in deployment workflows any more, you can still install a pkg on a target volume that is not the current boot volume “/“. We remain safe with our script and check for that possibility, and exit without loading the agent.

LaunchAgents have to be loaded or bootstrapped into the current user’s context. The script gets the current user and checks whether the system might be sitting at the login window. We exit without loading if the system is at the login window or if the current user is _mbsetupuser which means the package is being installed while the system is sitting at Setup Assistant during first setup. Now that the LaunchAgent plist is in place, it will be loaded the first time the user logs in.

When there is a user logged in, we bootstrap the LaunchAgent right away. This is where the modern launchctl syntax shows its advantages. With the legacy commands, this would be:

launchctl asuser "$uid" launchctl load "$launchdPlist"

When you load a LaunchDaemon in a postinstall, the script is much simpler, as you don’t need to check for a current user. You can directly load the daemon into the system target domain:

launchctl bootstrap system/ "$launchdPlist"

Unloading before installation

When you are updating existing software with a package, you have to consider the situation where a LaunchAgent or LaunchDaemon is already running. In this case, we should unload before the installation. We can add a preinstall script (also no file extension) to the scripts folder that will be run before the payload is installed. This closely parallels the postinstall script.

#!/bin/sh

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

label="com.scriptingosx.desktopprmanage"
launchdPlist="/Library/LaunchAgents/${label}.plist"

# do not unload when target volume ($3) is not current startup volume
if [ "$3" != "/" ]; then
    echo "not installing on startup volume, exiting"
    exit 0
fi

if ! launchctl list | grep -q "$label"; then
    echo "$label not loaded, exiting"
    exit 0
fi

# for LaunchAgent, check if user is logged in
# see https://scriptingosx.com/2019/09/get-current-user-in-shell-scripts-on-macos/

currentUser=$(echo "show State:/Users/ConsoleUser" | scutil | awk '/Name :/ { print $3 }')

# don't unload at loginwindow or Setup Assistant
if [ "$currentUser" = "loginwindow" ] || [ "$currentUser" = "_mbsetupuser" ]; then
    echo "no user logged in, exiting"
    exit 0
fi

uid=$(id -u "$currentUser")

echo "loading $label"
launchctl bootout "gui/$uid" "$launchdPlist"

Again, set the proper file privileges on the preinstall file with

> chmod 755 scripts/preinstall

There is one extra step compared to the postinstall script. The preinstall uses launchctl list to check if the launchd configuration is loaded, before attempting to unload it.

Building the Package Installer

Now we get to put everything together. You can use pkgbuild to build an installer package:

> pkgbuild --root payload/ --scripts scripts/ --version 1 --identifier com.scriptingosx.desktopprmanage --install-location / DesktopprManage-1.pkg   
pkgbuild: Inferring bundle components from contents of payload/
pkgbuild: Adding top-level preinstall script
pkgbuild: Adding top-level postinstall script
pkgbuild: Wrote package to DesktopprManage-1.pkg

This works well, but it is difficult to memorize all the proper arguments for the pkgbuild command. If you forget or misconfigure one of them it will either fail outright or lead to a pkg installer file with unwanted behavior. I prefer to put the command in a script. Create a script file buildDesktopprManages.sh at the top of your project directory with these contents:

#!/bin/sh

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

pkgname="DesktopprManage"
version="1.1"
identifier="com.scriptingosx.${pkgname}"
install_location="/"
minOSVersion="10.13"

# determine enclosing folder
projectfolder=$(dirname "$0")

# build the component package
pkgbuild --root "${projectfolder}/payload" \
         --identifier "${identifier}" \
         --version "${version}" \
         --ownership recommended \
         --install-location "${install_location}" \
         --scripts "${projectfolder}/scripts" \
         --min-os-version "${minOSVersion}" \
         --compression latest \
         "${pkgname}-${version}.pkg"

This way, you do not have to memorize the pkgbuild command and its arguments and it is easier to modify parameters such as the version. Remember to change the version as you build newer versions of this package going forward!

Some management systems deploy a pkg file using certain MDM commands. Then you need a signed distribution package instead of a simple component pkg file. (You can learn more about the different types of packages in this blog post or this MacDevOps YVR presentation. Building a sign distribution pkg is somewhat more involved. I have a template script for that in the repo.

Conclusion

As I said, building a LaunchAgent for desktoppr manage is not trivial. However, building LaunchAgents and LaunchDaemons and proper installer pkgs to deploy them is an important skill for Mac admins.

There are some other tools available, such as outset, or your management system might have options to run scripts at certain triggers. But in general, I recommend to understand the underlying processes and technologies before using tools, even when the tools are great and a good fit for the task at hand.

You can use this process of building a LaunchAgent and installer pkg for desktoppr as a template for your own scripts and projects. You can find all the files in the desktoppr repo example directory.

desktoppr 0.5 — Managed by profile

Earlier this year, I released a beta version of desktoppr 0.5 which added an option to control the tool using a configuration profile. You can read all the details of how I built out this workflow in this blog post.

I have not received any feedback on it, which can mean two things: either it is working just fine or people simply aren’t aware of the beta. I guess both could be true at the same time here?

The new release is now available in the desktoppr repo. Nothing has changed compared to the 0.5beta except the version number and updated documentation.

Using desktoppr manage

The new manage verb allows a Mac Admin to set the wallpaper using data in a configuration profile. Together with the new option to download an image file for the wallpaper from an URL, this removes the requirement of updating two or more packages when you want to update the wallpaper on managed devices.

To use the new manage option, you need three pieces

desktoppr LaunchAgent

The LaunchAgent plist file controls when desktoppr will run.

The sample LaunchAgent file has both the RunAtLoad key and a StartInterval key. This means that desktoppr will run when the LaunchAgent loads and repeatedly after the time set in the StartInterval (in seconds, the sample file has 10800, so every three hours, as long as the system is awake).

The desktoppr profile provides a setOnlyOnce key which stops desktoppr from re-setting the same wallpaper over and over again. This way we can run desktoppr frequently, but it will only reset the wallpaper when the picture key in the profile has changed.

Build a pkg to install the LaunchAgent with your preferred frequency (or none) in /Library/LaunchAgents and deploy that together with the desktoppr pkg.

Configuration Profile

Since macOS Ventura, managed binaries and scripts launched by a LaunchAgent or Daemon should be pre-approved with a configuration profile from the management system, so that the user gets less dialogs about them and cannot disable them. Since desktoppr is a signed binary, this is quite straightforward and the sample configuration profile already contains the com.apple.servicemanagement section.

The second payload in the sample profile contains the settings. The only one that is required is the picture key which contains either a path to a local image file for the wallpaper, or a URL to an image file. Using a hosted file allows you to provide a custom branded wallpaper and change it without needing to deploy files to the client using more custom pkgs.

When you are not using a local file, you can (and probably should) provide a sha256 key with a checksum of the image file. This should protect from attempt to inject something malicious by hi-jacking the download URL. You can generate a sha256 sum for the image file with shasum -a 256 <path/to/file>.

The setOnlyOnce key controls whether desktoppr manage will re-set the wallpaper every time it runs (when setOnlyOnce is false) or only when the picture key in the profile changed (when setOnlyOnce is true). Configuring setOnlyOnce to true allows a user to change the wallpaper and it will not be overridden until you change the configuration profile. Then the user can change it again. This seemed like a useful compromise between management and user choice to me.

The scale and color keys work just like the options of the command line tool. Note that setting the color option is broken in macOS 14 as of this writing.

You can of course use different combinations of LaunchAgent configuration and other triggers for desktoppr, such as a self service portal to get all kinds of different workflows and levels of ‘lockdown.’ Remember that Apple provides a config profile setting to fully set and lock the wallpaper.

Conclusion

I have found this new managed desktoppr option useful in my deployments. I hope you do, too. Let me know!

Prefs CLI Tools for Mac Admins

Recently I have been working on some… well… “stuff” that uses custom configuration profiles. Very custom, and since I am testing things, they need to be updated a lot.

The issue with defaults

When you are working with defaults/preferences/settings/property lists on macOS, you will be familiar with the defaults command line tool. But, as useful as defaults can be, it has some downsides.

One of the great advantages of macOS’ preference system is that settings can be provided on multiple levels or domains. In my book “Property Lists, Preferences and Profiles for Apple Administrators, I have identified 19 different levels where settings for a single application can originate.

You will be most familiar with plist files in /Library/Preferences (system), ~/Library/Preferences (user), and managed configuration profiles (managed). When an app or tool requests a setting, the preferences system will merge all those levels together and present only the most relevant value. When the developer uses the system APIs (correctly), they do not have to worry about all the underlying levels, domains and mechanisms very much, but automatically gain support for things like separated system and user level settings files and support for management through configuration profiles.

The macOS defaults command line tool can work with settings on different levels or domains, but will only show the settings from one at a time. By default it only works with the user domain settings stored in ~/Library/Preferences/. When you have settings in multiple levels or from configuration profiles, you may be able to point defaults directly at the files. Or in the case of managed settings from profiles, you have to use a different tool. Either way, you have to determine which setting might override another and which final value might be visible to the app or process.

A new prefs tool

Years back, I had built a python script, called prefs.py, which would not only show the coalesced set of settings but their origin level. When macOS removed Python 2 in macOS 12.3, this tool obviously broke.

While working with preferences and profiles recently, this feature would have been quite useful to debug and verify preferences. I could have adapted the existing tool to work with MacAdmins Python 3, but felt I would learn something from recreating it in Swift. I had already started down that road just a bit for my sample project in this post.

So, you can find the new Swift-based prefs command line tool on GitHub. You can also download a signed and notarized pkg which will install the binary in /usr/local/bin/.

If its most basic form, you run it with a domain or application identifier. It will then list the merged settings for that preference domain, showing the level where the final value came from.

% prefs com.apple.screensaver
moduleDict [host]: {
    moduleName = "Computer Name";
    path = "/System/Library/Frameworks/ScreenSaver.framework/PlugIns/Computer Name.appex";
    type = 0;
}
PrefsVersion [host]: 100
idleTime [host]: 0
lastDelayTime [host]: 1200
tokenRemovalAction [host]: 0
showClock [host]: 0
CleanExit [host]: 1

I find this useful when researching where services and applications store their settings and also to see if a custom configuration profile is set up and applying correctly. There is a bit of documentation in the repo’s ReadMe and you can get a description of the options with prefs --help.

plist2profile

Another tool that would have been useful to my work, but that was also written in python 2 is Tim Sutton’s mcxToProfile. Back in the day, this tool was very useful when transitioning from Workgroup Manager and mcx based management to the new MDM and configuration profile based methods. If you have a long-lived management service, you will probably find some references to mcxToProfile in the custom profiles.

Even after Workgroup Manager and mcx based settings management was retired, Tim’s tool allowed to create a custom configuration profile from a simple property list file. Configuration Profiles require a lot of metadata around the actual settings keys and values, and mcxToProfile was useful in automating that step.

Some management systems, like Jamf Pro, have this feature built in. Many other management systems, however, do not. (Looking at you Jamf School.) But even then creating a custom profile on your admin Mac or as part of an automation, can be useful.

So, you probably guessed it, I also recreated mcxToProfile in Swift. The new tool is called plist2profile and available in the same repo and pkg. I have focused on the features I need right now, so plist2profile is missing several options compared to mcxToProfile. Let me know if this is useful and I might put some more work into it.

That said, I added a new feature. There are two different formats or layouts that configuration profiles can use to provide custom setting. The ‘traditional’ layout goes back all the way to the mcx data format in Workgroup Manager. This is what mcxToProfile would create as well. There is another, flatter format which has less metadata around it. Bob Gendler has a great post about the differences.

From what I can tell, the end effect is the same between the two approaches. plist2profile uses the ‘flatter’, simpler layout by default, but you can make it create the traditional mcx format by adding the --mcx option.

Using it is simple. You just need to give it an identifier and one or more plist files from which it will build a custom configuration profile:

% plist2profile --identifier example.settings com.example.settings.plist

You can find more instructions in the ReadMe and in the commands help with plist2profile --help

Conclusion

As I had anticipated, I learned a lot putting these tools together. Not just about the preferences system, but some new (and old) Swift strategies that will be useful for the actual problems I am trying to solve.

I also learnt more about the ArgumentParser package to parse command line arguments. This is such a useful and powerful package, but their documentation fails in the common way. It describes what you can do, but not why or how. There might be posts about that coming up.

Most of all, these two tools turned out to be useful to my work right now. Hope they will be useful to you!

Build a macOS Application to Run a Shell Command with Xcode and SwiftUI – Part 2

A long time ago, I wrote a post on how to build a simple App using SwiftUI that would run a shell command. Amazingly, the tutorial still works!

This is particularly useful for MacAdmins, because we sometimes want to provide a simple UI for some command or script. However, some things have changed since I wrote this tutorial three years. While the code still “works,” there are some things that can be improved. Also, when I wrote the original post, there were some features I didn’t expand upon, because the post was already very long, and, to be honest, back then, I didn’t really know how they worked, myself.

When I re-visited this earlier because of a question in the MacAdmins Slack, I was really surprised that I didn’t need to update code to make it work. That said, there are the matters I didn’t explain last time. Also Swift has changed in some ways. That means it is time for the second part.

You can get the sample code from the first part as a starting point.

Get the output of the command

In the first part, we ran the say command, which doesn’t have a text output. Often times we want to run shell commands to get information from the command’s standard output (or sometimes standard error). To grab the output of commands, we need to prepare the Process object, which means we cannot use the convenience method Process.run() any more.

The say command has an option to change the voice being used. You can list the names of voices available to the say command by running say -v '?' in Terminal. We’d like to get that output and populate a menu with the available voices in our UI.

Note: the output of the say -v '?' command does not show most of the modern voices available in macOS. Most of the voices listed are quite old, funny, and, to be honest, awful. As has been mentioned before, I am using the say command as a convenient example for a shell command. If you want proper speech synthesis in your app, you should not use the say command but the proper speech synthesis API.

We will start out experimenting with code in a macOS Playground in Xcode and later add it to the app from the first part. When you create a new Playground in Xcode (File > New > New Playground), make sure you select ‘macOS’ for the platform above the template picker and then ‘Blank’ as the template.

Start with this code:

let process = Process()
process.launchPath = "/usr/bin/say"
process.arguments = ["-v", "?"]
let outPipe = Pipe()
process.standardOutput = outPipe
process.terminationHandler = { process in
  let outData = outPipe.fileHandleForReading.readDataToEndOfFile()
  let output = String(data: outData, encoding: .utf8) ?? ""
 print(output)
}
try? process.run()
process.waitUntilExit()

You will recognize some of the code from the first part of this tutorial. We create a Process object and set its launchPath and arguments. Note that the Process.run() convenience method, which we used in the first part, takes a URL to define the executable, but the launchPath is a String containing the path to the executable. We also (this is the new part) create a Pipe and set it as the standardOutput of the process.

The process will run the say command asynchronously. Since we want to work with the output , we have to give the process some code to execute when the command is done. We set this terminationHandler to a closure. The code in this closure grabs the data from pipe we connected to standardOutput of the process, converts it to a String and prints it.

At the end of our code, we tell the process object to run(). The waitUntilExit() is necessary here, because that is all this particular playground code does and we want to be sure that the command and the termination handler get a chance to do their work, before the code ends. In an app, where many things may be happening at the same time, you usually will not use waitUntilExit(). We will see that later when we implement our solution in SwiftUI.

Many outcomes

This code grabs the standard out from the command. Unix commands have different results that can be interesting. Some commands print to standard error instead of (or together with) standard out. All unix commands also have an exit code, which should return 0 for successful runs and a non-zero value for failures. Some commands use different non-zero value to give information for different errors.

All of this is available from Process objects. However, you have to set up the pipes and connections and get the data back in the termination handler, and the code gets quite complicated and tedious to set up. We have the seen the Process type has a convenience method to create and run a command without the pipes. I don’t know why it doesn’t have a convenience method when you are interested in all the data, but the good news is, we can create an extension to build our own.

Replace the code in the playground with the following:

import Foundation

extension Process {
  @discardableResult
  static func launch(
    path: String,
    arguments: [String] = [],
    terminationHandler: @escaping (Int, Data, Data) -> Void
  ) throws -> Process {
    let process = Process()
    let outPipe = Pipe()
    let errorPipe = Pipe()
    process.standardOutput = outPipe
    process.standardError = errorPipe
    process.arguments = arguments
    process.launchPath = path
    process.terminationHandler = { process in
      let outData = outPipe.fileHandleForReading.readDataToEndOfFile()
      let errorData = errorPipe.fileHandleForReading.readDataToEndOfFile()
      let exitCode = Int(process.terminationStatus)
      terminationHandler(exitCode, outData, errorData)
    }
    try process.run()
    return process
  }
}

let process = try? Process.launch(
  path: "/usr/bin/say", 
  arguments: ["-v", "?"]
) { exitCode, outData, errData in
  let output = String(data: outData, encoding: .utf8) ?? ""
  print(output)
}
process?.waitUntilExit()

You will recognize the code from our first example in the launch function in the extension. But here, we have added the configuration for a second Pipe for standardError, in the closure we get the Data for the standard out, standard error and the exit code and pass them in to the closure passed in a termination handler.

This method simplifies using the Process type. We just have to pass in the path to the command and the arguments array and give a closure that is called when the command completes.

async/await

But it still uses a closure for the termination handler. This breaks our code into different fragments that are not executed in the order they appear in. Since macOS Monterey 12 and iOS 15, Swift has had a concurrency feature called async/await. With this, your code appears in a more natural order.

When a function has the await marker, the system knows to suspend the code at that point until the function returns a result. While this code is “on hold” other threads or tasks can run, such as the UI handling, so your process or app isn’t blocked. When the function returns, the code continues after the function, so the code that processes the output of the function comes in the logical order, which makes it easier to read and understand.

For some reason, the Process type has not yet been updated to use this new feature. We can however, add this functionality using an extension. Add this method to the extension:

  static func launch(
    path: String,
    arguments: [String] = []
  ) async throws -> (exitCode: Int, standardOutput: Data, standardError: Data) {
    try await withCheckedThrowingContinuation { continuation in
      do {
        try launch(path: path, arguments: arguments) { exitCode, outData, errData in
          continuation.resume(returning: (exitCode, outData, errData))
        }
      } catch let error {
        continuation.resume(throwing: error)
      }
    }
  }

If you want to get more detail how async and await work, I recommend the WWDC session “Meet async/await in Swift” from WWDC 2021.

With this, we can change our code to run the say command to:

guard let (exitCode, outData, errData) = try? await Process.launch(
    path: "/usr/bin/say",
    arguments: ["-v", "?"]
) else { exit(0) }

let output = String(data: outData, encoding: .utf8) ?? ""
print(output)

The code is now in a more sensible order. If you can afford macOS Monterey as a minimum system requirement, you should consider adopting async/await.

Now that we have our output of the command, we have to parse out the names of the voices. This code will turn the output of the say into an Array of names, ignoring the language code and sample text:

func parseVoices(_ output: String) -> [String] {
  output
    .components(separatedBy: "\n")
    .map {
      $0
        .components(separatedBy: "#")
        .first?
        .trimmingCharacters(in: .whitespaces)
        .components(separatedBy: CharacterSet.whitespaces)
        .dropLast()
        .filter { !$0.isEmpty }
        .joined(separator: " ")
      ?? ""
    }
    .filter { !$0.isEmpty }
}

So we can add these lines to get an Array of `voices:

let voices = parseVoices(output)
print(voices)

You can find the stages of code for the playground in this gist.

Updating the app

Now that we have assembled all the pieces working in a playground, we can move on to putting these pieces in our SwiftUI app. The goal is to have a popup menu with all the voices above the field where you can enter text.

We will be using some features that were introduced in macOS Monterey 12.0. If you have built the project a while ago with an older version of macOS, the project may be set to build with older versions of macOS. To verify and, if necessary, update the deployment target for the project, select the blue project icon at the very top of the item sidebar, then select the gray “SayThis” icon under the “Targets” section. Then select “General” in the tab list. The second section is called “Minimum Deployments” and should show macOS 12.0 or later. When you change this from an older version, Xcode may prompt to update some other settings in the project, you can accept those.

First, we want to use the extension to the Process type we created in the playground earlier. Create a new file in the project (File > New > New File…), select “Swift File” from the templates, and name it ‘Process-launch’. Copy and paste the extension code from the playground (with both methods) to that file. You could have all the code in a single file, but it is cleaner and more maintainable to have one file per type or extension. This also allows you to copy that file to other projects to use it there.

Next we need two more state variables to track the state of the popup-menu that we are going to add. In the ContentView file, add these two lines right under the existing @State variables:

  @State var voices: [String] = []
  @State var selectedVoice: String = "Samantha"

The first is where we are going to store all the available voices that we parse from the output of say -v '?'. The second variable will store the current selection.

Next, we will insert the popup menu for the voices into the UI. In the body, insert this below the title and above the HStack:

      Picker(selection: $selectedVoice, label: Text("Voice:")) {
          ForEach(voices, id: \.self) { Text($0) }
      }

First we create a Picker view, which is not really proper terminology for macOS. But since SwiftUI is designed to share objects across all platforms, a ‘Picker’ will render as a popup menu in macOS. We attach the selectedVoice state variable to the selection of the picker. Then, we loop through all the items in the voices state variable to add them to the picker.

At this point, you should see the popup menu in the preview and when you run the project. You cannot select anything from it, though, which makes sense as the voices array is empty.

We need to populate the voices array before the UI appears. SwiftUI offers a special modifier to do this. You can attach a .task modifier to any view and its closure will be run asynchronously once, before the view appears for the first time. Add this code to the end of the VStack block, right below the line that reads .frame(maxWidth: .infinity, maxHeight: .infinity):

    .task {
      guard let (_, outData, _) = try? await Process.launch(
        path: "/usr/bin/say",
        arguments: ["-v", "?"]
      ) else { return }

      let output = String(data: outData, encoding: .utf8) ?? ""
      voices = parseVoices(output)
    }

You also need to copy the parseVoices() function from the test playground and add it to the ContentView struct.

This will populate the popup menu with all the voices. Build and run to test. It will still not actual use the selected voice. We need to update the runCommand() method to:

  func runCommand() {
    let arguments = [message, "-v", selectedVoice]
    let executableURL = URL(fileURLWithPath: "/usr/bin/say")
    self.isRunning = true
    try! Process.run(executableURL, arguments: arguments) { _ in
      self.isRunning = false
    }
  }

We have achieved what we set out to do. You can find the project with all the code for this point here.

Refinements

However, there are some refinements and improvements that we can and should still add to the code.

First, the runCommand() function still uses the old convenience handler with a termination handler. We can change it to use our new await Process.launch():

  func runCommand() async {
    let arguments = [message, "-v", selectedVoice]
    self.isRunning = true
    let _ = try? await Process.launch(path: "/usr/bin/say", arguments: arguments)
    self.isRunning = false
  }

This will now generate an error in the Button because we have changed the runCommand() function to be async. Change the code of the Button to

        Button("Say") {
          Task {
            await runCommand()
          }
        }
        .disabled(isRunning)

By wrapping the await runCommand() function in a Task { } we are telling the system that this code should run in the background and not return immediately. This is necessary to run code blocks with async functions from UI elements.

The say command will use the voice set in the system settings when no -v option is given. We want to provide a way to recreate that behavior in our UI. We can add items to the Picker by adding elements before the ForEach:

      Picker(selection: $selectedVoice, label: Text("Voice:")) {
        Text("System Default").tag("System Default")
        Divider()
        ForEach(voices, id: \.self) { Text($0) }
      }

Also change the default value of the selectedVoice variable to "System Default".

Then we have to change the runCommand() method to only use the -v and voice name arguments when their value is not “System Default”

  func runCommand() async {
    var arguments = [message]
    if selectedVoice != "System Default" {
      arguments.append(contentsOf: ["-v", selectedVoice])
    }
    self.isRunning = true
    let _ = try? await Process.launch(path: "/usr/bin/say", arguments: arguments)
    self.isRunning = false
  }

You can find the code for this here.

Out of the Sandbox

The say command provides a fun example to play around with. You will probably already have plenty of ideas that you want to try out. There is one thing that might trip you up here: sandboxed apps do not get access to all available command line tools. If you want to run such ‘prohibited’ commands, you will need to remove the sandbox configuration from the app.

Better Extension

The extension I put together here will work in most situations, but there are conditions where different parts might not work well. Quinn the Eskimo has published this code which goes further and is probably “even more” thread safe.

Launching commands is an “expensive” task. You should always do some research to see if there is some framework that can give you access to the same functionality natively in Swift. Nevertheless, sometimes it might be necessary, and sometimes it might just be easier to call out to an external command.

This tutorial should help you along the way of using these tools effectively.

RAID and macOS

RAIDs are a strange edge case that are rarely useful outside of servers, but when they are useful, they are very important. RAID is an acronym for ‘Redundant Array of Independent/Inexpensive Disks.’ It is a technology where you combine multiple, physical disks into a single virtual drive for redundancy, speed, or both.

Often the RAID system is handled by a dedicated controller in an external enclosure, but sometimes, you want or need to work with drives directly. macOS has some basic RAID functionality built-in and there are good third party options if you want to go further.

RAID Levels

If you are unfamiliar with RAIDs, we need to get some terminology sorted out first. There are different kinds of RAIDs which are called ‘levels.’

In this post, I am going to focus on level 1 or ‘RAID 1.’ With RAID 1, the data is ‘mirrored’ between two or more drives, so that each drive carries a complete copy of the data. This protects from data loss because of drive failure. Note that RAID 1 does not protect from other common reasons for data loss, such as file system or individual file corruption, accidental or malicious file deletion. A RAID is never a replacement for a good backup strategy. Since the data is mirrored on all the devices, a RAID 0 will only have the capacity of the smallest drive in the set. It is generally recommended that drives in any RAID set should be of the same capacity and type, for best performance and efficiency.

A level 0 RAID (or ‘RAID 0’) is not actually redundant. In a RAID 0 the data is ‘striped’ across two or more drives so that writes and reads happen in parallel, which increases the data bandwidth available. Since the data is spread evenly (striped) across all drives in the RAID 0 set, failure of a single drive will result in complete data loss. The capacity of the stripe raid is the capacity of the smallest drive in the set multiplied by the number of drives in the set. Having drives of the same type and capacity is even more relevant for RAID 0 performance.

There are more RAID levels, such as 0+1, 10 and 5 and dedicated disk controllers will have more options (such as combining multiple drives of different sizes more efficiently), but we will focus RAID 1 (mirror).

Here, there be dragons

Warning: many of the commands shown here to setup and experiment with disk drives and RAIDs may or will lead to loss of the data on the drives involved, so be careful. I strongly recommend disconnecting any drives other than those you are experimenting with from the Mac you are working on.

I would also recommend to experiment with a set of drives that contain no relevant data at all. Two USB sticks will do just fine to explore and test the functionality. Drives do not have to be of the same capacity and type for testing, but I do recommend that for actual use.

Apple RAID

macOS has built-in support for software-based mirror and stripe RAID called “AppleRAID.” This also provides a third option to concatenate drives, but concatenation provides neither redundancy, nor performance, so I do not recommend using it.

You can use the Disk Utility app to setup a RAID. It has a nice assistant that you an access from the File menu called RAID Assistant. It will ask you what kind of RAID you want to setup and allow you to select the drives and create a new RAID volume. This will delete the data on the disk drives and there are few features that are not exposed in the Disk Utility UI, so I will focus on how to do it in the command line.

You can keep Disk Utility open to get a visual representation of what is going on, though the Disk Utility app often has problems keeping up with changes done from the command line. You may have to quit and restart the app to force it to update its status. You want to enable “Show all Devices” from the View menu to see the physical drives as well as the file systems and virtual drives.

First, we need to identify the disks that we want to work with. When you run diskutil list it will list all the disk on the system. Usually disk0 will be the built-in drive, and disk3 will be the (synthesized) APFS container inside (with the System and Data volume). But depending on what Mac you are using, what your configuration is, and what devices you had attached to the Mac before you started this, the numbers may be different.

Again, to be safe, unmount and disconnect any drives or file servers with data that you care about at this point, and the connect the two drives you want to use for experimentation. Their data will be erased!

Run diskutil list again and identify the device identifiers for the drives you will be working with. They should look like this:

/dev/disk4 (external, physical):

For me, the two drives where disk4 and disk5, so I will be using those numbers in my examples, but be sure to replace those with the device numbers on your system, other wise you might be working with the wrong disk or volume.

Promoting a drive to mirror RAID

One of the features you can use from the command line is to ‘promote’ an existing drive to a mirror RAID without data loss. Apple RAID promotion works (as far as I can tell) only with HFS+ formatted Volumes, so let us reformat the first disk (disk4) as such:

> diskutil eraseDisk JHFS+ DiskName disk4 
Started erase on disk4
Unmounting disk
Creating the partition map
Waiting for partitions to activate
Formatting disk4s2 as Mac OS Extended (Journaled) with name DiskName
Initialized /dev/rdisk4s2 as a 59 GB case-insensitive HFS Plus volume with a 8192k journal
Mounting disk
Finished erase on disk4

Using the command line, we will be able to promote this HFS+ drive without have to erase it (again), so copy some (un-important) files to it now.

diskutil has a sub-group of commands dedicated to the RAID functions called appleRAID or ar for short. I am going to use the short form. You can run diskutil ar to get a list of the sub-commands for working with Apple Raid. You can read the diskutil man page for details.

Next we have to enable AppleRAID on the drive. Enabling RAID on single drive seems a bit pointless, but this prepares everything on that drive, so that we can add more drives later.

> diskutil ar enable mirror DiskName
Started RAID operation on disk4s2 (DiskName)
Resizing disk
Unmounting disk
Adding a booter for the disk
Creating a RAID set
Bringing the RAID partition online
Waiting for the new RAID to spin up "8D05B6EB-DCFB-426D-885B-ED8C76DC2484"
Finished RAID operation on disk4s2 (DiskName)

The volume and the files you had copied earlier are still there. But the volume is now listed under “RAID Sets” in Disk Utility. You can see the single drive in the RAID Set in the UI there. You can also get this info in command line with

> diskutil ar list   
AppleRAID sets (1 found)
===============================================================================
Name:                 DiskName
Unique ID:            8D05B6EB-DCFB-426D-885B-ED8C76DC2484
Type:                 Mirror
Status:               Online
Size:                 63.8 GB (63816400896 Bytes)
Rebuild:              manual
Device Node:          disk6
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk4s2   467826B1-BBA2-4671-99CE-5CBB04E06882  Online     63816400896
===============================================================================

To make this a real mirror RAID, we need to add the second drive:

> diskutil ar add member disk5 DiskName                
Started RAID operation on disk6 (DiskName)
Unmounting disk
Repartitioning disk5 so it can be in a RAID set
Unmounting disk
Creating the partition map
Adding disk5s2 to the RAID Set
Finished RAID operation on disk6 (DiskName)

This will add the drive to the RAID. This will delete any data that might be on disk5!

When you look at the RAID in Disk Utility (you might have to restart the app for it to pick up the new status) you will now see both drives, but one of them has the status “Rebuilding” with a percentage. The status of the entire RAID set is now “Degraded.” The RAID system is still in the process of mirroring data to the new member. You can use the volume to read and write data at this time, but it is not redundant yet. If the first drive fails during rebuilding, the data will be gone.

Once the rebuilding is done, the status of the RAID will change to “Online,” which is the “good” status. At this point the data on the RAID will be resilient to the failure or removal of either of the drives.

You can also create the RAID with both drives from the start, but this will erase all the data on both drives (this is what RAID Assistant in Disk Utility does.

Before we can try the other way of creating a mirror RAID set, we need to “break” the mirror we have built so far.

> diskutil ar delete DiskName
Started RAID operation on disk6 (DiskName)
Unmounting volume for RAID set 8D05B6EB-DCFB-426D-885B-ED8C76DC2484
Destroying the RAID set 8D05B6EB-DCFB-426D-885B-ED8C76DC2484
Finished RAID operation on disk6 (DiskName)

If you waited for the rebuilding to be complete, both individual drives will each contain the data of the former mirror RAID. If the rebuilding was not complete yet, only the first drive will contain the data, the second drive will be empty.

Create a new RAID set

Now let’s build a new empty mirror RAID with both drives included from the start:

> diskutil ar create mirror DiskName APFS disk4 disk5
Started RAID operation
Unmounting proposed new member disk4
Unmounting proposed new member disk5
Repartitioning disk4 so it can be in a RAID set
Unmounting disk
Creating the partition map
Using disk4s2 as a data slice
Repartitioning disk5 so it can be in a RAID set
Unmounting disk
Creating the partition map
Using disk5s2 as a data slice
Creating a RAID set
Bringing the RAID partitions online
Waiting for the new RAID to spin up "FE3E7A3F-E4BF-4AEE-BC3C-094A9BFB3251"
Mounting disk
Finished RAID operation

Note that we can build an APFS volume on the RAID set with the command line tool. RAID Assistant in Disk Utility will build a HFS+ volume. In this new empty RAID set both drives will be online immediately.

When you replace the mirror with stripe in this command you will get a striped RAID 0 volume. (performance instead of redundancy)

Drive failure

We can simulate a drive failure by unplugging one of the members. Sadly, macOS does not seem to have a notification for this event. Once you have removed the drive you can run disktutil ar list and see the status is “Degraded” and the which member is “Missing/Damaged.” You can keep using the volume in degraded mode.

Once you plug the drive back in it will appear as ‘failed.’ You can start the process of repairing or rebuilding the mirror with

> diskutil ar repairMirror FE3E7A3F-E4BF-4AEE-BC3C-094A9BFB3251 disk5  
Started RAID operation
Unmounting disk
Repartitioning disk5 so it can be in a RAID set
Unmounting disk
Creating the partition map
Adding disk5s2 to the RAID Set
Finished RAID operation

Note: Syncing data between mirror partitions can take a very long time.
Note: The mirror should now be repairing itself.  You can check its status using 'diskutil appleRAID list'.

The UUID is the UniqueID of the RAID set you see with diskutil ar list. The warning you get at the end is fair. The rebuilding process will take a while. How it takes depends on how full the volume is and how fast the new member drive is.

Downsides of AppleRAID

There are quite a few downsides to the built-in AppleRAID functionality. There is no notification or warning when one of the drives in a mirror goes offline and the RAID is running in degraded state. The RAID will also not automatically rebuild when a missing drive re-appears. (There is an AutoRebuild option mention in man page, but whenever I tried to enable that, the entire disk management stack froze in a way that required a reboot.)

AppleRAID can be useful to quickly stripe some random disks for performance. But generally, if the data was of so much concern that I am considering RAID 1, I would not rely on AppleRAID.

SoftRAID

There is a wonderful third party tool for managing software based RAIDs on macOS called SoftRAID (14-day free trial, then a tiered license). And, much to my delight, it also comes with a command line tool. Creating a RAID is not something you do regularly, so I went ahead and did this in the GUI app. Once that was setup, I used the command line tool to get the RAID’s status:

> softraidtool volume DiskName info

Info for "DiskName":

Mountpoint: /Volumes/DiskName
BSD disk: disk4
Total Bytes: 59.6 GB (64,016,478,208)
Free Bytes: 59.6 GB (64,016,478,208)
Volume format: unknown
Volume is DiskName
RAID level: RAID 1
DiskName ID: 09DF05C72BFFAD20
Optimized for: Workstation
Created: Jul 17, 2023 at 3:33:39 PM
Last Validated: never
Volume state: normal, 
Volume Safeguard: enabled
Total I/Os: 5,610
Total I/O Errors: 0
Total number secondary disks (including offline ones): 1

Disks used for this volume:
bsd disk:    SoftRAID ID:         Location and Size:
disk7     09DF053ECE82F980     (USB3 bus 0, id 4 - 59.8 GB)  secondary disk, 
disk6     09DF053D23386500     (USB3 bus 0, id 5 - 59.8 GB)  primary disk, 

The SoftRAID software also comes with a menubar app that shows the status of the RAID.

When you unplug one of the drives, the ‘Volume state’ changes to ‘missing disk.’ When you plug the missing drive back in, SoftRAID will automatically detect it and rebuild the RAID, when necessary. Rebuilding went so quickly that I had a hard time capturing the state from the command line. The more changes you apply to the degraded RAID the longer the rebuild takes.

> softraidtool volume SoftRAID info
SoftRAIDTool status: waiting for disk5 to finish (00:00:01)

Info for "SoftRAID":

Mountpoint: /Volumes/SoftRAID
BSD disk: disk5
Total Bytes: 59.6 GB (64,016,478,208)
Free Bytes: 59.6 GB (64,016,478,208)
Volume format: unknown
Volume is SoftRAID
RAID level: RAID 1
SoftRAID ID: 09DF06D95024CBE0
Optimized for: Workstation
Created: Jul 17, 2023 at 3:53:16 PM
Last Validated: never
Volume state: rebuiding, out of sync, 
Volume Safeguard: enabled
Volume progress: 15%
Current offset: 7,343,685,632
Time remaining: 00:07:24
Total I/Os: 22,752
Total I/O Errors: 0
Total number secondary disks (including offline ones): 1

Disks used for this volume:
bsd disk:    SoftRAID ID:         Location and Size:
disk4     09DF053ECE82F980     (USB3 bus 0, id 4 - 59.8 GB)  primary disk, 
disk7     09DF053D23386500     (USB3 bus 0, id 5 - 59.8 GB)  secondary disk, rebuiding, out of sync, 

You can parse this output using awk to get just the volume state. This is useful for reporting the state to Jamf Pro with an extension attribute:

#!/bin/sh

# reports the SoftRAID status

softraidtool="/usr/local/bin/softraidtool"

if [ ! -x "$softraidtool" ]; then
    echo "<result>SoftRAID not installed</result>"
fi

volumestate=$(softraidtool volume SoftRAID info | awk -F ': ' '/Volume state/ {print $2}')

echo "<result>$volumestate</result>"

Keep in mind that Jamf Inventory Updates (aka as recon) may run very infrequently (recommended default is once per day, and it shouldn’t run much more often than that to avoid database bloat), so the data in your Jamf Pro may be hours or sometimes longer out of date. If you want to react to changes in the RAID status more quickly, you should rely on other tools than Jamf Pro.

Conclusion

The best solution for RAIDs will always be a dedicated hardware controller. But there also good reasons (cost) to just put together a “bunch of disks” into a RAID. The built-in AppleRAID functionality works, but has limitations, especially for mirror RAIDs. SoftRAID is a great tool to overcome these limitations. For Mac admins, both can be managed and monitored with command line tools, which allows automation and integration with your management system.