Prefs CLI Tools for Mac Admins

Recently I have been working on some… well… “stuff” that uses custom configuration profiles. Very custom, and since I am testing things, they need to be updated a lot.

The issue with defaults

When you are working with defaults/preferences/settings/property lists on macOS, you will be familiar with the defaults command line tool. But, as useful as defaults can be, it has some downsides.

One of the great advantages of macOS’ preference system is that settings can be provided on multiple levels or domains. In my book “Property Lists, Preferences and Profiles for Apple Administrators, I have identified 19 different levels where settings for a single application can originate.

You will be most familiar with plist files in /Library/Preferences (system), ~/Library/Preferences (user), and managed configuration profiles (managed). When an app or tool requests a setting, the preferences system will merge all those levels together and present only the most relevant value. When the developer uses the system APIs (correctly), they do not have to worry about all the underlying levels, domains and mechanisms very much, but automatically gain support for things like separated system and user level settings files and support for management through configuration profiles.

The macOS defaults command line tool can work with settings on different levels or domains, but will only show the settings from one at a time. By default it only works with the user domain settings stored in ~/Library/Preferences/. When you have settings in multiple levels or from configuration profiles, you may be able to point defaults directly at the files. Or in the case of managed settings from profiles, you have to use a different tool. Either way, you have to determine which setting might override another and which final value might be visible to the app or process.

A new prefs tool

Years back, I had built a python script, called prefs.py, which would not only show the coalesced set of settings but their origin level. When macOS removed Python 2 in macOS 12.3, this tool obviously broke.

While working with preferences and profiles recently, this feature would have been quite useful to debug and verify preferences. I could have adapted the existing tool to work with MacAdmins Python 3, but felt I would learn something from recreating it in Swift. I had already started down that road just a bit for my sample project in this post.

So, you can find the new Swift-based prefs command line tool on GitHub. You can also download a signed and notarized pkg which will install the binary in /usr/local/bin/.

If its most basic form, you run it with a domain or application identifier. It will then list the merged settings for that preference domain, showing the level where the final value came from.

% prefs com.apple.screensaver
moduleDict [host]: {
    moduleName = "Computer Name";
    path = "/System/Library/Frameworks/ScreenSaver.framework/PlugIns/Computer Name.appex";
    type = 0;
}
PrefsVersion [host]: 100
idleTime [host]: 0
lastDelayTime [host]: 1200
tokenRemovalAction [host]: 0
showClock [host]: 0
CleanExit [host]: 1

I find this useful when researching where services and applications store their settings and also to see if a custom configuration profile is set up and applying correctly. There is a bit of documentation in the repo’s ReadMe and you can get a description of the options with prefs --help.

plist2profile

Another tool that would have been useful to my work, but that was also written in python 2 is Tim Sutton’s mcxToProfile. Back in the day, this tool was very useful when transitioning from Workgroup Manager and mcx based management to the new MDM and configuration profile based methods. If you have a long-lived management service, you will probably find some references to mcxToProfile in the custom profiles.

Even after Workgroup Manager and mcx based settings management was retired, Tim’s tool allowed to create a custom configuration profile from a simple property list file. Configuration Profiles require a lot of metadata around the actual settings keys and values, and mcxToProfile was useful in automating that step.

Some management systems, like Jamf Pro, have this feature built in. Many other management systems, however, do not. (Looking at you Jamf School.) But even then creating a custom profile on your admin Mac or as part of an automation, can be useful.

So, you probably guessed it, I also recreated mcxToProfile in Swift. The new tool is called plist2profile and available in the same repo and pkg. I have focused on the features I need right now, so plist2profile is missing several options compared to mcxToProfile. Let me know if this is useful and I might put some more work into it.

That said, I added a new feature. There are two different formats or layouts that configuration profiles can use to provide custom setting. The ‘traditional’ layout goes back all the way to the mcx data format in Workgroup Manager. This is what mcxToProfile would create as well. There is another, flatter format which has less metadata around it. Bob Gendler has a great post about the differences.

From what I can tell, the end effect is the same between the two approaches. plist2profile uses the ‘flatter’, simpler layout by default, but you can make it create the traditional mcx format by adding the --mcx option.

Using it is simple. You just need to give it an identifier and one or more plist files from which it will build a custom configuration profile:

% plist2profile --identifier example.settings com.example.settings.plist

You can find more instructions in the ReadMe and in the commands help with plist2profile --help

Conclusion

As I had anticipated, I learned a lot putting these tools together. Not just about the preferences system, but some new (and old) Swift strategies that will be useful for the actual problems I am trying to solve.

I also learnt more about the ArgumentParser package to parse command line arguments. This is such a useful and powerful package, but their documentation fails in the common way. It describes what you can do, but not why or how. There might be posts about that coming up.

Most of all, these two tools turned out to be useful to my work right now. Hope they will be useful to you!

Build a macOS Application to Run a Shell Command with Xcode and SwiftUI – Part 2

A long time ago, I wrote a post on how to build a simple App using SwiftUI that would run a shell command. Amazingly, the tutorial still works!

This is particularly useful for MacAdmins, because we sometimes want to provide a simple UI for some command or script. However, some things have changed since I wrote this tutorial three years. While the code still “works,” there are some things that can be improved. Also, when I wrote the original post, there were some features I didn’t expand upon, because the post was already very long, and, to be honest, back then, I didn’t really know how they worked, myself.

When I re-visited this earlier because of a question in the MacAdmins Slack, I was really surprised that I didn’t need to update code to make it work. That said, there are the matters I didn’t explain last time. Also Swift has changed in some ways. That means it is time for the second part.

You can get the sample code from the first part as a starting point.

Get the output of the command

In the first part, we ran the say command, which doesn’t have a text output. Often times we want to run shell commands to get information from the command’s standard output (or sometimes standard error). To grab the output of commands, we need to prepare the Process object, which means we cannot use the convenience method Process.run() any more.

The say command has an option to change the voice being used. You can list the names of voices available to the say command by running say -v '?' in Terminal. We’d like to get that output and populate a menu with the available voices in our UI.

Note: the output of the say -v '?' command does not show most of the modern voices available in macOS. Most of the voices listed are quite old, funny, and, to be honest, awful. As has been mentioned before, I am using the say command as a convenient example for a shell command. If you want proper speech synthesis in your app, you should not use the say command but the proper speech synthesis API.

We will start out experimenting with code in a macOS Playground in Xcode and later add it to the app from the first part. When you create a new Playground in Xcode (File > New > New Playground), make sure you select ‘macOS’ for the platform above the template picker and then ‘Blank’ as the template.

Start with this code:

let process = Process()
process.launchPath = "/usr/bin/say"
process.arguments = ["-v", "?"]
let outPipe = Pipe()
process.standardOutput = outPipe
process.terminationHandler = { process in
  let outData = outPipe.fileHandleForReading.readDataToEndOfFile()
  let output = String(data: outData, encoding: .utf8) ?? ""
 print(output)
}
try? process.run()
process.waitUntilExit()

You will recognize some of the code from the first part of this tutorial. We create a Process object and set its launchPath and arguments. Note that the Process.run() convenience method, which we used in the first part, takes a URL to define the executable, but the launchPath is a String containing the path to the executable. We also (this is the new part) create a Pipe and set it as the standardOutput of the process.

The process will run the say command asynchronously. Since we want to work with the output , we have to give the process some code to execute when the command is done. We set this terminationHandler to a closure. The code in this closure grabs the data from pipe we connected to standardOutput of the process, converts it to a String and prints it.

At the end of our code, we tell the process object to run(). The waitUntilExit() is necessary here, because that is all this particular playground code does and we want to be sure that the command and the termination handler get a chance to do their work, before the code ends. In an app, where many things may be happening at the same time, you usually will not use waitUntilExit(). We will see that later when we implement our solution in SwiftUI.

Many outcomes

This code grabs the standard out from the command. Unix commands have different results that can be interesting. Some commands print to standard error instead of (or together with) standard out. All unix commands also have an exit code, which should return 0 for successful runs and a non-zero value for failures. Some commands use different non-zero value to give information for different errors.

All of this is available from Process objects. However, you have to set up the pipes and connections and get the data back in the termination handler, and the code gets quite complicated and tedious to set up. We have the seen the Process type has a convenience method to create and run a command without the pipes. I don’t know why it doesn’t have a convenience method when you are interested in all the data, but the good news is, we can create an extension to build our own.

Replace the code in the playground with the following:

import Foundation

extension Process {
  @discardableResult
  static func launch(
    path: String,
    arguments: [String] = [],
    terminationHandler: @escaping (Int, Data, Data) -> Void
  ) throws -> Process {
    let process = Process()
    let outPipe = Pipe()
    let errorPipe = Pipe()
    process.standardOutput = outPipe
    process.standardError = errorPipe
    process.arguments = arguments
    process.launchPath = path
    process.terminationHandler = { process in
      let outData = outPipe.fileHandleForReading.readDataToEndOfFile()
      let errorData = errorPipe.fileHandleForReading.readDataToEndOfFile()
      let exitCode = Int(process.terminationStatus)
      terminationHandler(exitCode, outData, errorData)
    }
    try process.run()
    return process
  }
}

let process = try? Process.launch(
  path: "/usr/bin/say", 
  arguments: ["-v", "?"]
) { exitCode, outData, errData in
  let output = String(data: outData, encoding: .utf8) ?? ""
  print(output)
}
process?.waitUntilExit()

You will recognize the code from our first example in the launch function in the extension. But here, we have added the configuration for a second Pipe for standardError, in the closure we get the Data for the standard out, standard error and the exit code and pass them in to the closure passed in a termination handler.

This method simplifies using the Process type. We just have to pass in the path to the command and the arguments array and give a closure that is called when the command completes.

async/await

But it still uses a closure for the termination handler. This breaks our code into different fragments that are not executed in the order they appear in. Since macOS Monterey 12 and iOS 15, Swift has had a concurrency feature called async/await. With this, your code appears in a more natural order.

When a function has the await marker, the system knows to suspend the code at that point until the function returns a result. While this code is “on hold” other threads or tasks can run, such as the UI handling, so your process or app isn’t blocked. When the function returns, the code continues after the function, so the code that processes the output of the function comes in the logical order, which makes it easier to read and understand.

For some reason, the Process type has not yet been updated to use this new feature. We can however, add this functionality using an extension. Add this method to the extension:

  static func launch(
    path: String,
    arguments: [String] = []
  ) async throws -> (exitCode: Int, standardOutput: Data, standardError: Data) {
    try await withCheckedThrowingContinuation { continuation in
      do {
        try launch(path: path, arguments: arguments) { exitCode, outData, errData in
          continuation.resume(returning: (exitCode, outData, errData))
        }
      } catch let error {
        continuation.resume(throwing: error)
      }
    }
  }

If you want to get more detail how async and await work, I recommend the WWDC session “Meet async/await in Swift” from WWDC 2021.

With this, we can change our code to run the say command to:

guard let (exitCode, outData, errData) = try? await Process.launch(
    path: "/usr/bin/say",
    arguments: ["-v", "?"]
) else { exit(0) }

let output = String(data: outData, encoding: .utf8) ?? ""
print(output)

The code is now in a more sensible order. If you can afford macOS Monterey as a minimum system requirement, you should consider adopting async/await.

Now that we have our output of the command, we have to parse out the names of the voices. This code will turn the output of the say into an Array of names, ignoring the language code and sample text:

func parseVoices(_ output: String) -> [String] {
  output
    .components(separatedBy: "\n")
    .map {
      $0
        .components(separatedBy: "#")
        .first?
        .trimmingCharacters(in: .whitespaces)
        .components(separatedBy: CharacterSet.whitespaces)
        .dropLast()
        .filter { !$0.isEmpty }
        .joined(separator: " ")
      ?? ""
    }
    .filter { !$0.isEmpty }
}

So we can add these lines to get an Array of `voices:

let voices = parseVoices(output)
print(voices)

You can find the stages of code for the playground in this gist.

Updating the app

Now that we have assembled all the pieces working in a playground, we can move on to putting these pieces in our SwiftUI app. The goal is to have a popup menu with all the voices above the field where you can enter text.

We will be using some features that were introduced in macOS Monterey 12.0. If you have built the project a while ago with an older version of macOS, the project may be set to build with older versions of macOS. To verify and, if necessary, update the deployment target for the project, select the blue project icon at the very top of the item sidebar, then select the gray “SayThis” icon under the “Targets” section. Then select “General” in the tab list. The second section is called “Minimum Deployments” and should show macOS 12.0 or later. When you change this from an older version, Xcode may prompt to update some other settings in the project, you can accept those.

First, we want to use the extension to the Process type we created in the playground earlier. Create a new file in the project (File > New > New File…), select “Swift File” from the templates, and name it ‘Process-launch’. Copy and paste the extension code from the playground (with both methods) to that file. You could have all the code in a single file, but it is cleaner and more maintainable to have one file per type or extension. This also allows you to copy that file to other projects to use it there.

Next we need two more state variables to track the state of the popup-menu that we are going to add. In the ContentView file, add these two lines right under the existing @State variables:

  @State var voices: [String] = []
  @State var selectedVoice: String = "Samantha"

The first is where we are going to store all the available voices that we parse from the output of say -v '?'. The second variable will store the current selection.

Next, we will insert the popup menu for the voices into the UI. In the body, insert this below the title and above the HStack:

      Picker(selection: $selectedVoice, label: Text("Voice:")) {
          ForEach(voices, id: \.self) { Text($0) }
      }

First we create a Picker view, which is not really proper terminology for macOS. But since SwiftUI is designed to share objects across all platforms, a ‘Picker’ will render as a popup menu in macOS. We attach the selectedVoice state variable to the selection of the picker. Then, we loop through all the items in the voices state variable to add them to the picker.

At this point, you should see the popup menu in the preview and when you run the project. You cannot select anything from it, though, which makes sense as the voices array is empty.

We need to populate the voices array before the UI appears. SwiftUI offers a special modifier to do this. You can attach a .task modifier to any view and its closure will be run asynchronously once, before the view appears for the first time. Add this code to the end of the VStack block, right below the line that reads .frame(maxWidth: .infinity, maxHeight: .infinity):

    .task {
      guard let (_, outData, _) = try? await Process.launch(
        path: "/usr/bin/say",
        arguments: ["-v", "?"]
      ) else { return }

      let output = String(data: outData, encoding: .utf8) ?? ""
      voices = parseVoices(output)
    }

You also need to copy the parseVoices() function from the test playground and add it to the ContentView struct.

This will populate the popup menu with all the voices. Build and run to test. It will still not actual use the selected voice. We need to update the runCommand() method to:

  func runCommand() {
    let arguments = [message, "-v", selectedVoice]
    let executableURL = URL(fileURLWithPath: "/usr/bin/say")
    self.isRunning = true
    try! Process.run(executableURL, arguments: arguments) { _ in
      self.isRunning = false
    }
  }

We have achieved what we set out to do. You can find the project with all the code for this point here.

Refinements

However, there are some refinements and improvements that we can and should still add to the code.

First, the runCommand() function still uses the old convenience handler with a termination handler. We can change it to use our new await Process.launch():

  func runCommand() async {
    let arguments = [message, "-v", selectedVoice]
    self.isRunning = true
    let _ = try? await Process.launch(path: "/usr/bin/say", arguments: arguments)
    self.isRunning = false
  }

This will now generate an error in the Button because we have changed the runCommand() function to be async. Change the code of the Button to

        Button("Say") {
          Task {
            await runCommand()
          }
        }
        .disabled(isRunning)

By wrapping the await runCommand() function in a Task { } we are telling the system that this code should run in the background and not return immediately. This is necessary to run code blocks with async functions from UI elements.

The say command will use the voice set in the system settings when no -v option is given. We want to provide a way to recreate that behavior in our UI. We can add items to the Picker by adding elements before the ForEach:

      Picker(selection: $selectedVoice, label: Text("Voice:")) {
        Text("System Default").tag("System Default")
        Divider()
        ForEach(voices, id: \.self) { Text($0) }
      }

Also change the default value of the selectedVoice variable to "System Default".

Then we have to change the runCommand() method to only use the -v and voice name arguments when their value is not “System Default”

  func runCommand() async {
    var arguments = [message]
    if selectedVoice != "System Default" {
      arguments.append(contentsOf: ["-v", selectedVoice])
    }
    self.isRunning = true
    let _ = try? await Process.launch(path: "/usr/bin/say", arguments: arguments)
    self.isRunning = false
  }

You can find the code for this here.

Out of the Sandbox

The say command provides a fun example to play around with. You will probably already have plenty of ideas that you want to try out. There is one thing that might trip you up here: sandboxed apps do not get access to all available command line tools. If you want to run such ‘prohibited’ commands, you will need to remove the sandbox configuration from the app.

Better Extension

The extension I put together here will work in most situations, but there are conditions where different parts might not work well. Quinn the Eskimo has published this code which goes further and is probably “even more” thread safe.

Launching commands is an “expensive” task. You should always do some research to see if there is some framework that can give you access to the same functionality natively in Swift. Nevertheless, sometimes it might be necessary, and sometimes it might just be easier to call out to an external command.

This tutorial should help you along the way of using these tools effectively.

RAID and macOS

RAIDs are a strange edge case that are rarely useful outside of servers, but when they are useful, they are very important. RAID is an acronym for ‘Redundant Array of Independent/Inexpensive Disks.’ It is a technology where you combine multiple, physical disks into a single virtual drive for redundancy, speed, or both.

Often the RAID system is handled by a dedicated controller in an external enclosure, but sometimes, you want or need to work with drives directly. macOS has some basic RAID functionality built-in and there are good third party options if you want to go further.

RAID Levels

If you are unfamiliar with RAIDs, we need to get some terminology sorted out first. There are different kinds of RAIDs which are called ‘levels.’

In this post, I am going to focus on level 1 or ‘RAID 1.’ With RAID 1, the data is ‘mirrored’ between two or more drives, so that each drive carries a complete copy of the data. This protects from data loss because of drive failure. Note that RAID 1 does not protect from other common reasons for data loss, such as file system or individual file corruption, accidental or malicious file deletion. A RAID is never a replacement for a good backup strategy. Since the data is mirrored on all the devices, a RAID 0 will only have the capacity of the smallest drive in the set. It is generally recommended that drives in any RAID set should be of the same capacity and type, for best performance and efficiency.

A level 0 RAID (or ‘RAID 0’) is not actually redundant. In a RAID 0 the data is ‘striped’ across two or more drives so that writes and reads happen in parallel, which increases the data bandwidth available. Since the data is spread evenly (striped) across all drives in the RAID 0 set, failure of a single drive will result in complete data loss. The capacity of the stripe raid is the capacity of the smallest drive in the set multiplied by the number of drives in the set. Having drives of the same type and capacity is even more relevant for RAID 0 performance.

There are more RAID levels, such as 0+1, 10 and 5 and dedicated disk controllers will have more options (such as combining multiple drives of different sizes more efficiently), but we will focus RAID 1 (mirror).

Here, there be dragons

Warning: many of the commands shown here to setup and experiment with disk drives and RAIDs may or will lead to loss of the data on the drives involved, so be careful. I strongly recommend disconnecting any drives other than those you are experimenting with from the Mac you are working on.

I would also recommend to experiment with a set of drives that contain no relevant data at all. Two USB sticks will do just fine to explore and test the functionality. Drives do not have to be of the same capacity and type for testing, but I do recommend that for actual use.

Apple RAID

macOS has built-in support for software-based mirror and stripe RAID called “AppleRAID.” This also provides a third option to concatenate drives, but concatenation provides neither redundancy, nor performance, so I do not recommend using it.

You can use the Disk Utility app to setup a RAID. It has a nice assistant that you an access from the File menu called RAID Assistant. It will ask you what kind of RAID you want to setup and allow you to select the drives and create a new RAID volume. This will delete the data on the disk drives and there are few features that are not exposed in the Disk Utility UI, so I will focus on how to do it in the command line.

You can keep Disk Utility open to get a visual representation of what is going on, though the Disk Utility app often has problems keeping up with changes done from the command line. You may have to quit and restart the app to force it to update its status. You want to enable “Show all Devices” from the View menu to see the physical drives as well as the file systems and virtual drives.

First, we need to identify the disks that we want to work with. When you run diskutil list it will list all the disk on the system. Usually disk0 will be the built-in drive, and disk3 will be the (synthesized) APFS container inside (with the System and Data volume). But depending on what Mac you are using, what your configuration is, and what devices you had attached to the Mac before you started this, the numbers may be different.

Again, to be safe, unmount and disconnect any drives or file servers with data that you care about at this point, and the connect the two drives you want to use for experimentation. Their data will be erased!

Run diskutil list again and identify the device identifiers for the drives you will be working with. They should look like this:

/dev/disk4 (external, physical):

For me, the two drives where disk4 and disk5, so I will be using those numbers in my examples, but be sure to replace those with the device numbers on your system, other wise you might be working with the wrong disk or volume.

Promoting a drive to mirror RAID

One of the features you can use from the command line is to ‘promote’ an existing drive to a mirror RAID without data loss. Apple RAID promotion works (as far as I can tell) only with HFS+ formatted Volumes, so let us reformat the first disk (disk4) as such:

> diskutil eraseDisk JHFS+ DiskName disk4 
Started erase on disk4
Unmounting disk
Creating the partition map
Waiting for partitions to activate
Formatting disk4s2 as Mac OS Extended (Journaled) with name DiskName
Initialized /dev/rdisk4s2 as a 59 GB case-insensitive HFS Plus volume with a 8192k journal
Mounting disk
Finished erase on disk4

Using the command line, we will be able to promote this HFS+ drive without have to erase it (again), so copy some (un-important) files to it now.

diskutil has a sub-group of commands dedicated to the RAID functions called appleRAID or ar for short. I am going to use the short form. You can run diskutil ar to get a list of the sub-commands for working with Apple Raid. You can read the diskutil man page for details.

Next we have to enable AppleRAID on the drive. Enabling RAID on single drive seems a bit pointless, but this prepares everything on that drive, so that we can add more drives later.

> diskutil ar enable mirror DiskName
Started RAID operation on disk4s2 (DiskName)
Resizing disk
Unmounting disk
Adding a booter for the disk
Creating a RAID set
Bringing the RAID partition online
Waiting for the new RAID to spin up "8D05B6EB-DCFB-426D-885B-ED8C76DC2484"
Finished RAID operation on disk4s2 (DiskName)

The volume and the files you had copied earlier are still there. But the volume is now listed under “RAID Sets” in Disk Utility. You can see the single drive in the RAID Set in the UI there. You can also get this info in command line with

> diskutil ar list   
AppleRAID sets (1 found)
===============================================================================
Name:                 DiskName
Unique ID:            8D05B6EB-DCFB-426D-885B-ED8C76DC2484
Type:                 Mirror
Status:               Online
Size:                 63.8 GB (63816400896 Bytes)
Rebuild:              manual
Device Node:          disk6
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk4s2   467826B1-BBA2-4671-99CE-5CBB04E06882  Online     63816400896
===============================================================================

To make this a real mirror RAID, we need to add the second drive:

> diskutil ar add member disk5 DiskName                
Started RAID operation on disk6 (DiskName)
Unmounting disk
Repartitioning disk5 so it can be in a RAID set
Unmounting disk
Creating the partition map
Adding disk5s2 to the RAID Set
Finished RAID operation on disk6 (DiskName)

This will add the drive to the RAID. This will delete any data that might be on disk5!

When you look at the RAID in Disk Utility (you might have to restart the app for it to pick up the new status) you will now see both drives, but one of them has the status “Rebuilding” with a percentage. The status of the entire RAID set is now “Degraded.” The RAID system is still in the process of mirroring data to the new member. You can use the volume to read and write data at this time, but it is not redundant yet. If the first drive fails during rebuilding, the data will be gone.

Once the rebuilding is done, the status of the RAID will change to “Online,” which is the “good” status. At this point the data on the RAID will be resilient to the failure or removal of either of the drives.

You can also create the RAID with both drives from the start, but this will erase all the data on both drives (this is what RAID Assistant in Disk Utility does.

Before we can try the other way of creating a mirror RAID set, we need to “break” the mirror we have built so far.

> diskutil ar delete DiskName
Started RAID operation on disk6 (DiskName)
Unmounting volume for RAID set 8D05B6EB-DCFB-426D-885B-ED8C76DC2484
Destroying the RAID set 8D05B6EB-DCFB-426D-885B-ED8C76DC2484
Finished RAID operation on disk6 (DiskName)

If you waited for the rebuilding to be complete, both individual drives will each contain the data of the former mirror RAID. If the rebuilding was not complete yet, only the first drive will contain the data, the second drive will be empty.

Create a new RAID set

Now let’s build a new empty mirror RAID with both drives included from the start:

> diskutil ar create mirror DiskName APFS disk4 disk5
Started RAID operation
Unmounting proposed new member disk4
Unmounting proposed new member disk5
Repartitioning disk4 so it can be in a RAID set
Unmounting disk
Creating the partition map
Using disk4s2 as a data slice
Repartitioning disk5 so it can be in a RAID set
Unmounting disk
Creating the partition map
Using disk5s2 as a data slice
Creating a RAID set
Bringing the RAID partitions online
Waiting for the new RAID to spin up "FE3E7A3F-E4BF-4AEE-BC3C-094A9BFB3251"
Mounting disk
Finished RAID operation

Note that we can build an APFS volume on the RAID set with the command line tool. RAID Assistant in Disk Utility will build a HFS+ volume. In this new empty RAID set both drives will be online immediately.

When you replace the mirror with stripe in this command you will get a striped RAID 0 volume. (performance instead of redundancy)

Drive failure

We can simulate a drive failure by unplugging one of the members. Sadly, macOS does not seem to have a notification for this event. Once you have removed the drive you can run disktutil ar list and see the status is “Degraded” and the which member is “Missing/Damaged.” You can keep using the volume in degraded mode.

Once you plug the drive back in it will appear as ‘failed.’ You can start the process of repairing or rebuilding the mirror with

> diskutil ar repairMirror FE3E7A3F-E4BF-4AEE-BC3C-094A9BFB3251 disk5  
Started RAID operation
Unmounting disk
Repartitioning disk5 so it can be in a RAID set
Unmounting disk
Creating the partition map
Adding disk5s2 to the RAID Set
Finished RAID operation

Note: Syncing data between mirror partitions can take a very long time.
Note: The mirror should now be repairing itself.  You can check its status using 'diskutil appleRAID list'.

The UUID is the UniqueID of the RAID set you see with diskutil ar list. The warning you get at the end is fair. The rebuilding process will take a while. How it takes depends on how full the volume is and how fast the new member drive is.

Downsides of AppleRAID

There are quite a few downsides to the built-in AppleRAID functionality. There is no notification or warning when one of the drives in a mirror goes offline and the RAID is running in degraded state. The RAID will also not automatically rebuild when a missing drive re-appears. (There is an AutoRebuild option mention in man page, but whenever I tried to enable that, the entire disk management stack froze in a way that required a reboot.)

AppleRAID can be useful to quickly stripe some random disks for performance. But generally, if the data was of so much concern that I am considering RAID 1, I would not rely on AppleRAID.

SoftRAID

There is a wonderful third party tool for managing software based RAIDs on macOS called SoftRAID (14-day free trial, then a tiered license). And, much to my delight, it also comes with a command line tool. Creating a RAID is not something you do regularly, so I went ahead and did this in the GUI app. Once that was setup, I used the command line tool to get the RAID’s status:

> softraidtool volume DiskName info

Info for "DiskName":

Mountpoint: /Volumes/DiskName
BSD disk: disk4
Total Bytes: 59.6 GB (64,016,478,208)
Free Bytes: 59.6 GB (64,016,478,208)
Volume format: unknown
Volume is DiskName
RAID level: RAID 1
DiskName ID: 09DF05C72BFFAD20
Optimized for: Workstation
Created: Jul 17, 2023 at 3:33:39 PM
Last Validated: never
Volume state: normal, 
Volume Safeguard: enabled
Total I/Os: 5,610
Total I/O Errors: 0
Total number secondary disks (including offline ones): 1

Disks used for this volume:
bsd disk:    SoftRAID ID:         Location and Size:
disk7     09DF053ECE82F980     (USB3 bus 0, id 4 - 59.8 GB)  secondary disk, 
disk6     09DF053D23386500     (USB3 bus 0, id 5 - 59.8 GB)  primary disk, 

The SoftRAID software also comes with a menubar app that shows the status of the RAID.

When you unplug one of the drives, the ‘Volume state’ changes to ‘missing disk.’ When you plug the missing drive back in, SoftRAID will automatically detect it and rebuild the RAID, when necessary. Rebuilding went so quickly that I had a hard time capturing the state from the command line. The more changes you apply to the degraded RAID the longer the rebuild takes.

> softraidtool volume SoftRAID info
SoftRAIDTool status: waiting for disk5 to finish (00:00:01)

Info for "SoftRAID":

Mountpoint: /Volumes/SoftRAID
BSD disk: disk5
Total Bytes: 59.6 GB (64,016,478,208)
Free Bytes: 59.6 GB (64,016,478,208)
Volume format: unknown
Volume is SoftRAID
RAID level: RAID 1
SoftRAID ID: 09DF06D95024CBE0
Optimized for: Workstation
Created: Jul 17, 2023 at 3:53:16 PM
Last Validated: never
Volume state: rebuiding, out of sync, 
Volume Safeguard: enabled
Volume progress: 15%
Current offset: 7,343,685,632
Time remaining: 00:07:24
Total I/Os: 22,752
Total I/O Errors: 0
Total number secondary disks (including offline ones): 1

Disks used for this volume:
bsd disk:    SoftRAID ID:         Location and Size:
disk4     09DF053ECE82F980     (USB3 bus 0, id 4 - 59.8 GB)  primary disk, 
disk7     09DF053D23386500     (USB3 bus 0, id 5 - 59.8 GB)  secondary disk, rebuiding, out of sync, 

You can parse this output using awk to get just the volume state. This is useful for reporting the state to Jamf Pro with an extension attribute:

#!/bin/sh

# reports the SoftRAID status

softraidtool="/usr/local/bin/softraidtool"

if [ ! -x "$softraidtool" ]; then
    echo "<result>SoftRAID not installed</result>"
fi

volumestate=$(softraidtool volume SoftRAID info | awk -F ': ' '/Volume state/ {print $2}')

echo "<result>$volumestate</result>"

Keep in mind that Jamf Inventory Updates (aka as recon) may run very infrequently (recommended default is once per day, and it shouldn’t run much more often than that to avoid database bloat), so the data in your Jamf Pro may be hours or sometimes longer out of date. If you want to react to changes in the RAID status more quickly, you should rely on other tools than Jamf Pro.

Conclusion

The best solution for RAIDs will always be a dedicated hardware controller. But there also good reasons (cost) to just put together a “bunch of disks” into a RAID. The built-in AppleRAID functionality works, but has limitations, especially for mirror RAIDs. SoftRAID is a great tool to overcome these limitations. For Mac admins, both can be managed and monitored with command line tools, which allows automation and integration with your management system.

On viewing man pages (Ventura update)

I have written about man pages before. That post references an even older post, which is actually the second oldest post on this blog. Most of the recommendations in the posts still hold true, but there is one change relevant to macOS Ventura and one other thing that is worth adding.

Ventura’s Preview app lost the ability to render postscript or ps files. This breaks the previous, popular shell alias to open a man page in Preview. However, the amazing community in the #scripting channel of the MacAdmins Slack have figured out a replacement. They have asked me to share it.

Add this function to your shell configuration file: (bashzsh

preman() {
    mandoc -T pdf "$(/usr/bin/man -w $@)" | open -fa Preview
}

Then you can run preman <command> in Terminal and the man page will render beautifully in Preview. If you want to override the man command to actually use this function instead of the built-in command, add this alias:

alias man preman

If you then need to revert to the actual man command for a test of something, just add \ before the command: \man <command>

Update: Pico has expanded this into a full blown script which caches the pdfs.

Myself, I had not noticed this change, because I prefer opening the ‘yellow’ man pages in Terminal app. You can do so by entering a command in the Help menu, or by using the x-man-page URL scheme. In these yellow terminal windows, you can scroll and search in the text with command-F. You can also do a secondary click (right/ctrl/two-finger click) on any word and it will offer to open that man page in the context menu.

If the yellow man page windows annoy you, you can change their appearance by modifying Terminal’s “Man Page” profile. I modify it to use my favorite mono spaced font at a larger size. I like the yellow background because it stands out.

In the previous posts, I had a simple function you could add to your shell configuration files, but I have since refined this to also support man page sections (the number you sometimes see after a command, you can learn what they mean in the man page for the man command).

xmanpage() {
  if [[ -z $2 ]]; then
    open x-man-page://"$1"
  else
    open x-man-page://"$1"/"$2"
  fi
}

With this, you can open the desired section in the ‘yellow’ man window with xman 2 stat. This works already with the preman function. The mandoc command knows how to deal with the extra argument, the URL scheme needs a bit of extra work.

If you want to override the normal man command you can, again, use an alias in your shell configuration file:

alias man xmanpage

This way, I can have both functions in the configuration file and choose or change which function (if any) overrides the normal man by just changing the alias.

When you work in Terminal for a while you may accumulate a lot of yellow man page windows. You can use this AppleScript/osascript one-liner to close all Terminal windows which use the ‘Man Page’ profile at once.

osascript -e 'tell application "Terminal" to close (every window where name of current settings of every tab contains "Man Page")'

And while this one-liner is succint, it is still easier to wrap this in a function for your shell configuration file:

closeman() {
  osascript -e 'tell application "Terminal" to close (every window where name of current settings of every tab contains "Man Page")'
}

If you enjoyed this excursion into macOS Terminal and command line tricks and configuration, you might like one of my books: “macOS Terminal and shell” and “Moving to zsh”.

Launching Scripts #4: AppleScript from Shell Script

In the last post, we discussed how to run shell commands and scripts from an Apple Script environment. In this post, we will look at how we can run AppleScript commands and scripts from the shell environment.

Open Scripting Architecture

The key to running AppleScript from the shell is the osascript command. OSA is short for ‘Open Scripting Architecture’ which is the framework that powers AppleScript. This framework allows AppleScript to have its native language, but also use JavaScript syntax.

The osascript command allows us to run AppleScript commands from Terminal and shell. The most common use is the user interaction commands from AppleScript, like display dialog:

osascript -e 'display dialog "Hello from shell"'

The -e option tells osascript that it will get one or more lines of statements as arguments. The following argument is AppleScript code. You can have multiple -e options which will work like multiple lines of a single AppleScript:

> osascript -e 'display dialog "Hello from shell"' -e 'button returned of result'
OK

osascript prints the value of the last command to stdout. In this case, it is the label of the button clicked in the dialog. (The ‘Cancel’ button actually causes the AppleScript to abort with an error, so no label will be returned for that.)

When you have multiple lines of script, using multiple -e statements will quickly become cumbersome and unreadable. It is easier to use a heredoc instead:

osascript <<EndOfScript
   display dialog "Hello from shell"
   return button returned of result
EndOfScript

This also avoids the problem of nested quotation marks and simplifies shell variable substitution.

Shell variables and osascript

There are a few ways to pass data into osascript from the shell.

Since the shell substitutes variables with their value before the command itself is actually executed, this works in a very straightforward manner:

computerName=$(scutil --get ComputerName)

newName=$(osascript -e "text returned of (display dialog \"Enter Computer Name\" default answer \"$computerName\")")

echo "New Name: $newName"

This works well, but because we want to use shell variable substitution for the $computerName, we have to use double quotes for the statement. That means we have to escape the internal AppleScript double quotes and everything starts to look really messy. Using a heredoc, cleans the syntax up:

computerName=$(scutil --get ComputerName)

newName=$(osascript <<EndOfScript
    display dialog "Enter Computer Name" default answer "$computerName"
    return text returned of result
EndOfScript
)

echo "New name: $newName"

I have a detailed post: Advanced Quoting in Shell Scripts.

Environment Variables

Generally, variable substitution works well, but there are some special characters where it might choke. A user can put double quotes in the computer name. In that case, the above code will choke on the substituted string, since AppleScript believes the double quotes in the name end the string.

If you have to expect to deal with text like this, you can pass data into osascript using environment variables, and using the AppleScript system attribute to retrieve it:

computerName=$(scutil --get ComputerName)

newName=$(COMPUTERNAME="$computerName" osascript <<EndOfScript
    set computerName to system attribute "COMPUTERNAME"
    display dialog "Enter Computer Name" default answer computerName
    return text returned of result
EndOfScript
)

echo "New name: $newName"

The shell syntax

VAR="value" command arg1 arg2...

sets the environment variable VAR for the process command and that command only. It is very useful.

Retrieving environment variables in AppleScript using system attribute is generally a good tool to know.

Interpret this!

osascript can also work as a shebang. That means you can write entire scripts in AppleScript and receive arguments from the shell. For example, this script prints the path to the front most Finder window:

#!/usr/bin/osascript

tell application "Finder"
    if (count of windows) is 0 then
        set dir to (desktop as alias)
    else
        set dir to ((target of Finder window 1) as alias)
    end if
    return POSIX path of dir
end tell

You can save this as a text file and set the executable bit. I usually use the .applescript extension.

> print_finder_path.applescript
/Users/armin/Documents

To access arguments passed into a script this way, you need to wrap the main code into a run handler:

#!/usr/bin/osascript

on run arguments
    if (count of arguments) is 0 then
        error 2
    end if
    return "Hello, " & (item 1 of arguments)
end

You can combine this into a longer script:

macOS Privacy and osascript

When you ran the above script, you may have gotten this dialog:

If you didn’t get this dialog, you must have gotten it at an earlier time and already approved the access.

AppleEvents between applications are controlled by the macOS Privacy architecture. Without this, any process could use AppleEvents to gather all kinds of data from any process. These dialogs are easy enough to deal with when running from Terminal. But if you put your AppleScript code (or shell scripts calling AppleScript) into other apps or solutions, it could get messy quite quickly.

Mac Admins generally want their automations to run without any user interactions. You can avoid these dialogs by creating PPPC (Privacy Preferences Policy Control) profiles that are distributed from an MDM server. In this case you have to pre-approve the application that launches the script, which can sometimes also be challenge. The other option is to find solutions that avoid sending AppleEvents altogether.

I have a longer post detailing this: Avoiding AppleScript Security and Privacy Requests

osascript and root

Management scripts often run as a privileged user or root. In this case, certain features of AppleScript may behave strangely, or not at all. I generally recommend to run osascript in the user context, as detailed in this post: Running a Command as another User

Conclusion

AppleScript’s bad reputation may be deserved, because its syntax is strange, and often very inconsistent. Nevertheless, it has features which are hard to match with other scripting languages. You can use the strategies from this and the previous posts to combine AppleScript with Shell Scripting and other languages to get the best of both worlds.

Launching Scripts #3: Shell scripts from AppleScript

In this series of posts, I am exploring the various different ways of launching scripts on macOS. In the first two posts, we explored what happens when you launch scripts from Terminal. We already explored some concepts such as the shell environment and how that affects scripts. In this post we are going to explore a different, but very common way to launch shell commands and scripts: AppleScript’s do shell script command.

do shell script

When AppleScript made the transition from Classic Mac OS 9 and earlier to Mac OS X, it gained one command that allowed AppleScripts to interact with Mac OS X’s Unix environment. The do shell script command executes a command or script in the shell and returns the output as a text to the AppleScript.

We have used this in an earlier post:

do shell script "echo $PATH"
    --> "/usr/bin:/bin:/usr/sbin:/sbin"

But you can use this to run any shell command:

do shell script "mdfind 'kMDItemCFBundleIdentifier == org.mozilla.firefox'"
    --> "/Applications/Firefox.app"

Note the use of single quotes inside the double quotes which delineate the AppleScript text. I have a post on the challenges of quoting in these mixed scripting environments.

You can assemble the command you pass into do shell script using AppleScript text operators:

set bundleID to "org.mozilla.firefox"
do shell script "mdfind 'kMDItemCFBundleIdentifier == " & bundleID & "'"
    --> "/Applications/Firefox.app"

Note that the PATH variable for AppleScripts that are run from Script Editor or as an AppleScript applet is different than the PATH in your interactive environment. Most notably, it does not include /usr/local/bin. When you want to use a command or script that is not stored in the four default directories, you will have to use the full path in the do shell script:

do shell script "/usr/local/bin/desktoppr"

(Desktoppr is a small tool I built to work with desktop pictures on macOS, you can get it here.)

When you are unsure what the full path to a command is, you can use the which command in Terminal:

> which desktoppr
/usr/local/bin/desktoppr

Errors

Keep in mind that which uses the same logic to lookup a command as the shell does when it looks up a command with no path. So, if you think you can trick AppleScript by using the which command to lookup a non-standard command, it will still fail:

do shell script "which desktoppr"
    --> error "The command exited with a non-zero status." number 1

When the command in do shell script returns a non-zero exit code, you will get an interactive dialog informing the user of the error. The AppleScript will not continue after the error. You can handle the error the same way you would handle any AppleScript error, with a try… on error block:

set filepath to "/unknown/file"

try
    do shell script "/usr/local/bin/desktoppr" & quoted form of filepath
on error
    display alert "Cannot set desktop picture to '" & filepath & "'"
end try

Files and Paths

AppleScript has its own methods of addressing files and folders. Actually, there are multiple ways, which is one of the confusing things about AppleScript. Neither of the native forms of addressing files and folder in AppleScript use the standard Unix notation with forward slashes separating folders. But there are built-in tools to convert from Unix notation to AppleScript and back.

Use the POSIX path attribute to get a Unix style file path from an AppleScript file or alias. Unix style paths used with commands need spaces and other special characters escaped. You can use the quoted form attribute to escape any AppleScript string. This is usually used directly with POSIX path:

set imagefile to choose file "Select a Desktop"
    --> alias "Macintosh HD:Library:Desktop Pictures:BoringBlueDesktop.png"
set imagepath to quoted form of POSIX path of imagefile
    --> '/Library/Desktop Pictures/BoringBlueDesktop.png'
do shell script "/usr/local/bin/desktoppr " & imagepath

You can convert a Unix style file path into an AppleScript file with the POSIX file type:

set bundleID to "org.mozilla.firefox"
set appPaths to do shell script "mdfind 'kMDItemCFBundleIdentifier == " & bundleID & "'"
    --> "/Applications/Firefox.app"

if (count of paragraphs of appPaths) = 0 then
    display alert "No app found"
else
    set appPath to first paragraph of appPaths
    -- convert the path to an AppleScript file
    set appFile to POSIX file appPath
        --> file "Macintosh HD:Applications:Firefox.app:"
    tell application "Finder" to reveal appFile
end if

Shell Scripts in AppleScript Bundles

Sometimes you are writing an AppleScript and want to use a script for some functionality which is difficult to achieve in AppleScript. When you have an AppleScript file and a shell script file that work together this way, you want to store them together and also have an easy way for one to get the location of the other.

For this, AppleScript provides the notion of script bundles and AppleScript applets (which are also bundles). A script bundle is not a flat file, but a folder which contains the script itself and other resources, such as script libraries, or shell scripts. The script can easily locate items in its bundle and call them.

For example, we want a script that needs to unzip a file. We can use the unzip command line tool to do that, but in my experience it is better to use ditto. The ditto expansion seems to be closer to how the expansion from Archive Utility works and do better with extended attributes and resource forks and other macOS specific things.

This is the shell script for our example:

#!/bin/sh

# target dir for expansion
targetdir="/Users/Shared/Script Bundle Demo"

# sanity checks for argument 1/filepath
if [ -z "$1" ]; then
    exit 2
fi

filepath=$1

# is it a file?
if [ ! -f "$filepath" ]; then
    exit 3
fi

# note: ditto seems to work better than unzip
ditto -x -k "$filepath" "$targetdir"

This is simple enough that you could just do it in a one-line do shell script, but I want to keep it simple. You could extend this shell script to use different tools to expand different types of archives, such as xar, tar, aa etc. If you want a more complex script, feel free to build one!

Now we can build the AppleScript/shell script combination. Open Script Editor, create a new script and save it right away. In the save dialog, change the ‘File Format’ to ‘Script bundle’ before saving.

After you have saved the Script as a bundle, you can see the Bundle Info in a pane on the right side of the script window. If you don’t see this pane, choose ‘Show Bundle Contents’ from the ‘View’ menu, or click the right most icon in the tool bar.

In this pane, you can set the name, identifier, version and some other data for the script bundle. You can also see a list of the ‘Resources’ which shows the contents of the Contents/Resources folder in the script’s bundle. When you find the script you save, you will see it has a scptd file extension and when you open the context menu on it in Finder, you can choose ‘Show Package Contents’ and dig into the bundle contents.

Note: AppleScript applications (or applets) work the same way. Their .app bundles have a few more sub folders, but the Resources work the same way. The difference is that AppleScript applets work on double-click, drag’n drop, and some other events that we will get to in later posts. Script bundles have to run from Script Editor.

Save the shell script from above into the script bundle’s Resources sub-directory with the name unarchive.sh. You should see it appear in the ‘Resources’ list in the script window.

This way, the AppleScript bundle can contain all the resources it might need, including shell (or other) scripts.

Now we still need to find a way to access the Resources from the script. To run our shell script, add the following code to the AppleScript in Script Editor:

-- Script Bundle Demo
set theArchive to choose file "Select a zip archive:" of type {"zip"}
set archivePath to quoted form of POSIX path of theArchive

-- assemble command
set scriptPath to quoted form of POSIX path of my (path to resource "unarchive.sh")
set commandString to scriptPath & space & archivePath

-- for debugging 
log (commandString)

do shell script commandString

First we prompt the user to choose a file with a zip extension, and the we convert the result into a quoted Unix path.

Then, we use the path to resource "unarchive.sh" to get the path to our shell script in the Resources folder in the bundle. Then we get the quoted Unix notation, and assemble the command for the do shell script. The log command there will print the commandString to the output in the script window and is useful for debugging. Then we run the command with do shell script.

Environment for do shell script

Our example script expands the archive into a subfolder of /Users/Shared. If you wanted to use a different location, you could use a second argument in the script.

There is a different way of passing data into scripts and that is environment variables.

First of all it is important to note that the shell environment for commands and scripts run with the do shell script command from an AppleScript in Script Editor or an AppleScript application is very different from the shell environment in an interactive shell in the Terminal. We have already seen that the PATH environment variable has a different value, which influences the command lookup behavior.

You can check the environment variable by running the env command. This will list all environment variables. (To be nitpicky, there is more to a shell environment than just the env variables, there are also shell options, but those will be different for each shell, sh, bash or zsh, anyway.)

do shell script "env"
    --> "SHELL=/bin/zsh
TMPDIR=/var/folders/2n/q0rgfx315273pb4ycsystwg80000gn/T/
USER=armin
COMMAND_MODE=unix2003
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/usr/bin:/bin:/usr/sbin:/sbin
__CFBundleIdentifier=com.apple.ScriptEditor2
PWD=/
XPC_FLAGS=0x0
SHLVL=1
HOME=/Users/armin
LOGNAME=armin
_=/usr/bin/env"

Interestingly, we have USER and HOME to use in this environment.

We can also add environment variables to a do shell script command:

do shell script "TARGET_DIR='/Users/Shared/Script Bundle Demo' " & scriptPath & space & filePath

You can use this to set the value of the TARGET_DIR env variable for the next command, which is our script in the script bundle.

Administrator Privileges

No matter which way you use do shell script, it has one big benefit. You can prompt the user to get the command or script to run with administrative privileges:

do shell script "systemsetup -getRemoteLogin" with administrator privileges

This will prompt for the user name and password of an administrator user. This can allow to build some simple workflows that require user interaction and administrative privileges really easily.

Conclusion

Combining Script Bundles and AppleScript Applications with shell scripts can create a powerful combination. You can use the “best of both worlds” with some of AppleScript’s user interaction commands and the shell’s strength in file manipulation and similar workflows. You can also sign AppleScript applications with a valid Apple Developer ID and pre-approve them for privacy exemptions with a PPPC profile.

If this explanation is not detailed enough for you, there is an amazing Tech Note in Apple’s Documentation Archive.

This post covered launching shell scripts from AppleScript. In the next post we will launch AppleScript code from shell scripts.

Launching Scripts #2: Launching Scripts from Finder

In this series of posts, I will explore the many ways that you can launch a script on macOS. In the previous, inaugural post, I described what happens when you launch a script from an interactive terminal shell.

There are several virtual terminal applications available for macOS. iTerm is very popular. Some text editors like Visual Studio Code and Nova, have terminals built-in. Since the actual launching of an executable is done by the shell running inside the virtual terminal, the launch process remains the same.

That said, Terminal app has a useful trick up its sleeves.

command file extension

When you change the file extension of a script to .command, double-clicking the file will open it in a new Terminal window and run it there. Any input or output the script requires will happen in that Terminal window. When the script exits, the shell session in the Terminal window will exit.

Let’s take this simple script:

#!/bin/sh
echo "Enter your name: "
read -r username
echo "Hello, $username"

When you put this this in a .command file and double-click it, you get a new Terminal window with:

/Users/armin/Desktop/hello_name.command ; exit;                                 
~ % /Users/armin/Desktop/hello_name.command ; exit;
Enter your name: 

You can see that Terminal opens a new window with a new, default shell and all your configurations, then launches the script right away. The script prints its output and then waits for the user input (the read command). When you enter the name at the prompt, the script continues.

Armin
Hello, Armin

Saving session...completed.

[Process completed]

When the script ends, the shell in the Terminal window exits, as well. No more interactive prompt will be shown.

This script expects user input in the Terminal and then presents output to stdout in the same window. While you could re-write a script to use AppleScript’s display dialog to handle both in the input and the output, it would make the script significantly more complex.

Instead, you can change the file extension to .command and then a double-click will create a new Terminal window where the user interaction (input and output) takes place. For the right kind of user and workflow, this can be a sufficient solution with practically no overhead.

You can also remove the file extension completely. The behavior when you double-click such a file in Finder will be the same. Extension-less executables also get a different the icon. Either way, you need to have the executable bit set for the script.

Note: iTerm can also open .command files, but I have had some trouble with user interaction in these cases. Since I usually don’t use iTerm, maybe I have something setup wrong?

Quarantine and Gatekeeper

A Terminal window is not a user interface that many users will appreciate, but this allows you share scripts with other users in a form they understand. “Double-click this to run” is something that fits with most users’ idea of how macOS works.

Before you start creating dozens of .command scripts and share them, there is a major tripwire that macOS security has set up.

When you share an executable file through a website, email or a chat message, macOS will attach a quarantine flag. With applications, this flag triggers a GateKeeper scan before the app is launched and it will show the standard dialog, even when the app is signed and notarized and a much more “scary” warning when it is not.

When you launch a command or script from Terminal, either directly or indirectly with a double-click, and it still has the quarantine flag set, it will not launch. You will not get one of the standard Gatekeeper dialogs. Just an opaque operation not permitted error in the Terminal output.

You can check if a file has the quarantine flag set with the xattr command:

> xattr hello_name.command
com.apple.TextEncoding
com.apple.lastuseddate#PS
com.apple.quarantine 

Your list list of extended attributes may be different. The quarantine flag has the label com.apple.quarantine.

The xattr command also can remove the quarantine flag:

> xattr -d com.apple.quarantine hello_name.command

Apple seems to assume that when you are using Terminal, you know what you are doing. That means you can bypass most of the mechanisms that attach a quarantine flag with command line tools. When you download something with curl it won’t get quarantined. You can install an unsigned, unnotarized pkg installer using the installer command. Because this is possible, it doesn’t mean it is always wise. Piping a curl command directly into sh or bash or any interpreter is still poor security.

Most of the time though, a script file shared to another Mac or another user will almost certainly get the quarantine flag. Users who are comfortable with using Terminal should be able to use the xattr command to disable this protection, but this is not something for ‘normal’ users. So quarantine, makes the use of the command file extensions far less effective than it could be. This is probably intentional, since executables that are opened by double-click can be an easy way to sneak malware and other unwanted software onto a system, leveraging a user’s ignorance of what is happening.

This is generally true when you move scripts and executable files between macOS systems. I have also seen the quarantine flag getting set when you store a script in a cloud sync service (especially iCloud) or when you edit an executable with a sandboxed application.

One way around the quarantine, would be to distribute and properly install the scripts with an installer pkg. Then you can properly sign and notarize the installer pkg and the scripts will not be quarantined, as they come from a trusted and verified source. This may be a good solution for some workflows, but generally feels a bit “over-designed.”

Output flashes by so quickly

There is a setting in Terminal’s preferences which determines what happens with windows when the shell exits. You can find it under the ‘Shell’ tab in the ‘Profiles’ area. This setting can be different for each profile. Under ‘When the shell exits’ there is a popup menu with the options ‘Close the window,’ ‘Close if the shell exited cleanly,’ and ‘Don’t close the window.’ The last ‘Don’t close’ is the default.

When you have this option set to ‘Close the window,’ the new Terminal window from a command file might only be active and visible for a short time. This may or may not be a good thing, depending on what you want.

Conclusion

Script files with the command file extension can be a simple, straightforward way to make scripts easily ‘launchable’ from Finder. You can also put them in the Dock or in the Login Items. The user experience is, well, a terminal, so not terribly nice, but it can be useful, and does not require any modification of the script.

When you share executable scripts, whether they have the command file extension or not, Gatekeeper quarantine on macOS can prevent the script from running. You should get familiar with the quarantine flag and the xattr command to manipulate it.

In the next post, we launch shell scripts from AppleScripts.]

On env Shebangs

There was a comment to my previous post about using the /usr/bin/env shebang to gain system portability.

Regarding shebang, a more system independent way would be to use e.g. ‘#!/usr/bin/env bash’. With this the script would also be usable on FreeBSD, where bash would be installed from FreeBSD Ports and then be available as ‘/usr/local/bin/bash’ or some Linux systems (e.g. Debian) where it is ‘/usr/bin/bash’. Unfortunately there are some other unixode systems around, where ‘env’ is not in ‘/usr/bin/’ and so the shebang needs to be adjusted.

(I replied to the comment there, but then realized this deserves its own post.)

The /usr/bin/env shebang provides a means for system portability. It has many valid use cases. However, you don’t just magically gain portability by switching to an env shebang. There are many trade-offs to consider.

The note on how the env binary may not be in /usr/bin on all platforms, hints at some of these trade-offs, but there are more.

The trade-offs are a loss of predictability and reliability, or functionality, or increased maintenance and management.

Let me elaborate…

How the /usr/bin/env shebang works

When used in the shebang, the /usr/bin/env binary will use the current environment’s PATH to lookup the interpreter binary for the script, in the same way the shell looks up commands.

As an example, let us imagine you have installed bash v5 on your Mac. Either manually or using brew or some other package management system.

This will (usually) put the bash v5 binary at /usr/local/bin/bash. The /usr/local/bin directory is a common choice for custom command line tools, because it is part of the default PATH for interactive shells on macOS and not protected by SIP/SSV. The default PATH on macOS for interactive shells is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Some installations will put the binary in a different location in the file system. Then you you have to pre-pend the directory containing the binary to your PATH variable in your shell configuration. The order of the directories in the PATH is important, because the shell and env will stop the search when they find the first match. If /usr/local/bin came after /bin in the PATH the new binary would not be ‘seen’ since the pre-installed, old /bin/bash binary is found first.

Some installations solve this by placing a symbolic link to the binary in /usr/local/bin.

When you run a script from the interactive shell with a shebang of #!/usr/bin/env bash, then env would find the bash v5 binary first in /usr/local/bin, so your script is interpreted with bash v5. This is probably what you were hoping for, when you installed bash v5 in the first place.

When you run the same script on a different Mac (same macOS version, but it doesn’t have bash v5 installed) env will pick up /bin/bash. Your script will work even though that other Mac doesn’t have /usr/local/bin/bash, so you gained portability.

However, /bin/bash is bash v3.2, so your script may behave differently. If the script uses bash v5 features that are not available in the 15-year-old bash v3.2, it will generate errors. Since you actively chose to install bash v5 on the first Mac, it is likely you needed some of these bash v5 features, so it is likely your script will fail on other Macs, which don’t have bash v5 installed.

You lost either predictability and reliability (which version and features are available? Does my script run successfully?), or you lose functionality (the features added to bash v5 since v3.2). To retain reliability, you can restrict the script to features that work in both bash versions. But then using the env shebang gives you no advantage, and you might as well use /bin/bash as the shebang.

Some solutions

One alternative is to use a /usr/local/bin/bash shebang for scripts which use bash v5 functionality and continue to use /bin/bash for scripts that need to run across multiple Macs, where you pay attention to using only features available in bash v3.2. You gain predictability and reliability, but your bash v5 scripts aren’t portable to other Macs. They may even fail on other Macs with bash v5 installed, if the bash v5 binary is installed in a different location.

When you use /usr/bin/env bash for a bash v5 script, it will run fine on all Macs which have bash v5 installed and the PATH configured properly to find it. (Configuring and maintaining the PATH does not happen on its own.) But the script will still fail on Macs without any bash v5. You can (and probably should) add a version check to the script, but now you are increasing code maintenance.

When you are managing a fleet of Macs, you also have the option (or in this case, I would say, the duty) to install bash v5 in a consistent location and version across all Macs in your fleet and pre-configure the proper PATH in the contexts the script will run in. Then you get predictability and functionality, but it requires extra effort in deployment and maintenance.

This requires a decently experienced MacAdmin and the proper tooling, neither of which comes for free.

Note: There are great open source solutions for macOS in this area, but I consider them ‘free, as in puppy,’ so they come with higher skill requirements and/or maintenance effort for the admin. And this isn’t supposed to imply that all commercial solutions are ‘easy to use,’ either. It’s trade-offs all the way down.

Context changes the PATH

Notice that so far I kept talking about “the default PATH for the interactive shell.”

The PATH variable may be different depending on the context, even on the same Mac with the same user. For example, when you run your script with the AppleScript do shell script command, the PATH in that context is not the same as the PATH in your interactive shell. It will be:

/usr/bin:/bin:/usr/sbin:/sbin

You can verify this by opening Script Editor and running the do shell script "echo $PATH". Other context, like scripts in installation packages will see other PATH values.

Most importantly, the PATH in these other contexts, does not contain /usr/local/bin, or any other addition you made to your PATH in the shell configuration files. An /usr/bin/env shebang will not ‘see’ a bash 5 binary you installed on the system. The same script with the same user on the same computer, will behave differently when run in a different context.

These different PATH values are an intentional choice. In these contexts, especially installation package scripts, reliability and predictability are extremely important. You do not want user and third-party installed custom binaries to interfere with the command lookup.

Sidenote on Python

With python and python3 and other run time interpreters, it gets even more difficult. There may multiple different versions installed and the behavior and functionality between versions varies more. My Mac currently has four different Python 3 binaries, each with a different version, and I not even remotely a full-time Python developer. When you call python3 on a non-developer Mac it will trigger the ‘You have to install Developer Command Line Tools’ dialog when Xcode is not installed. (Developers seem to have a hard time considering that there are Macs without Xcode installed.)

With the demise of python 2 in macOS 12.3, some developers reacted by changing the shebang in their python scripts from /usr/bin/python to /usr/bin/env python which solves nothing, when the binary goes away without replacement. Some switched to /usr/bin/env python3 which can makes things worse, by triggering the Developer Tools installation or picking a random python3 binary of the wrong version.

The only reliable solution for the mess that is python is to deploy a consistent version of Python 3 in a consistent location. You can do this by either bundling the python framework and module your tool needs together with the tool, or by deploying and maintaining the Python frameworks and modules with a management system.

MacAdmin perspective

As a MacAdmin, my scripts don’t need to be portable to systems other than macOS. They usually use tools and access files and folders that only exist on macOS. Predictability and reliability, however, are paramount. Configuration and installation scripts need to run reliably on thousands of Macs across multiple versions of macOS (even future versions) regardless of what else is installed.

As MacAdmins, we also (should) have the tools and experience to deploy and maintain binaries in predictable locations. But then, like everyone, we only have limited time, and often need to prioritize business critical software over our own tooling. So, the pre-installed interpreter binaries have little ‘friction’ to use, even if they may have a reduced functionality when compared to the latest version available elsewhere.

This is the reason bash v3.2 is still present on macOS 12.3 and it will never be easy when Apple ultimately decides to remove it. So many tools and scripts rely on /bin/bash.

(I don’t expect the removal to be any time soon, but there is a limit to how long Apple will or can keep this interpreter from 2007 on the system. We got the first warning when Apple switched the default interactive shell to zsh in Catalina. There will be more warnings …I hope. With the removal of the Python 2 binary we saw that Apple can move quickly when they feel the need. They did not even wait for a major macOS release.)

In this context, there is no gain in using /usr/bin/env. The trade-offs favor the absolute shebang very strongly.

Cross-platform portability

After this rant, you may think that I recommend against using /usr/bin/env shebangs always. But there are very good use cases. Using /usr/bin/env shebangs is the solution for workflows where cross-platform portability is required.

When your scripts need to run across multiple platforms, installing the binaries in the same location in the file system may not be possible, or require an unreasonable effort. For example, the /bin and /usr/bin are protected by SIP and the Sealed System Volume on macOS, so you cannot add your tooling there without significantly impacting the integrity and security of the entire system.

In these cases, using a /usr/bin/env shebang provides the required flexibility, so your scripts can work across platforms. But the portability does not come magically from just switching the shebang.

The target platforms need the binary to be installed and the versions should match. The installation location of the binary has to be present in the PATH in the context the script runs in. To get reliable behavior, the systems you are porting between need to be well managed, with a predictable setup and configuration of the interpreter binary and environment.

When your scripts work ‘effortlessly’ across systems with the env shebang, it is thanks to the work of the platform developers and your sysadmins/devops team for creating and maintaining this consistency. Even if you are the sole developer and devops admin, maintaining all the systems, you have to put in this work. Also the platform developers put in a lot of effort to achieve much of this consistency out of the box. As the commenter noted, some platforms don’t even agree where the env binary should be.

You gain portability at the price of increased maintenance.

Trade-offs all the way down

Alternatively, you can keep the scripts simple – restricted to a subset of features common across platforms and versions – so that the differences have no impact. Then you trade for reliability and portability at the price of functionality.

This is often the trade-off with Python (restrict the use of python features to those common among versions) and one of the reasons Python 2 was kept around for so long on macOS.

Using POSIX sh, instead of bash or zsh, is another option to gain portability, but that has its own trade-offs. Most of these trade-offs will be in functionality, but also consider not all sh emulations are equal, and supporting multiple different emulators or the real common subset of functionality, requires extra effort.

Conclusion

Shebangs with absolute paths have their strengths and weaknesses, as do shebangs with /usr/bin/env. Each has their use case and you have to understand the trade-offs when using either. Neither env shebangs nor absolute path shebangs are generally ‘better.’ Either one may get you in trouble when used in the wrong context.

When someone says ‘you should be using,’ or ‘a better way would be,’ you always need to consider their context, use case, and which trade-offs they are accepting. You need to understand the tools to use them efficiently.

Some CLI updates in macOS Monterey

The other day on Twitter, I got a question about a flag for the readlink command that I was not familiar with. As it turns out, the readlink command (which tells you where a symbolic link points to) got an update in macOS 12.3 and now has a -f option. With this new option, readlink will resolve symbolic links anywhere in the path and print the ‘actual’ absolute path to the item. This is equivalent to the realpath command available on Linux and some programming languages.

I have written about this before, and then I mentioned that there is a python function to resolve the path. However, even back then I anticipated the removal of python and suggested using a zsh parameter expansion modifier instead:

resolved_path=${path_var:A}

The removal of Python 2 is the likely explanation for why Apple chose to update readlink in 12.3.

It will be nice to have the new readlink -f option available going forward, but if your script still needs to support versions of macOS older than 12.3 then you should prefer to use the zsh expansion modifier.

More Monterey Command Line Changes?

This was discovered mostly by chance. While Apple’s release notes are improving, there are still nowhere near detailed enough and missing this level of detail, even though that would be amazingly useful.

I remembered that the Kaleidoscope app team had posted a script that allows me to compare man pages between versions of macOS. They published this back when macOS 12 was released to track the changes of the plutil command line tool. With the help of this tool I determined a few more interested changes in macOS 12, the most interesting of which I will summarize here.

(Even with this script, the process was tedious. Many changes to the man pages are just reformatting whitespace and/or typos. I may have missed something. Please, let me know when you find more changes!)

cut

  • new -w option (splits fields on whitespace)

du

  • new -A option (apparent size)
  • new --si option (human-readable, in 1000 based units)
  • new -t option (only show items over a certain threshold)

aa (Apple Archive)

  • new options for encryption
  • new aea command for encrypted Apple Archives

tar

  • new encryption and compression types

find

  • new -quit primary
  • new -sparse primary (so you can find APFS sparse files)

grep

  • new rgrep, bzgrep, bzegrep, and bzfgrep
  • new --label option
  • new -M, --lzma option

hdituil

  • segment subcommand and Segmented images are deprecated
  • UDBZ dmg format (bzip2 compression) is deprecated
  • udifrez and udifderez are deprecated (this allows to embed a license in a dmg)

head

  • new -n and -c options (probably just the man page updated)

killall

  • new -I option (confirm)
  • new -v option (verbose)

ls

open

  • new -u option to open file paths as URLs

pkgbuild

  • new --large-payload option
  • new --compression option
  • new --min-os-version option

I have an article on the new pkgbuild options.

plutil

  • new -type option for extract
  • new -raw option for extract
  • new type subcommand to query type
  • new create subcommand to create a new empty plist

pwd

  • new -p option prints working directory with symbolic links resolved

readlink (12.3)

  • new -f option to resolve symbolic links

rm (12.3)

  • new -I option which prompts only when more than three files will be deleted or a directory is being removed recursively

shortcuts

  • new command to run, list, or interact with Shortcuts

smbutil

  • new multichannel and snapshot verbs

Also, the nano command now actually opens pico. (Thanks, @rgov) Most people won’t notice this, as the two are quite similar. The excision of GNU tools from macOS continues.

Scripting macOS, part 7: Download and Install Firefox

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Follow this blog or the Twitter account for updates on the book’s progress!

Download and Install Firefox

To further illustrate the progress from the idea of a workflow to a working script, let us look at another, more involved example.

To download and install the latest version of Firefox a user has to go to the Firefox website and download the latest version, which will come as a disk image (dmg) file. Then the user needs locate the dmg in the ~/Downloads folder and open it to mount the virtual disk image. Finally, they need to copy the Firefox application from the virtual disk to the Applications folder.

When we want to automate the task ‘Download and Install Firefox,’ we have the following steps:

  • download latest Firefox disk image
  • mount downloaded disk image
  • copy Firefox application to /Applications
  • unmount disk image

From this list of steps, we can build the first ‘frame’ of our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

This breaks the workflow into smaller pieces, that we will now tackle individually.

Download from the Command Line

You can use the curl command to download data in the command line. The curl command is very complex and has many options. We will only discuss the few options that we require for our task here. As always, you can find a detailed description of the curl command and its options in the curl man page.

The URI to download the latest Firefox is
https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US

However, when you try to curl this URI, you only get the following:

> curl "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US"
<a href="https://download-installer.cdn.mozilla.net/pub/firefox/releases/86.0.1/mac/en-US/Firefox%2086.0.1.dmg">Found</a>.

This is a re-direction, that is commonly used to have a single URI, that is redirected to different final URIs, so that when the software updates, the same URI always returns the latest version.

We can tell curl to follow these redirections with the --location option.

By default, the curl command will output the download to standard out. To save the download to a file, we can use the --output option with a file name.

> curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" --output Firefox.dmg

This command will download the latest Firefox disk image to a file named Firefox.dmg in your current working directory. We can use this as our first step:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

Note: Like many other command line tools, curl has short and long options. The short options for –location and –output are -L and -o.
Short options are convenient in the interactive shell, as they save typing and reduce the potential for typos. But they are much less readable, and you usually have to look up their function in the documentation. For that reason, I recommend using the long, descriptive options in scripts.

Working with Disk Images

The command line tool to work with disk image (dmg) files on macOS is hdiutil. This is also a very powerful command with many verbs and options. You can find all the detail in the hdiutil man page.

To mount a disk image, use the attach verb:

> hdituil attach Firefox.dmg

This will output some information and mount the virtual disk. The last line ends with the path to the mounted virtual disk /Volumes/Firefox.

By default, you can see the mounted volume in Finder. We do not really need the disk image to appear in Finder while the script is running. We can suppress this behavior with the -nobrowse option.

Since we are only going to read from the disk image, we can tell hdiutil to mount the dmg in readonly mode with the -readonly option. This speeds things up a bit.

> hdiutil attach Firefox.dmg -nobrowse -readonly

You can unmount or eject the virtual disk with

> hdiutil detach -force /Volumes/Firefox

The -force option will unmount the disk image, even when another process is still using it.

Thehdiutil command covers two of our steps, so we can fill them in:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications

# unmount disk image
hdiutil detach /Volumes/Firefox -force

Copying the Application

When you manually install Firefox the disk image shows you a nice graphic that reminds you to drag the app to the Applications folder. Once the disk image is mounted, the cp command can be used to do this in the shell:

> cp -R /Volumes/Firefox/Firefox.app /Applications/

This provides the last missing step in our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications
echo "copying Firefox to /Applications"
cp -R /Volumes/Firefox/Firefox.app /Applications/

# unmount disk image
hdiutil detach /Volumes/Firefox/ -force

You can now test the script. If Firefox is running, you want to quit it before you run the script. You may also want to delete the existing copy of Firefox from the Applications folder, to be sure that your script is doing the work.

Lists of Commands—Conclusion

We have been able to automate a fairly complex workflow with a script of four commands.

To be perfectly honest, this script (as well as all the others we have built so far) is not complete yet.

A ‘proper’ script needs to be able to react to errors that occur. In our example, imagine the download fails. The script should be able to detect the failure before it overwrites the installed, functional Firefox application.

We will get to this kind of error handling later.

Nevertheless, this script is already useful in its current form. You can try to adapt this script to work with some other software you can download as a disk image.

You can also add extra commands that

  • delete the downloaded disk image at the end
  • open the newly installed Firefox app after installation
  • quit or kill the Firefox process before copying the new version

In the book “Scripting macOS”, you will learn more scripting techniques, and we will re-visit some of these sample scripts and keep improving them.

Follow this blog or the Twitter account for updates on the book’s progress!

Note: After using different variations of these kinds of workflows, I did put together a more generic script to download and install various kinds of software, called ‘Installomator.’ You can see the script at its open source repository on GitHub.