Imaging is still Dead

At WWDC last week, there was a very interesting session on “Apple File Systems” (APFS). It covered the new split system layout in macOS Catalina with a read-only system volume, volume replication with APFS, and how external USB drives and SMB works on iPadOS.

The entire session is very interesting and well worth watching. Go ahead, I’ll wait…

Around the 13 minute mark, during the ‘Volume Replication’ segment, the engineer on stage talks about using asr (Apple Software Restore) tool to ‘replicate’ a system volume to several computers at once and gives the example of a computer lab. He then proceeds to explain the new options in asr regarding APFS volumes and snapshots.

Slide from WWDC 2019, Session 710

The new features are hugely interesting and I think they will be very useful for backup solutions. There will probably be some applications for MacAdmins, but I disagree with the engineer on stage and some MacAdmins on Twitter and Slack:

Catalina will not bring a revival of imaging.

Note: I wrote a book on this: “macOS Installation for Apple Administrators

What killed imaging?

Back in the Sierra days, there was this idea that the introduction of APFS would ‘kill’ imaging. The asr tool relied on many HFS+ behaviors and it was questionable that Apple could or would maintain that for APFS. But while there were some changes to asr in the High Sierra and Mojave upgrades, it still worked.

What killed imaging as a process for MacAdmins was the T2 system controller, first introduced with the iMac Pro. There are two main aspects:

  • NetBoot and external boot are defunct
  • Firmware needs to be updated with the system

Netboot and external boot are defunct

To re-image or re-install a system, you have to boot it off a different system volume (NetBoot, Recovery, external drive). Alternatively, you can put the system into target disk mode and image or install the system directly on the internal drive.

On Macs with the T2 system controller, NetBoot is explicitly defunct. External boot is disabled by default. It can be re-enabled, but the process is convuluted, requires at least one full setup process, and cannot be automated.

This leaves Recovery as the system to use to replace the system volume and, not surprisingly, there are a few tools that have focussed on using Recovery in the new T2 Mac world:

Firmware needs to be updated with the system

You could also put the target Mac in target disk mode and image its system. This will work, as long as the system on the image is the same version as the system that was installed before. We have been warned about this in the infamous HT208020 support article:

Apple doesn’t recommend or support monolithic system imaging as an installation method. The system image might not include model-specific information such as firmware updates.

Modern Macs don’t just require a few files on disk to make a bootable system. Inside your Mac are several subsystems that require their own systems (i.e. firmware) to run. Most prominent are the T1 or T2 system controllers which are actually independent custom ARM-based processers running a system called ‘iBridge’ which is an iOS derivate.

If you just exchange the ‘normal’ system files on the hard drive over TDM, without also updating the various firmwares in the system, you may get your Mac into state where it cannot boot.

This was most obvious with the macOS High Sierra upgrade. After re-imaging a 10.12 Sierra Mac to High Sierra running on APFS, would lead to a Mac that could not read the new system volume. The firmware update that came with High Sierra is needed, so the firmware can mount, read and start the APFS system volume.

How can I update or upgrade?

For security, only Apple’s ‘Install macOS’ application and the intermediate software and security update packages have the necessary entitlements to change the built-in firmware(s).

Firmware updates can be in system updates (minor version updates, i.e. 10.14.4 to 10.14.5), security updates, and major system upgrades (i.e. 10.13 to 10.15).

There are three options to apply a system update (e.g. from 10.14.4 to 10.14.5) or security update:

  • ‘Install macOS *’ application, either manually or with the startosinstall tool
  • Software Update, either manually or through the command line tool
  • system or security update pkg installer downloaded from support.apple.com

When you want to upgrade a Mac through a major version change (e.g. 10.13 to 10.14 or 10.15), there is only one option:

  • ‘Install macOS *’ application, either manually or with the startosinstall tool

The one remaining use case for imaging

Given the above limitations, there is one use case left for imaging. When you have full control over the macOS version installed on the Mac and its firmware and the image matches that version, then you can image.

However, since NetBoot and external boot are defunct, you will have to image either over target disk mode (fast, but only a few Macs at a time) or using the Recovery (hard to automate, comparatively slow).

The remaining strength of the imaging workflow is the raw speed. Some application suites measure several Gigabytes, if not tens of Gigabytes. With installation workflows, these have to be downloaded, decompressed (pkg installers are compressed archives) and copied to the system drive, a process that takes a lot of time. With imaging, these can be layed down with fast block copies.

For example, the re-installation of a MacBook Pro I tested recently took about 25 minutes. This time includes downloading the 6GB ‘Install macOS’ application and the entire re-installation process. (I could probably have sped this up with a caching server or by pre-installing the full ‘Install macOS’ applications.) If I could have used imaging this would take 2–3 minutes.

If you are in a situation where you have to restore Macs to a pre-defined state frequently and quickly, then imaging might still be a useful workflow. One use case may be MacBooks that get frequently handed out as loan units, where the users get administrative privileges, so they can install extra software and configure the loan units.

You will have to invest extra effort during updates or upgrades to apply them first on the devices, to ensure the firmware gets updated, and then to update the image, as well. In some use cases this extra effort can be worthwhile.

MDM (and DEP) is required

With modern macOS there are other considerations for deployment that make classic imaging workflows less practical. Before macOS 10.13 High Sierra, MacAdmins could manage their Mac fleet without an MDM server. In High Sierra 10.13.4 Apple added two things to the MDM protocol:

  • ‘user-approved’ MDM
  • Kernel extension white listing via configuration profile

The second feature (white listing Kernel extensions) requires the first (user-approved MDM). You cannot manage Kernel Extensions or Privacy Preferences Control settings in Mojave, with out a user-approved MDM. In mosts organizations, these are not limitations you can work around. An MDM is now a requirement to manage Macs in an organization.

From what we can glean from the WWDC sessions, the (UA)MDM controls will be increased even further with Catalina. It will be driven even further: DEP or ‘Automated Device Enrollment’ with Apple Business Manager or Apple School Manager will be required for some new management features, such as ‘bootstrap tokens’ for FileVault.

Each Mac client needs to be enrolled in the MDM individually. The MDM enrollment cannot be part of an image. The easiest way to get a Mac enrolled is with Automated Device Enrollment (formerly known as DEP), which happens at first boot after installation.

Third party software

It is not just the macOS system that needs to individually enroll with the MDM server. Many third party solutions now also require subscriptions or licenses to be activated on each device individually. All these additional configurations that need to happen after installation or imaging, decrease the usefulness of including all software and configuration in an image.

Patching and software updates

Most imaging deployments, used a workflow where the image was kept ‘static’ or ‘frozen’ for longer periods of time, usually six or twelve months. This will minimize the effort to update the image, system and software.

However, modern operating systems and third party software have update frequencies of 4–10 weeks. Modern security requirements will require these updates to be applied in a timely matter. Critical security problems can strike at any time, requiring fast updates from the vendors and the Mac Admins.

As with the MDM above, having a system in place that allows the MacAdmin to easily and quickly deploy and, when necessary, enforce an update or patch to the entire fleet of devices is an important requirement.

Software and patch management of non-App Store applications is not part of the MDM protocol. Nevertheless, many MDM solutions also include additional functionality for software management, with varying degree of usefulness.

Some MacAdmins prefer to combine their MDM solution with the open source solution Munki instead. Munki is considered to be the best software management solution for macOS, but does not include MDM functionality itself.

Whichever software management solution you use, once you have that in place, it will be easier to manage (i.e. install and enforce) software through the management system, than to keep an image up-to-date and re-applying it.

You will end up with a ‘thin’ base image and everything else deployed and managed by the management system. At that point you might as well switch to an installation based workflow.

But, the engineer on stage said…

Here are all the limitations on imaging, summarized:

  • NetBoot and external boot are defunct
  • system firmware needs to be updated with the system
  • MDM and DEP are required
  • frequent security updates and patches require continuous software management

None of these limitations are addressed by the changes to the asr tool in Catalina. Changes in other areas of the system in Catalina will actually re-inforce some of these limitations.

Imaging is still dead.

But why even have asr, then?

The asr tool exists because Apple needs a tool to image the operating system to new Macs in the factory. Obviously, Apple has absolute control over the versions of macOS and firmwares deployed to the systems, so they ensure they all match. Speed is a priority, so Apple needs and maintains asr.

Other uses of asr, including the use as an imaging tool for administrators have always been secondary.

As mentioned earlier, when your environment has similar requirements (fast re-deployment) and can provide tight control over the macOS and firmware versions, then imaging might still be a useful workflow for you.

You can already do this with High Sierra or Mojave. You do not have to wait for the new Catalina features for this.

In general, a simpler (albeit slower) installation-based workflow is less complex to deploy and maintain. (Imaging might seem less complex, because it is more familiar.)

So, the new features in the presentation are pointless?

The other use case for asr in the presentation, backups, are very exciting. They will allow the system to take a snapshot and then copy the data of the snapshot to a backup while the system keeps running and changing files. You may also be able to restore a system from a snapshot stored elsewhere.

The split of system volume and user data volume in Catalina is also very intriguing for Mac Admins. This may of course, break some third party software. (Start testing now.) But it may also open up new options for management. One of these (user enrollment) is introduced in the “Managing Apple Devices” WWDC video.

One possible workflow could be to snapshot and/or image the data volume and leave the system volume intact (you have to, it is read-only and SIP protected). It is still questionable how well this might work, since the firmlink connections between the system and the data volume might not survive the replacement of their targets. You can start testing this now, but keep in mind that the details of the new file system layout will still change during the beta phase.

Summary

  • The changes introduced to the file system in macOS Catalina at WWDC are major and will enable new workflows for MacAdmins.
  • Start testing Catalina now.
  • The limitations that ‘killed’ imaging, still apply or might be re-inforced. Imaging is still dead

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.