NetInstall is Dead, too

I have written a book which expands on this topic and is regularly updated. Please check it out: “macOS Installation for Apple Administrators“

Tim Perfitt of Twocanoes Software (Winclone, SD Clone, etc.) got an iMac Pro.

For obvious reasons he immediately looked at the details of the new boot process, and has found some details that were speculated or unknown so far. Most of this happened on Twitter, which is a quite hard to put together afterwards, so here is a summary: (there were several members of the Mac Admin community involved, thanks to all of them!)

Update: Tim Perfitt now as an excellent detailed post on his findings here.

macOS build for the iMac Pro is 17C2120the iMac Pro will not NetBoot, even with Secure Boot disabled
a local administrator’s password is required to disable Secure Boot or enable External Boot
you cannot disable Secure Boot or enable External Boot before the first installation as there is no local administrator
you can still erase the system volume without a local administrator’s password
re-enabling Full Security in the Secure Startup Utility requires an internet connection
reseting the NVRAM will reset SIP, but not the Secure Boot setting
FileVault/Full Disk Encryption is not enabled by default
Secure Boot requires an internet connection when it attempts to fix the boot files. This will not work with proxies. Also requires access to 17.* addresses.
iMac Pros will still boot when offline
for Bootcamp, Secure Boot will verify the Windows bootloader
the iMac Pro does not only kill imaging and NetBoot, but also the remaining EFI ROM tones (also noted in this support article, startup chimes already went away with 2017 MacBook and MacBook Pro models, thanks to Arek for pointing that out)
iMac Pro Essentials (iBook Manual)
iMac Pro Help Pages

There are probably a few more details which will come out as other admins get their iMac Pros in the following days and weeks. But this gives us enough confirmation of facts to know:

NetBoot is dead!

Don’t Panic!

So the news of NetBoot’s demise has not been exaggerated. Also it is to be expected that all new hardware from Apple going forward will have Secure Boot and probably not NetBoot.

MacAdmins will all need to plan ahead and look at the options that are on the table for Mac management going forward.

There is speculation that the current TouchBar MacBook Pros might get Secure Boot added in a future update to 10.13. Even if that is not the case, then it is a safe assumption that future Mac releases will contain the T2 system controller or something similar and have the same Secure Boot features (or lack thereof).

Deployment Strategies Going Forward

Device Enrollment Program and Mobile Device Management (DEP + MDM) is certainly Apple’s deployment method of choice. They have been pushing to this for a few years now and it has also been the way to manage iOS devices.

DEP is a process where a new device (iOS or Mac) is registered to your organization at purchase and you can assign it to your Mobile Device Management server through Apple’s website.

At it’s very first boot, the Mac will check with Apple’s DEP servers and get the MDM’s information, register with the MDM and then the management settings take over, adding configuration, software and, with some management systems, local tools to install and manage non-AppStore software.

When a Mac’s system volume is erased and macOS is re-installed the process starts over, keeping the Mac managed by the same MDM.

Apple has also made DEP+MDM a requirement to manage Kernel Extensions without user interaction. Furthermore, Apple states that going forward, the “approved” level of MDM (either by DEP or explicit user interaction) will be used for more configurations in the future.

This is similar to “supervised” iOS devices. However, Apple provides two means to supervise an iOS device: with DEP and by manually connecting an iOS device to a Mac with Apple Configurator. The process with Configurator can be automated, aside from the manual connection. On macOS the manual (“user-approved” MDM enrollment) cannot be automated and cannot even be performed over remote control.

In general, DEP+MDM works well. It enables certain management styles and workflows that were not possible before. An organization can order a device from Apple or a reseller and have it sent directly to an employee. When the employee unboxes the new device it is registered with the organization’s MDM and receives configuration profiles and software, even when the device is off-site.

Apple and MDM vendors like to call this workflow “zero-touch” deployment, because the IT department does not have to touch the device. This is a great improvement for many Mac administrators.

However, there are a few downsides to DEP+MDM:

External Dependency

Apple’s DEP servers are an external dependency and a single-point-of-failure in the deployment workflow. There were a few outages of the DEP system this year. Even worse, Apple does not include the DEP service in their status overview page. This leaves Mac admins wondering if a problem is on their side or with Apple.

DEP availability

DEP is not available in all regions where Macs are sold. With imaging and NetInstall off the table, this is leaves only manual installation and MDM enrollment/approval for management.

Also, a client has to be online and the network to have access to Apple’s servers. This requires un-proxied access to Apple’s 17.* IP range. However, especially when you are outside of the US, the processess at installation may attempt to connect to other IP addresses as well.

With Apple Configurator an administrator can also add existing iOS 11 devices into DEP, not just new ones. This option is not available for existing Macs.

User Interaction

DEP + MDM allows to automatically enroll devices without requiring IT to touch a device. However, the process is not automated. There has to be a user present to interact at several points with the Mac for the initial setup. While profiles can manage and reduce the user input required during setup, there are a few steps you cannot automate away.

Also the enrollment process will only install the management tools then show the user the desktop. Actual installation of software packages takes place in the background and might take a long time. This can leave the user confused as to what is going on. Certain management options, such as enabling FileVault may require a logout or restart, interrupting the user with whatever they started to do on their new Mac or leaving the Mac in an unsecure state until the user restarts.

This downside of DEP is so glaring that many open source solutions have sprung up to provide a user interface for the post-DEP initial configuration cycle.

(I am probably missing some, let me know!)

This innovation and initiative of individual admins and the community as a whole is admirable. Thanks to all who provide!

Either way, administrators using DEP + MDM have to be aware of the time required for the download and installation of large software packages and choose which pieces are absolutely required and which can be deferred to be installed later at a time of the user’s choice through a self service portal.

Software and Configuration Management

DEP handles the initial connection to the MDM. The MDM can push and enforce profiles to control some settings. The MDM can also initiate installation of (Mac) App Store software through the Volume Purchase Programm (VPP).

To manage software and configurations that are not in the Mac App Store or not supported by configuration profiles, administrators need to install a local tool on the client system.

The MDM protocol provides a tool called InstallApplication that will instruct a client Mac to download a pkg file and install it. For example, the Jamf Pro management suite uses this to install the jamf binary tool, which then can take over and perform many other management tasks, which the MDM system does not provide.

Some management systems (so far I know of SimpleMDM and AirWatch, let me know if I missed any) allow admins to provide their custom installer to install a local management tool (e.g. Munki, Puppet)

Notably, Apple’s reference MDM implementation, Profile Manager (part of the macOS Server app) does not allow for custom installs.

Erik Gomez has done outstanding work documenting his experiences with this process. The entire series is worth reading. but if you want to catch up quickly, the recent posts have a good summary of the status quo and a real-world implementation.

Offboarding, Re-installation and Re-purposing

Once the initial configuration is complete, MDM+VPP+management system will take care of installations and software updates. However, there are situations, where you will want to ‘nuke and pave’ or ‘wipe and re-install.’

There are many reasons an admin may want to do this, most of them involve ‘configuration drift.’ I.e. over time as more and more software gets installed and configured on a given system, errors and conflicts pile up and cause problems. At this point it is usually easier to ‘nuke and pave’ or ‘erase and install’ than to track down the actual conflicts.

In an ideal world, all the configuration owned by a user would be exclusively in that user’s home directory and you would only have to delete and recreate that user, rather than the entire system. However, we do not live in that world.

Many pieces of software store configuration in central locations, but still assume that these central locations are writable by the user. In most setups this is the case, because by default users are admins on macOS. However, this spreads configuration and other data all through the system, making it impossible to isolate all changes.

With imaging and NetInstall, admins could use the same workflow for the initial installation and configuration than for subsequent-installations.

Furthermore, the process could be automated to the point were no local interaction was necessary, or just the minimal interaction of someone restarting a Mac and holding the ‘N’ key. From then on all steps could run fully automatically and without interaction. Further more, imaging with block copy was fast.

With High Sierra, Imaging is not supported anymore, except in very specific circumstances. Automated Installations with NetInstall are broken. And with the iMac Pro the option for NetInstall goes away completely. (Which may explain why automated NetInstall was not fixed in High Sierra.)

This leaves manually booting to (Internet) Recovery, erasing the startup volume in Disk Utility and re-installing macOS as the only means of re-installing a Mac. After the installation DEP should re-connect the Mac with the MDM and management should take over.

However, you cannot completely automate the DEP interaction, so after waiting for ~30minutes for the installation to complete, someone has to confirm a few dialogs before managed installation can kick in and do the rest of the work. All this interaction is time-consuming and error-prone.

“Erase all Contents”

On iOS, you rarely have to re-install the system. Instead there is a function ‘Erase all Contents and Settings’ which restores a device to a clean unconfigured state, from which DEP and then the MDM can take over. You can even send the wipe command over the air with an MDM. (On iOS you also have to manually confirm a few dialogs before DEP and MDM can take over, but the entire process is much faster.)

Until Apple provides this feature on macOS locally and remotely, admins who rely on fast restores either have to stay on Sierra, postpone new hardware purchases, or revisit and redesign their workflows.

This mostly affects education customers with lab deployments. Other large Mac deployments were a Mac is “owned” by a single user, are less affected by this.

Sidenote on Mac App Store and VPP

Mac App Store applications have to be sandboxed, which means they can’t even access all of the user home directory, only their designated sandbox. Managing these applications and the data is much more manageable than other Mac applications and tools where anything goes.

On iOS, the App Store and VPP are the only means of distributing applications, and this is much simpler and manageable. However, the App Store rules prohibit entire classes of tools and services. While the Mac App Store enforces similar rules, users can still download and install applications and tools outside of the Mac App Store. For most Mac users, this is the defining advantage of macOS.

However, this higher complexity of software and deployment methods, requires more complex deployment and configuration workflows.

MDM + VPP cannot handle this complexity, which is why management systems, such as Jamf Pro, Filewave, and Munki, exist and need to exist for macOS management.

Also I need to say that not all software and installers need to be as complex as they are. Most software that comes with complex installation tools are unnecessarily complex and error prone. Often the developers are just taking a cheap shortcut, by assuming the current user has write access to the application bundle, or has admin privileges, etc.

However, there are still entire classes of professional software, that even if they did simplify their applications and installers, would not currently be allowed in the Mac App Store (IDEs and developer tools, hardware drivers, certain virtualization software features, anything that needs root access, etc.)

Also the App Stores (on macOS and iOS) have features, that cannot be purchased or distributed by VPP. In-App-Purchases and subscriptions cannot be purchased or distributed. Recently, Pre-orders for a special price were added as a feature to the App Stores, but these can also not be used for VPP.

Overall, I would love if all software were available in the (Mac) App Store and could be managed and distributed with VPP, but we are a long way from that reality.

Why all this?

By now it seems fairly obvious that Apple wants to get macOS system security to a point where only Apple can ever affect and change system software and firmware. That is a worthwhile goal. It means that your data is secure on an encrypted drive, they decryption key is locked in the secure enclave, but Apple can design solutions like TouchID and FaceID to unlock everything quickly.

To close the loop for all this security, the system needs to be able to verify and confirm that the software running the system (both the OS and firmware) are up to date and in their original state.

Imaging, NetBoot and NetInstall bypass most of this security. I believe it could be possible to create a networked installation workflow with all the security in mind, but it might just not be worth the effort. Apple seems to think this is not worth doing right now.

And remember that imaging and NetInstall are not valuable in themselves, but they are valuable as tools to achieve something useful, namely: automated installation and configuration of Macs.

DEP + MDM + VPP gets us there in many situations. In many use cases, DEP allows for workflows that were not possible before. Other technologies in macOS High Sierra (snapshots) promise some more useful tools, but they are not quite there yet.

Right now the gap between what we currently use as admins and what will come down the road is getting really wide.

Where to go from here?

There are two things a system administrator needs to balance:

provide a stable and efficient environment to manage the computers, software, configurations, and users
adapt the environment and workflows for new and future requirements and technologies

These to goals are often at odds and balancing them is a circus act in the best of times. Right now Apple is making our collective lives harder by shaking the rope we are standing on and throwing a few new balls in to the juggling act at the same time.

Apple and the MDM vendors are providing a powerful new solution DEP + MDM which works well for some deployement styles (1-to–1 deployments).

History has shown, that going against Apple’s vision of how their devices should be used, will not result in a smooth experience. Take a good look at your deployments using imaging and NetInstall: might a different deployment scenario work?

In education, labs are often used because certain software is too complex or expensive to be provided to all laptops. In this case virtualization, switching to another software solution or different OS might be a solution. (And please let your Apple rep know you are considering switching to another OS. That is the great leverage we have on Apple to support better workflows.)

To buy some time, you could hold on to Sierra for a while longer. Right now, all Macs except the iMac Pro still support Sierra. As new hardware gets released next year, your options will dwindle. Maybe your organization can accelerate or postpone purchases with that in mind. This cannot last forever, but buy you some time.

However, any new Mac releases will, like the iMac Pro, require High Sierra and (most probably) have the same Secure Boot features. How will you support those when they are purchased? The high price of the iMac Pro might discourage purchases, but for how much longer. You will need to have an answer in place.

(Also, this is great argument that you need an iMac Pro for testing, now… 🙂 )

Your answer may very well be, that you will have to accept the extra manual affort required to (re-)install High Sierra based Macs. However, then you had better have an idea of how much more effort and time will be required, to justify the extra workload to your organization.

Test, test, test!

Do you have a DEP + MDM solution in place? Did you get the budget for it? Are you testing deployment workflows with it? For the past year and more, the writing has been on the wall that this is the way to go. If you haven’t started on this by now, you really, really have to.

Do you have an idea/solution on how to smoothen the new deployment workflow for you or your users? Whether it is just an idea or a finished workflow, please discuss and share it in the MacAdmin community. The MacAdmins Slack is a great place to start. Maybe someone in the community will figure out how to use APFS snapshots to quickly and reliably restore a Mac to a well-known state before Apple does.

There are already some interesting ideas out there.

Maybe you’ve already done all this and found a setup that works for you. Well done! This would be a great time to present your solution and how you got there at a Mac Admin meeting or conference. Many other admins would love to learn from you. (Or just write a blog post.)

Talk with your Apple Reps, file bugs, etc. Don’t expect Apple to bring back imaging or NetInstall, but do point out the shortcomings of Apple’s solutions going forward.

The orchestration for Apple to get the new hardware and software components and pieces in place must be enormous. Some pieces take longer and with patience we will see how everything fits together.

We are living in interesting times!

Happy New Year 2018!