Weekly News Summary for Admins — 2017-10-27

Back from vacation, back into work. JamfNation User Conference was this week. Looks like everyone had a great time!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

🍏Apple Support

♻️Updates and Releases

📺To Watch

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Imaging is Dead… Long Live the Installer!

The writing has been on the wall for a long time. With the release of macOS High Sierra, Apple has finally confirmed that imaging is dead.

Apple doesn’t recommend or support monolithic system imaging for macOS upgrades.

The Final Nail

The final nail in the coffin for imaging is this support article: Upgrade macOS on a Mac at your institution

It states the limitations to installing and upgrading macOS with High Sierra:

  • the Mac being installed or updated must be connected to the internet
  • installations and updates cannot be done on external devices, like those connected via Target Disk Mode, Thunderbolt, USB, or Firewire
  • there are four supported methods of installing macOS High Sierra

These methods are also re-iterated in this section in the macOS Deployment Reference

Why?

Apple’s stated reason for requiring the installer is to ensure that a Mac’s firmware is all up to date and matches the OS installed on it.

Only the macOS Installer can download and install the firmware update. Firmware updates can’t be done on external devices, like those connected via Target Disk Mode, Thunderbolt, USB, or Firewire.

This is especially important in High Sierra, because to boot into a system on an APFS formatted (or converted disk) the Mac’s firmware needs to be able to mount and read APFS. The firmware that was installed with 10.12 or earlier is not able to read APFS volumes and. When you image a Mac with High Sierra and APFS without updating the firmware, you will get the question mark at boot, because the firmware cannot find a system.

However, the EFI, which manages (among other things) the boot process is not the only “firmware” that needs to be managed on your Mac. Many of the hardware components in your Mac, such as the SSD, the power controller (SMC) and the TouchBar controller on the new MacBooks Pro, have their own firmware that needs to be installed and updated.

Apple has been increasing the protection of the vital parts of the system and hardware. The firmware in these components cannot just be changed by any process. Only the Apple macOS Installer application has sufficient privileges and entitlements to perform these updates. The installer process has to run on the Mac itself, it cannot run over target disk mode.

Future Mac hardware might introduce even more components that require firmware.

On iOS, a secure boot chain prevents tampering with the system on a device after it has been installed. The secure boot chain also prevents replacing the system with an image from another device.

It is conceivable that Apple wants to implement a secure boot system on future Macs as well. Current Macs probably do not have the hardware required to implement this. (The TouchBar MacBooks Pro have a Secure Enclave chip like the iPhone and iPad and might already have the necessary pieces in place.)

So now what?

This has been coming for a long time. Even though APFS is not, as originally predicted, the direct culprit. It is still end-of-the-line for imaging.

However, the news is not entirely dire. Apple has been surprisingly forthright about the direction they want to go. While the documentation is a bit lacking, there are instructions on what the solutions for Mac System Adminstrators should be.

The four supported means of installing and upgrading macOS and the firmware for Macs are a clear direction of what needs to be done. However, Apple is a bit shy on how administrators can and should implement them.

There are a few options:

Put the Burden on the Users

This will work in some deployments where users are in control of their Macs. Either a full BYOD (Bring Your Own Device) scenario, or one where devices provided by the organisation are in full control of the user.

Even when the devices are enrolled in an MDM, users are still administrators and in control. Administrators can use reporting tools, to determine which Macs are capable of installing High Sierra and not upgraded yet.

You can even use reporting tools to gather information on the firmware and whether it matches the latest version.

You can then instruct users with email or notifications to download the High Sierra installer and initiate the upgrade themselves. You should warn them to have a current backup and that the process might take some time, so it should be run overnight.

If you have a software management system in place, you can use that to load the macOS Installer application on the clients and notify the user when it is ready.

The Mac App Store only downloads a “stub” installer application which is then filled in with an extra download. If you have blocked access to Apple Software Update Servers or redirected clients to a local, managed Software Update Server, clients might not get the complete installer application. Greg Neagle has a great post on this.

To save download time for the users, you can also provide USB/Thunderbolt drives with a bootable installer drive. This might speed things up a bit, though it does not really change the process.

How do we Automate this?

As system administrators we want to automate the process, so that it ideally does not require any human interaction. That way we can replicate the process hundreds and thousands of times.

Ideally, we also want to inject some custom steps into the process. Apple provides two means of achieving both of these steps, and some open source tools are providing solutions as well.

NetInstall

A custom NetInstall set built with Apple’s System Image Utility is one of the supported means of installing and updating macOS (and the firmware) to High Sierra.

You can find System Image Utility in /System/Library/CoreService/Applications. You can customize the process and even add your own installer packages, scripts or profiles. Packages used in System Image Utility have to be Distribution Packages.

Since a NetInstall system is like a Recovery system, you can use scripts to control behavior of your Mac like allowed NetBoot IPs, or ‘User-Approved Kernel Extension Loading’ (UAKEL) here as well.

You can even automate the NetInstall process to a point where, once you have chosen the NetInstall volume (when holding the the option key at boot) the remaining process is without interaction. (Though this is a dangerous choice, as it might simply wipe and re-install Macs. Use the other limitation options such as by MAC address or hardware type to keep this safe.)

NetInstall requires Mac running macOS Server. However, BSDPy can replace a NetBoot/NetInstall server and runs on many platforms, including VMs.

Note: as of 10.13.0 there still seem to be a few bugs with NetInstall on HighSierra. It works mostly but seems exceedingly slow. Also some admins have reported problems with adding mulitple packages, profiles or scripts for configuration. 10.13.1 is already in beta and seeding and you should be testing that.

NetInstall on USB

If you do not have the infrastructure to run a NetInstall or BSDPy server, you can also restore the NetInstall.dmg image that System Image Utility creates to an external drive. When mounted on a Mac, it will the High Sierra installer applications, and double-clicking it will start the proper installation process.

Any additional packages, profiles or scripts will be included with this custom external install application as well.

The startosinstall Command

When you already have a management system (Munki, Jamf, Filewave, etc.) you want to initiate the update process with rules or policies. At first glance the supported means of installing macOS seem to be at odds with managed client workflows, since they require user interaction.

However, there is a tool hidden inside the macOS Installer application (since macOS 10.12 Sierra) called startosinstall. The full path to the tool is

/Applications/Install macOS High Sierra.app/Contents/Resources/startosinstall

Note: I believe it was Rich Trouton who first documented this tool in his notes for WWDC 2016. Since then many admins and open source projects have worked to figure out how to use this tool in the best way.

When you run it with the --usage argument you get the following:

$ /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/startosinstall --usage
Usage: startosinstall

Arguments
--applicationpath, a path to copy of the OS installer application to start the install with.
--license, prints the user license agreement only.
--agreetolicense, agree to license the license you printed with --license.
--rebootdelay, how long to delay the reboot at the end of preparing. This delay is in seconds and has a maximum of 300 (5 minutes).
--pidtosignal, Specify a PID to which to send SIGUSR1 upon completion of the prepare phase. To bypass "rebootdelay" send SIGUSR1 back to startosinstall.
--converttoapfs, specify either YES or NO on if you wish to convert to APFS.
--installpackage, the path of a package to install after the OS installation is complete; this option can be specified multiple times.
--usage, prints this message.

Example: startosinstall --converttoapfs YES 

There is also an undocumented --nointeraction flag which can be used to run the tool without any user interaction. This is obviously useful for management systems.

Once you have used to your management system to make sure the macOS Installer application is on the client system, you can execute a script with the startosinstall command to initiate the installation process. Remember that the installation process can take a long time, so it should be initiated by the user in a Self Management portal or run during off-hours for kiosk like Macs in labs or classrooms.

startosinstall --applicationpath /Applications/Install\ macOS\ High\ Sierra.app \
    --agreetolicense \
    --nointeraction

The --converttoapfs [YES|NO] argument allows you to suppress automatic APFS conversion on SSD Macs.

There is also a --volume argument to target the non-boot volume. However, this will only work when SIP is disabled or when you run startosinstall from a Recovery/NetInstall disk.

The --installpackage option allows you to add one or more custom packages that will be installed after the OS installation is complete. This is very useful for customization and cleanup. Packages used with startosinstall --installpackage also have to be Distribution Packages.

Note: even though the usage states that you can repeat the --installpackage argument, as of 10.13.0 only the first package given will run the installation with fail with more than one package. Make that one package count. (Note: edited this paragraph. Thanks to Greg for clarifying.)

Tool Support

Is Imaging completely dead?

The imaging tools (like Disk Utility, hdiutil and asr) will work with APFS volumes. However, the support article states:

You can use system images to re-install the existing operating system on a Mac.

So when you need a workflow that requires quick re-imaging, you can use one of the supported methods to install or update the Mac (Firmware and OS) and then use monolithic (or thin) imaging over network or thunderbolt for fast restores. This is useful for scenarios where fast imaging turnaround is required, such as classrooms and labs or loaner laptop setups. However, you have to use extra care to make sure the image system version matches the version that was installed.

Going forward, I expect imaging to be less and less feasible as future Mac hardware and security features will make it harder and harder to use. The times where we could have just one image which will run on all supported Macs might be over as hardware (and the software required to run the hardware) becomes more and more fractured. Note that there is not a unified single ‘iOS’ image/installer for all iOS devices.

Summary

macOS High Sierra 10.13.0 works well for individual users. However, there still are quite a few issues that are relevant for managed deployments. There are many problems with Active Directory and Filevault, NetInstall is slow and adding multiple packages to an installation is broken. Just to mention a few.

Even though it makes sense for some deployments to hold back from High Sierra right now, you will want or have to upgrade soon.

Apple said at WWDC the iMac Pro will ship in December 2017. Its tech specs page, states it will run High Sierra. You can expect it to require High Sierra.

Also, critical security patches might only be pushed for High Sierra.

Imaging is dead. In an unusual move Apple has come right out and said it loud and clear. If you have not done so already, start testing and implementing one of the above strategies right now, so you are ready to move to High Sierra.

On the Shebang

Every script you want to run from the command line should have a shebang as the first line.

Note: I talked about this in my MacSysAdmin talk. I wanted to go into more detail here.

You can have scripts without the shebang. In this case the shell that you launch the script from will be used to interpret the script. This can, obviously, lead to complications, on other systems or with other users.

A shebang looks like this:

#!/bin/bash

The eponymous first two characters of the shebang are #! (hashbang or shebang). These to characters form a ‘magic number.’ Files are usually identified by certain codes (i.e. magic numbers) in the first few bytes of data. The hex code 23 21 converts to the ascii characters #! and tells the system that a file is script.

After the shebang comes the command that should interpret the script (the interpreter). This is either a shell, as in #!/bin/sh or #!/bin/bash or another interpreter, for example #!/usr/bin/python or #!/usr/bin/osascript. Generally, any command that can interpret text files or streams can be used in the shebang.

Since the status or value of the PATH variable is not guaranteed or certain in many contexts that scripts can be run in, the path after the shebang needs to be absolute.

The env shebang

There are environments where you cannot predict the absolute path of a given tool. For example the bash v3.2 shell on macOS is installed by default in /bin/bash. Users can also download and install bash version 4.4 onto their computers. The location for the the bash 4 binary is usually at /usr/local/bin/bash (this might be different depending on the installation method you used). Since /usr/local/bin is the first item of the default PATH on macOS the newer bash 4 will be chosen before the built-in bash 3.2 when the user types bash into their shell.

When you use the absoute path to /bin/bash in a shebang, you are ensuring that the macOS provided built-in version of bash will be used. For macOS system administrators, this should be the preferred shebang, since it provides a known, certain environment.

However, there are cases where you want scripts to be run with the user’s preferred tool, rather than a set path. You may also want your script to be able to on multiple different unix (and unix-like) systems where the location of the bash binary can be unpredictable. In this case you can use the /usr/bin/env tool as the shebang with bash (or another interpreter) as the parameter:

#!/usr/bin/env bash

This shebang means: “determine the preferred bash tool in the user’s environment and use that to interpret the script.”

Note: obviously this also presumes the absolute path to the /usr/bin/env. However, most unix and unix-like system seem to agree here.

Shebang Arguments

The shebang line can take a single argument. For weird syntactic reasons the entire line after the interpreter path is passed as a single argument to the interpreter tool.

So a shebang line that looks like this:

#!/bin/bash -x

will be executed like this:

/bin/bash "-x" /path/to/script

But if you added another variable to the shebang:

#!/bin/bash -x -e

then it would be executed as

/bin/bash/ "-x -e" /path/to/script

where "-x -e" is a single argument, making bash choke.

Some tools (like env) that are regularly used in shebangs are able to split the single argument. bash is not one of them. Read a tool’s man page and documentation and test. In general it is considered bad style to add arguments or options to the shebang.

Note: with bash you can also set these options in the script with the set command:

set -x
set -e

bash versus sh

Many Unix and unix-like systems have sh as well as bash and other shells available. sh goes back to the very early UNIX shells in the seventies. sh has survived because it serves as the lowest common standard for shell scripting. These standards are defined in the POSIX specification, though POSIX defines more than just shell scripting.

When you have to build scripts that need to run across many different flavors and versions of Unix and Linux, where you cannot rely on bash being present, then conforming to POSIX and sh might be necessary.

Note: bash is also POSIX compliant, but it has more features. When you script against sh you ensure you don’t use any of those additional features.

However, as macOS administrators, we can rely on a larger ‘common’ set. bash has been present on macOS since Mac OS X 10.0 (and earlier incarnations in NeXTSTEP). bash gives you several extra features over plain sh such as better testing with the double square bracket, ‘here documents’, parameter substitution and arrays.

As a macOS system administrator you should leverage these and always choose /bin/bash over /bin/sh.

bash versus Other Shells

The argument that bash is better than sh works for other shells as well. You can make a very good argument that other shells are better than bash, or at the very least bash version 3 included in macOS. I often hear this with zsh, so I will be using zsh as an example, but the arguments work with other shells as well.

(You can find a very good comparison of different shells here.)

zsh and other shells have many features that bash 3 (and even 4) lacks. There are two main reasons I would still recommend bash for scripting:

  • most script examples and shared code is written in bash so you need to be proficient in bash anyway (and know the quirks of bash)
  • the zsh is not included on the macOS Recovery System or other macOS installation environments, so you cannot write post-installation scripts for these contexts in zsh
  • when you do reach the limitations of bash for scripting, other languages such as Python and Swift are even more useful and powerful on macOS than zsh, so the step up from bash should not be another shell, but a different scripting language entirely

Interactive shell versus shell scripting

While I argue that that other shells don’t have that much edge on bash for scripting, they certainly do have an edge on bash in some of their interactive and customization features.

Two of the more interesting interactive shells for macOS are zsh and fish.

The good news here is that you can set one shell for interactive use, and still continue using and writing bash scripts and get the best of both worlds.

The default interactive shell is set in your user record, you can change this in the ‘Advanced Options’ in the Users & Groups preference pane, or with chsh command:

$ chsh -s /bin/zsh
Changing shell for armin.
Password for armin: 

The shell that interprets a script is, of course, set with the shebang.

Summary

  • the first line in a shell script starts with a shebang #! and tells the system which tool interprets the script
  • env should only be used in the shebang when the script needs to
  • macOS administrators should use /bin/bash as the shebang
  • the ‘step up’ from bash scripting should be a more complex language with native macOS API access such as Python or Swift
  • you can use one shell for interactive Terminal work and another for scripting

Weekly News Summary for Admins — 2017-10-13

Squeezing in so much work between a week of conference and a week of vacation. I will be mostly offline next week and there will be no posts, updates or news summary next week. (Sorry!) Me and the summary will be back in the week after!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

🍏Apple

♻️Updates and Releases

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

High Sierra Update for ‘Property Lists, Preferences and Profiles’

You may have already seen it in your iBooks app: the High Sierra update (v1.2) for my PR3 book finally cleared. There was a broken hyperlink in the book somewhere and that confused the automated verification system. Support was very helpful in helping me figure out the problem and fix it!

You can learn more about the High Sierra changes for PR3 and the ‘Packaging’ book in this post.

If you have already bought the book, you can check the ‘Updates’ section in iBooks to get it. Or you can go buy the book!

Weekly News Summary for Admins — 2017-10-06

Bit late today because of MacSysAdmin. Thanks again to the organizers and speakers. Such a great conference!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

🍏Apple Support

♻️Updates and Releases

📺To Watch

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

MacSysAdmin 2017 Session Video(s)

The video of my MacSysAdmin session: “Scripting Bash” is already online:

MacSysAdmin 2017 Session Documentation (Direct link to video)

Look at the other sessions that have been posted (so far), too! Most of them go online within a day, which is a quite impressive turnaround.

All the links and resources for my session can be found here.

This is my first time presenting and attending MacSysAdmin 2017 and I have to say it is a wonderful conference. I am flattered to be part of this group of amazing speakers as well as all the awesome attendees. It is well organized in a gorgeous (if wet) location (Göteborg).

My thanks to the organizers for putting this together and I hope to return next year.

Weekly News Summary for Admins — 2017-09-29

Welcome all the new readers and subscribers. My post summarizing High Sierra News for Admins certainly caused some extra traffic. Yesterday was the busiest day on this website ever, beating even the book launches by a good margin.

I will repeat some of the links from the High Sierra post here, so they don’t drown completely in that long list. A huge thank you to everyone in the MacAdmin community who share their time and knowledge, so we can all be better admins.

Next week, I will be attending and speaking at MacSysAdmin in Göteborg. I am really looking forward to meeting everyone there (again and for the first time). Feel free to say “Hi!” and introduce yourself.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

🍏Apple Support

♻️Updates and Releases

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!