macOS Installation: Strange New World

This is a section out of my next book: “macOS Installation for Apple Administrators”
This book is not quite finished and I don’t have a fixed timeline, other than “soon…” I will post more about this book as things get more concise.

High Sierra came with many changes for Mac administrators. Some of them were anticipated, some of them weren’t.

When Apple announced APFS would be deployed to every OS by 2018, Mac Admins thought that would be the main challenge and the reason for sweeping changes in the structure and functionality of macOS.

At the same time it was obvious that Apple liked the way MDM was in the center of all management for iOS and wanted to have it play a similar role for macOS. The extent how far Apple would push it was unclear.

While APFS has certainly come with some challenges, mainly for managed FileVault users, it was not the “Macnarök” that we were expecting.

I am not including these links to make fun of the authors – quite the opposite. At the time there was justified anxiety about the future of macOS adminstration and these posts (and many others) put this into words. I believe that because of MacAdmins blogging and presenting about their concerns and anxiety, talking with Apple Reps and filing bugs, as a community, were able to influence the course Apple is taking. It is definitely a slow, frustrating and opaque process, but not entirely fruitless.

However, other new “features” of High Sierra such as UAKEL and, later with 10.13.4, UAMDM provide their own set of hurdles administrators have to clear.

These are the main challenges for macOS deployment:

  • Imaging is Dead, Long Live the Installer
  • MDM is Essential
  • Secure Boot

Let’s look at these challenges and some solutions to them in detail.

Imaging is Dead, Long Live the Installer

Imaging has been the means to quickly and reliably install macOS to clients. It could scale from an external drive for a few clients to a NetBoot infrastructure for large deployments.

Apple has come out explicitly and said that imaging in most forms is not recommended or supported any more.

In the early days imaging meant building a ‘Golden Master’ image with all configuration and software ‘baked in,’ which then would be deployed to all the Macs. Creating this ‘Golden Master’ image was a lot of work and the result quite fragile and error-prone. Macs would usually be kept on a specific version of ‘the image’ for months or a year without updates.

Since the process of manually building the Golden Master was tedious and error-prone, administrators started to automate the process. Tool chains of packages, scripts and other tools could build the master image (or most of it) automatedly.

InstaDMG was one of the first tools to do this. AutoDMG is a direct successor of this approach and still useful for many tasks today.

The same packages and scripts that automated the configuration of the master image could be used to install and update software on ‘live’ Macs. Management tools, like Munki, Casper (now Jamf Pro) and Filewave, can deploy updates to clients automatically without user interaction. As work was moved from preparing the master image to maintaining the live system, the master image became thinner and thinner.

This resulted in the ‘thin imaging’ approach, where the image only had the minimal ‘base OS’ plus enough configuration to connect it to the management system. Once the Mac booted the management system takes over. Imaging would only be used to wipe a Mac and lay down a ‘base image.’

While this approach is slower than monolithic imaging, it is more flexible and allows to keep the system and software on the client continually updated, without having to re-image.

Some workflows even skipped the imaging entirely. When you get a Mac out of the box it already has macOS installed, all you need is to configure it to connect to your management system, which will take care of the rest.

Some administrators had already moved to ‘thin’ imaging or ‘no-imaging’ workflows before High Sierra and pronouncements that ‘Imaging is Dead!’ have been made years ago. Administrators using thin imaging or no-imaging workflows will find they are better prepared for the High Sierra installation workflows.

Mac administrators now have to build new workflows based on the Apple’s supported installation methods for High Sierra. These all rely on the macOS Installer application in one form or another.

The supported means of installing macOS High Sierra are:

  • the macOS Installer application
  • a bootable installer on an external drive
  • the macOS Recovery System
  • a NetInstall image built with Apple’s System Image Utility

Using the macOS installer application to upgrade a system from 10.12 is straightforward enough and similar to what was possible with the startosinstall command since OS X Yosemite 10.11.

The main challenge are ‘nuke-n pave’ workflows, where the system on a Mac is erased and overwritten with a new system.

This workflow is easy with traditional imaging. In fact, it was easier and faster to just image over a system, rather than upgrade.

The startosinstall --eraseinstall option added in 10.13.4 makes an installer based ‘nuke’n pave’ workflow possible with the macOS Installer application.

In the new High Sierra world of Mac deployment, we face the same challenge: how do you connect an ‘out-of-the-box’ macOS to your management system.

Apple’s answer to that is clearly DEP.

MDM is Essential

DEP will tell a freshly installed Mac which MDM server is ‘responsible’ for it. The MDM server can then use the InstallApplication MDM command to install and configure a management agent, which together with the mdmclient performs the remaining configuration and software installation.

Up until 10.13 Mac administrators could ignore and not use an MDM for Mac deployment and management. There were a few useful features, but there are also many tasks an MDM cannot do or that a local agent can do better.

With High Sierra MDM became essential to manage and support UAKEL. In the early versions of High Sierra, merely being connected to an MDM would disable UAKEL, starting with 10.13.4 UAKEL can be managed with a configuration profile which has to be pushed from a user approved MDM.

Prepare for changes to kernel extensions in macOS High Sierra – Apple Support

Like supervison for iOS devices, UAMDM adds an extra level of approval and provides an additional level of control over the macOS client device. UAMDM will be required for more and more essential functionality going forward.

However, as the name implies, a ‘user approved’ MDM needs to be interactively approved by a user at some time during deployment which creates an enormous challenge for automated workflows.

Apple claims that DEP enrolled devices are automatically user-approved. However, even the DEP enrollment still requires user interaction in the Setup Assistant.

Also DEP is not available in all regions and it may not be possible for organizations to add all existing Macs to their DEP account. DEP also requires Apple DEP servers to be accessible. So during this transition phase and for some other situations, Mac administrators have to have a non-DEP workflow solution to get a UAMDM enrolled device.

Secure Boot

Finally the iMac Pro introduces a new system controller with secure boot. By default all external means of booting an iMac Pro are disabled.

NetBoot/NetInstall is not supported on the iMac Pro at all. You can enable booting off external drives, but the process to do so is quite convoluted and cannot be automated. So for administrators, the option is fairly useless.

With the startosinstall command administrators can achieve the necessary management steps from within the local system. You can either upgrade the existing system or ‘nuke’n pave’ with the eraseinstall option.

However, the eraseinstall option is limited to systems that are already on 10.13 with an APFS file system. On the iMac Pro you can safely make that assumption, but for ‘older’ Macs and systems you usually cannot.

So administrators need a second workflow for systems and hardware that cannot run the eraseinstall option (yet).

The Recovery system, external installer volume or NetInstall are the Apple Supported means which can ‘nuke’n pave’ a Mac. NetInstall is the only means that can be automated.

However, there are other solutions such as Imagr or a modified DeployStudio workflow which can achieve the same goal using the macOS Installer application and startosinstall.

The New macOS Deployment Workflow

Deployment strategies and workflows are as varied and individual as the organizations that use them. But with the challenges listed above, I believe we can state a few pieces that are necessary for a new deployment workflow for High Sierra and hopefully going forward.

Enrolling a ‘new’ Mac

First, any deployment workflow should be designed to start with a clean ‘out-of-box’ macOS. This way the same workflow can be applied to new Macs and to Macs that were just re-installed.

At the end of the installation workflow the Mac is enrolled and user-approved with your management system, so that it can take over and perform the remaining configuration and installation. All configuration steps should be controlled from the management system and be executed independently of which deployment workflow was used.

Deployment Workflows from an out-of-box Mac

DEP is the Apple-preferred means of connecting a new device to your MDM. Even when you require a non-DEP workflow for older Macs you should implement a DEP workflow. It is likely that Apple may make DEP enrollment mandatory in the future.

However, right now in the transition phase, you will need a second workflow for machines that cannot be registered with DEP.

For the non-DEP workflow, you can choose between manual enrollment, which happens after the user has booted and logged in for first time, or some sort of ‘side-loading’ where you add the MDM and management system configuration to a system, before it even boots for the first time.

For sideloading you can boot to Recovery and install software on the system volume. (Bootstrappr)

On Macs that still support NetBoot, you can also use tools like NetInstall, Imagr or DeployStudio. Either way you should do this before first boot of the system.

UAMDM requires user interaction to approve the management in some form or another. This is a change from previous deployment workflows which could be ‘fully automated.’

You will have to plan for this user interaction. In some cases, such as individual-use devices, the approval of the management will be part of standard first setup and will be done by the end user. In other situations such as classroom setups, it will be harder to make this part of the workflow.

Some management systems let you scope software installations and configurations depending on criteria reported from the device. In this case you can set up your workflow so that installations that rely on UAMDM (third party kernel extensions) are deferred until the MDM is approved.

Either way, the end-result of each of these paths should be a Mac that is enrolled (user-approved) with your MDM where the management system is installing, configuring, maintaining the system and software.

Restoring a Mac to ‘new’

You will also need a second set of workflows that bring a Mac from any state (even a broken one) back to the ‘out-of-box’ state so that you can start over (or retire it).

Restoring a Mac to new

The first option uses the startosinstall --eraseinstall command. This is easiest to automate and works on new hardware like the iMac Pro. However, it will not work on Macs without APFS systems and older versions of macOS.

The Macs that cannot use the eraseinstall option, however, can use NetInstall or a similar NetBoot based tool. You can use the macOS Install application and startosinstall with Imagr and DeployStudio, so this combination will run the installer and can make sure the firmware is up to date as well.

Note that the workflows and tools for restoring a Mac are also used to ‘sideload’ the agent configuration on to a Mac. You can add the MDM/agent configuration to the restore workflow to save some steps here. However, since NetBoot based workflows are transition solutions to support ‘old’ Macs and systems, your deployment workflows need to work with ‘out-of-box’ systems as well as pre-configured systems.

In the transition phase until all hardware supports eraseinstall you will need both workflows available.

You should also have documentation, training, and support articles to instruct administrators, techs, and users to restore a Mac to ‘out-of-box’ state with (Internet) Recovery or an external boot drive. While this should be an option of last resort, it will be necessary in situations where a Mac cannot boot otherwise.

Obviously, these are very high level views of the workflow and the devil is very much in the details of the implementation. Even a book can only skim the surface of any implementation. Nevertheless, once you understand and accept this framework, the necessary detailed steps become obvious, though not necessarily easy.

Many thanks and credit need to go to Anthony Reimer, whose excellent presentation at MacAD.UK 2018 helped me sort through my own thoughts on this topic.

Thoughts on Books

Apple had an event yesterday at a school. You may have heard.

If you are not working in education tech the news probably seemed underwhelming. Don’t worry, Apple will have more events and some of them will have great news for you.

I am not going to discuss the entire event. I don’t currently do much work in education IT and when I do it is for Macs, so I don’t feel very qualified to address most of what was shown yesterday.

However, there was one topic on which I was hoping for news: iBooks. However, as it is often the case with Apple, it was neither the news I was expecting nor hoping for.

In case you didn’t know: I am interested in iBooks because I have published two books written with iBooks Author.

iBooks was introduced with the iPad in 2010 and the Mac version followed in 2013 (with Mavericks). Along the way Apple also introduced iBooks Author in 2012 which allows building rich media, interactive digital books tailored for the iPad. (Though they also work on Mac and iPhone.)

Since then the iBooks app, iBooks Store and iBooks Author have received a few updates. Overall, Apple’s attention to the books part of their software and services can only be described as ‘lacking.’

There were some signs in the recent iOS beta releases that Apple seems to be preparing to change the iBooks app’s name to simply ‘Books’, and iBooks Author has found its niche in education, so this education focused event seemed like a place for an announcement. The hopes were up!

Pages 7 vs iBooks Author

As I said, the announcement on the book publishing front is underwhelming. The new Pages (version 7.0) has improved ePub3 fixed layout support and some new templates.

This is not a new feature. Pages has had ePub export for a long time. So far it has focussed on free-flowing layout. Last year, Pages 6.3 gained the ability to create and export fixed-layout ePub formats. Back then I was really interested to have a solution other than iBooks Author. However, the feature set of Pages for fixed layout is quite rudimentary compared to iBooks Author. I have not had time to really test the latest version of Pages yet, but I can already confirm that some really important features are still missing.

iBooks Author forces you to structure your book with chapters and sections, but also builds an interactive table of contents automatically. iBooks Author provides interactive widgets for images, movies, and even Keynote presentations that go fullscreen on tap. Also scrollable sidebars which I use in my books for scripts and source text that is longer than fits on a screen. I can also create links in text to other parts and objects in the book.

None of these features are present in Pages.

In iBooks Author you can send a preview to iBooks on a connected device. In Pages I have to export as ePub and find some way to get it to the target device.

But the most damning thing is, there is no way to import an iBooks Author document or template into Pages. I’d have to start over and rebuild the book in a new layout.

Future of iBooks Author

However, the situation is not all bad. Apple has also stated that iBooks Author is not going anywhere. For now I would presume we are in a transition phase, where the ‘old software’ is kept around for compatibility and certain features until the replacement can live up to expectations. Apple has done that before with iMovie, QuickTime and even AppleWorks.

I’d rather have a brand new iBooks Author or a feature complete Pages sooner than later, but as long as iBook Author keeps working, well, I guess I can wait.

So, what am I going to do with my books? You might have guessed from my analysis: I am sticking with iBooks Author for the existing books.

I have about 500 pages in several books in various stages of completion. As long as there is no easy way to export them out of iBooks Author, they are staying in there.

iBooks Publishing Frustrations

However, there are some limitations of iBooks Author that I am quite aware of and that I have been looking for solutions to.

Since the topics of the books are very Mac specific, I felt that restricting myself to the iBooks Store and Apple platforms for publishing wouldn’t be much of a limitation. (Though I am still surprised that I repeatedly have to tell people iBooks exists on the Mac and you don’t need an iPad to buy and read iBooks.)

However, self-publishing in the iBooks store is not available in every region. Even some countries where an iBooks store is available don’t allow me to publish to them.

It is not easy or even possible for everyone to get a US Apple ID to purchase items in the US store. So far I have only gotten a handful of requests from readers with this problem, but I do feel sorry for every one of them (and am aware that there will be many more who just gave up without asking).

I’d also like to be able to write not just on a Mac but on an iPad or even the iPhone, whenever the inspiration (or boredom) strikes me. Right now I solve that by typing (and sometimes even dictating) my ideas into Notes and then copying them into iBooks Author later. However, this workflow does not allow me to review or rewrite parts of a book on the road.

Finally, as an admin and coder, I have gotten used to git and some sensible form of version control for the writing process would be nice, too.

The Leanpub Alternative?

So, to address these problems I have been looking for an alternative means of publication. Last week I was recommended Leanpub (thanks, Rob!). I believe their approach to publishing is well suited for technical books.

However, their workflow is entirely based on a markdown variant and builds free-flow ePub books. That means they are more flexible and can support many more reader platforms. On the other hand, the options for rich media and interaction are much more limited (if non-existent). I am not even sure if videos can work reliably in this format (though I plan to try).

On the other hand Leanpub provides better rates for publishers/authors and have a ‘publish early and often’ mentality. They actually allow (and encourage) publishing and selling an ‘incomplete’ book and pushing updates to customers. They even integrate git into the publishing workflow.

As far as I can tell they also sell to anywhere in the world.

Plans

So to summarize here are my specific (mid-term) plans:

  • I will keep ‘Packaging’ and ‘PR3’ in iBooks Author format and the iBooks Store for the forseeable future. Right now converting them to another format would take too much work and time that I’d rather spend on other projects. I will keep updating these books with fixes and new topics.
  • There is a third book that is more than half complete which I plan to keep in iBooks Author format for the same reason (this one has quite a number of screen capture videos, which would be a pain to convert to something else). I hope to get to a point where I can talk more about this project soon.
  • Finally, I have (yet) another project, which I believe is well suited for Leanpub. Last summer, I posted a series of ‘Terminal Primer’ articles and have kept working on that series with a ‘Terminal for Apple Adminstrators’ book in mind. I will (re-)build and publish this in Leanpub as an experiment. For now there is only a ‘stub’ page available, but you can sign up for updates on this project. Of course, I will publish updates on this project here and on Twitter as well.

Let me know what you think!

Update 1.7 for ‘Packaging for Apple Administrators’

The advantage of self-publishing a digital book is that I can write and make updates. Some people have pointed out typos (thank you!) or had questions or comments about the book.

In some cases a question from a client or a MacAdmin Slack forum member will make me re-visit a section of the book and then add or re-write a paragraph or two.

The really great thing is that when you have already purchased the book, you will get the new improved update for free! And even if you have not bought the book yet, then you can know that when you buy it now, you will keep getting improvements in the future. It’s all just bits and bytes after all.

All of that said, the latest version of my book ‘Packages for Apple Administrators’ is now available on the iBooks Store.

I have fixed many typos and clarified a few sentences and paragraph. I have expanded and clarified the instructions for signing packages. There is also a new segment on the order of installation with distribution packages.

If you have already purchased books, all I ask in return for the free update, is to go to the iBooks Store and leave a review. The iBooks Store segregates reviews by territory, so every single one of them will be very important for other users to find and evaluate the book.

Thank you!

defaults – the Plist Killer

Last week, fellow MacAdmin Kyle Crawford has discovered that on macOS High Sierra the defaults will delete a property list file with invalid plist/XML syntax, even when you just attempt to read data. Erik Holtham has a more detailed OpenRadar bug.

Patrik Wardle has found the relevant code, which deletes the file when it doesn’t validate.

This is bad. Thanks to all involved for finding, documenting and sharing this.

This is new behavior in High Sierra. I am not yet sure which version of High Sierra this new behavior was introduced. The behavior makes sense in the context of an application attempting to read a settings file, but the defaults tool deleting arbitrary files is, of course, dangerous.

Update: This behavior has been fixed in 10.13.4. However, it is still good practice to avoid defaults for anything other than actual preferences files.

What to do?

As usual, don’t panic. This will only affect Macs running High Sierra and corrupt or broken files. However, if you have a script that accidently points the defaults command at a different file, it will delete that. So you have to use it with care.

It is probably a good practice to verify a file before you attempt to modify it with the defaults command in a script with the plutil command:

if ! plutil -lint path/to/file.plist; then
    echo "broken plist"
    exit 1
else
    defaults path/to/file …
fi

Alternatives to defaults

Alternatively, you can and should use plutil or PlistBuddy to read and modify property list files.

Learn more about plutil, PlistBuddy and other tools to read and write property lists in my book: “Property Lists, Preferences and Profiles for Apple Administrators”

plutil is unfortunately not really useful to read a single value from a property list file. The extract verb will show any value as its own plist. The plutil command is useful to edit existing plist files. (Read details on the plutil command.)

PlistBuddy, however is very useful for both reading an writing values to a property list file:

$ /usr/libexec/PlistBuddy berry.plist -c "print size"
$ /usr/libexec/PlistBuddy berry.plist -c "set size enormous"

PlistBuddy has the additional advantage of allowing to add or edit values nested deep in dict or array structures.

You can get more information on PlistBuddy in its man page or my book.

The interactive mode of PlistBuddy is also very useful.

So the defaults command is dead?

No.

Apple has been warning us to not use defaults for generic property list file editing and parsing for quite a while in the defaults command’s man page:

WARNING: The defaults command will be changed in an upcoming major release to only operate on preferences domains. General plist manipulation utilities will be folded into a different command-line program.

As this warning states, the defaults tool reads and writes data to plist files through macOS’s preferences system. This has the advantage that the tool gets (and changes) the current value whether it is cached in memory or not. When an application is listening for notifications that a preference has changed (not many do) then it will be notified.

Files for preference domains are usually stored in /Library/Preferences/, ~/Library/Preferences or their ByHost subfolders. Sandboxed applications will have their preference plist files in their container.

There, however, many other files that are property list files which are not part of the user defaults system: launchd files, configuration profiles and AutoPkg` recipes to name just a few.

Mac Admins commonly use the defaults tool, despite Apple’s warning, to create, read and edit generic plist files. As mentioned above, plutil, PlistBuddy or direct manipulation through Obj-C, Python or Swift, are better choices for generic plist files.

You can learn about all these options in my book: “Property Lists, Preferences and Profiles for Apple Administrators”

Another Update for Packaging Book

I pushed another update for “Packaging for Apple Adminstrators.”

Greg Neagle posted an article recently where he describes how he sleuthed a way past an annoying High Sierra bug.

In High Sierra when you add more than one package to a custom NetInstall workflow with System Image Utility or startosinstall all the custom packages will fail.

Greg’s workaround involves adding an identifier and version to each of the distribution packages. It is still an open question why this data is required, but the workaround is easy enough.

I have added a section on how to do this to the “Building Distribution Packages” section of the book. As always: if you have already purchased the book, you can download the update with new content for free in the iBooks application on your Mac, iPad or iPhone. Or go and get the book now!

Rich Trouton also updated his “First Boot Package Install Generator Tool” to provide this data.

One year of “Packaging for Apple Administrators”

I published my book “Packaging for Apple Administrators” one year ago. Flashback time (insert harp chord sounds)!

When I started out the goal was to write the big guide on everything a Mac Admin needs to know. It quickly dawned on me that this would result in a huge tome which would be impossible to write and maintain. I realized that with digital publications, I could publish smaller books and could “publish early and often.” I would publish a specialized book and update that when I learnt new information or when software updates revealed or retired certain strategies. That realiziation freed me of “perfection anxiety.”

The choice for the first topic was easy. Analysing and creating package installers is an essential skill for Mac Administrators. But it is also poorly documented. Apple has no real documentation on the package format on their website. Also no best practices. For years the Mac Admin community has been very helpful on guiding newcomers through the process, but in the end everyone relied some information transfer through “osmosis”. New admins would pick up bits and pieces on how to build packages, what to do and what not to do from posts in IRC and Slack, the occasional blog post and maybe a presentation or workshop at a conference.

This was the perfect “low-hanging fruit” topic to start out with. After about six months of part time writing it was in some form that I was not embarassed too much about any more. That was last November.

I have since pushed four updates to the books. (The first 1.1 update was to remove some placeholder text that was persisting in the default Glossary entries. The iBooks store reviewers don’t like that.) The other four updates have added two appendices, eight new sections and countless other fixes, extensions and clarifications. And everyone who bought the book got the updates for free.

I believe I now have included most of the fundamental concepts on packaging. However, some sections might profit form having more examples. In particular more complex examples would be useful. My main challenge in providing complex examples is that I cannot use software which requires pricey software licenses. However, this is usually the most problematic software which requires complex and challenging workarounds. (If you have recommendations for useful, free software, which requires complicated packaging workarounds, please let me know.)

So keep expecting updates and new content!

So what’s next? The obvious sequel for “Packaging” seemed to be “Automated Packaging” and would cover AutoPkg. The AutoPkg book is sitting, half-written, on my computer. However, it turns out that AutoPkg requires much more prerequisite knowledge. I found myself adding chapters and sections on other topics. One of these topics grew enough that I published it as its own booklet: “Property Lists, Preferences and Profiles for Apple Administrators” or “PR3” for short.

While “Packaging” and “PR3” are selling and I am getting lots of wonderful feedback from the readers, the numbers sold do not justify going in on writing full-time. This does not come as surprise, since Mac Administration is quite a niche market and I have neither time nor budget for any form of marketing, other than word-of-mouth. (So, please, if you liked the books and want to do me a favor, tell some fellow Mac admin about my books, and/or leave a review on the iBooks store.)

Also at that time (Spring 2017) the rumors of Apple drastically changing access to macOS, even (or especially) for administrators and their tools were rampant. Finally, Jamf was promising a great new “Patch” feature for the long-expected Jamf 10. All of these trends together might have invalidated the information in a book on AutoPkg (and maybe even the Packaging book). It seemed like a good time to put the AutoPkg book on the back burner and write about something else.

I have written a lot of blog articles over the summer. I am currently sorting through them, filling in the sections and chapters and trying to assemble useful books from them.

I have also presented at MacSysAdmin in Göteborg and am looking forward to presenting at more conferences in the future. Finally, I recently started working as a System Engineer/Consultant for a Dutch reseller. While all this other work is limiting the time available for writing, I also expect it to inspire real-world experiences which should lead to better writing, both on this weblog and the books.

So, one year after “Packaging”: the books are still being updated, the Macpocalypse did not happen and packaging is still a skill you need to have as a Mac Admin! Expect more books sometime in the future! Until then, buy the books I have!

(If you have already purchased and read the books, please leave a review on the iBooks store. The iBooks store segregates reviews per territory, so every single one counts! Thank you!)

High Sierra Update for ‘Property Lists, Preferences and Profiles’

You may have already seen it in your iBooks app: the High Sierra update (v1.2) for my PR3 book finally cleared. There was a broken hyperlink in the book somewhere and that confused the automated verification system. Support was very helpful in helping me figure out the problem and fix it!

You can learn more about the High Sierra changes for PR3 and the ‘Packaging’ book in this post.

If you have already bought the book, you can check the ‘Updates’ section in iBooks to get it. Or you can go buy the book!

Book Updates for High Sierra

Happy Update Day!

macOS High Sierra (10.13) will be released some time tonight. There have already been many articles on many of the new features (or issues) in High Sierra, especially in my Weekly Newsletter. But how does High Sierra affect my books and information therein?

The good news is: surprisingly little. There were many rumors and concerns in the build-up to WWDC this year, but the worst did not happen. I posted about my reaction to the news in WWDC here.

Nevertheless, the tutorials in the books needed to be tested on High Sierra and there were quite a few changes that had accumulated over time so I threw those in as well. The advantage of digital books is if you have already purchased the books (Thank you!) you will get these updates for free in the iBooks Store (you might have to check in the ‘Updates’ tab).

If you don’t have the books yet, you can go and buy them now and get future updates to these books as well!

PR3 is still in review limbo, but should be through soon. (Update: Available now). ‘Packaging 1.5’ is available on the iBooks store already!

If you have already purchased books, all I ask in return for the free update of new information, is to go to the iBooks Store and leave a review. iBooks Store segregates reviews by territory, so every single one of them will be very important for other users to find and evaluate the books.

Thank you!

Some notes on each of the books in particular:

Packaging for Apple Administrators

The basic tools and methods for packaging in High Sierra have not changed. But since I had to go through the book to test all the examples again, I made quite a few minor corrections and clarifications.

Note: The current version of Whitebox Packages, does not run on High Sierra. I believe there will be an update soon, so I did not change the section in the book to reflect that right now.

I also added two entirely new sections: (not dependant on High Sierra)

  • a simple example on how to build Un-Installer scripts, something macOS does not automatically provide.
  • based on this blog post: how to extract a component from a distribution package.

Other than that, Packaging remains very relevant to a Mac Administrator’s skill set with High Sierra, so go and get the book! (and please leave a review)

Property Lists, Preferences and Profile for Apple Adminstrators (PR3)

Note: as mentioned before, PR3 is still in Apple review limbo. I will post as soon as it clears. (Update: it cleared!) If you haven’t bought it yet, you can buy the current version now and will get the update pushed in iBooks, as soon as it clears review!

First, with High Sierra comes Swift 4, which brings a new Property List serialization API. I added new sample code to the Swift section for Swift 4.

Second, the profiles tool comes in a new version in High Sierra, with new syntax and some new functionality. You can see the new command syntax in the man page of the profiles command in High Sierra. (You can also still get the old syntax on High Sierra by calling the man page for profiles.old.)

However, while the old syntax is considered deprecated the new version on High Sierra still supports it. So there is no reason (yet) to run out and change your scripts. Nevertheless, both versions are documented in the relevant section in the book.

There is some new functionality in the new syntax (startup type profiles) and I assume that new features will only added to the new syntax going forward. As long as you still need to support Sierra Macs and older, you will have to use the old syntax or maintain both versions.

And, like the Packaging book, while I was working through the examples in the books, there are many corrections, additions and small clarifications added.

With many interesting new features in MDM, profiles will increase in relevance for adminstrators. Go get the book! (and please leave a review)

Terminal Primer – Part 4 – Commands

Arguments

So far we have use three commands: pwd, cd, and ls

These commands are already quite different.

pwd is a single word command. You enter it and it prints information (the working directory to the terminal).

cd, however, requires additional information from you: where do you want to change to? The cd command requires an argument:

$ cd ~/Documents

(You can enter cd without an argument, and it will change to your home directory, but usually you want an argument.)

The command itself cd and the argument ~/Documents are separated by a space.

Some commands can have more than one argument. In that case all arguments are separated from each other by a space. (Or more. bash doesn’t care about multiple spaces.)

This is why we have to treat spaces in paths and filenames so carefully, because otherwise the shell might interpret the path as two or more arguments.

Finally ls has an optional argument. When you just write ls. it will list the contents of the current working directory. When you give an argument it will list the contents of that path. The ls command also has several options that modify its behavior.

When a shell command is written in documentation optional arguments are usually enclosed in square brackets:

ls [-options] [path]

Mandatory arguments, on the other hand, are shown without the square brackets.
When you enter an ls command with completely wrong options (surprisingly difficult, since its options cover nearly the entire alphabet, and some extra characters as well.) it will print a “usage” line:

$ ls --a
ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

The extra ... after the optional file command tells us, that you can give ls more than one path argument:

$ ls ~/Desktop ~/Documents

Read the Manual

When you want detailed information on a command, there are a few approaches.

Because of the long and diverse history of shells, bash and macOS in particular, not all commands support all of these options. Behavior here can be very inconsistent.

First, as we just saw with ls, some commands will print a brief usage note, when you enter something that the command cannot parse.

With some commands you can provoke the usage message with the -h or --help option:

$ sw_vers -h
Usage: sw_vers [-productName|-productVersion|-buildVersion]

The usage message is commonly very brief and usually does not explain all the options.

To get more detailed in information on command you can read its man page. man pages are documentation, often very detailed, stored in an file format optimized for display in ASCII terminals.

To get the man page for a command run the man command:

$ man ls

This will take over the current Terminal window and display the information.

This special display mode is actually controlled by another command called less. There many key commands you can use for navigation in this mode.

Key
q exit and return to command line prompt
up/down arrow scroll up/down by a line
space or z scroll down by a page
w scroll up a page
g top of document
G (shift-g) end of document
/word<return> find next occurrence of word in document
n find next occurrence of search term
N find previous occurrence of search term
h help

You can also scroll in this mode with the mouse wheel or two-finger scrolling on a trackpad.

You can also open man pages in terminal from the Help menu. When you enter a shell command in the help search field of Terminal it will suggest a man page, when one is available. When you select a suggested man page, it will open in a new yellow window.

You can modify the appearance of the man page window by changing the ‘Man Page’ window profile in Terminal’s Preferences.

You can also open a man page by selecting text and choosing ’Open man page from the context menu.

Some commands are ‘built-in’ to the bash shell. These do not always have man pages. Requesting the man page for a built-in command will show the man page for builtin instead.

cd is one example for a built-in command.

You can get documentation for built-in commands with

$ command help cd

Finding commands

We just learned that some commands, like cd, are ‘built-in’ to the shell. Others are not, so what and where are they?

All commands are files in the file system. They have a special file privilege set which makes them executable. Obviously, you cannot make any file executable, it has to have some form of code which makes sense so the system can interpret it as commands.

If you want to know where a given command resides in the file you can use the which command

$ which ls
/bin/ls
$ which sw_vers
/usr/bin/sw_vers

However, you do not have to type /bin/ls every time you want to execute ls. How does the shell know where to look?

The shell has an environment variable called PATH which contains a list of directories where it will look for commands that are typed without an absolute path. You can print the contents of this variable with the echo command:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Note: commands and variable names in the shell are case-sensitive. It is convention that environment variables are written in all-caps. You have to use the correct case for the PATH variable to get or set the proper value.

When you are new to shell and bash, there is a lot to process in this simple command, so let’s take this apart piece by piece:

The echo command simply tells the shell to print something to the terminal, so

$ echo hello
hello

prints ‘hello’ back to the terminal. This alone is rarely useful, but can be used to get at the results of another process.

$ echo $(( 6 * 7 ))
42

The $(( … )) means ‘evaluate this arithmetically,’ so this command prints the result of this arithmetic to the terminal.

In general in bash the $ stands for ‘substitute contents.’ echo $PATH means: print the contents of the PATH variable.

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

When you forget the $ and just write

$ echo PATH
PATH

bash will interpret PATH as a literal string and prints it to the terminal.

The actual contents of the PATH variable is a list of directories separated by colons.

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The order of the directories in the PATH is important as the shell will stop looking when it finds a command.

When you enter a command without a path, e.g. ls, bash will start looking for the command executable in /usr/local/bin, then in /usr/bin, and then in /bin, where it will find an executable ls, stop looking and execute that.

Note: if there were another executable named ls in a later directories it would not be used, since the shell will stop looking at the first match it finds. Changing the order of the standard directories in the PATH or even inserting other directories earlier in the PATH can lead to unexpected behavior.

The PATH on your system may be different when you have extra software installed. Xcode, Server.app, Xquartz, munki, Python3 and many other software packages insert paths to their command directories in the search path.

Note: some software solutions will attempt to modify the PATH on a system to make their commands available to the shell, other will place the commands or links to the commands in /usr/local/bin to make them available (e.g. text editors like BBEdit or Atom).

We will look at strategies to on how and why to modify the search path later.

Some third party solutions will instruct you to modify the PATH to include their commands rather than doing it during the installation.

Running Other Commands

When you need to execute a command or script that is not in the PATH, you have to type the full or relative path to the command:

$ /usr/libexec/PlistBuddy
Usage: PlistBuddy [-cxh] <file.plist>

or

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport --getinfo

These are commands that are usually considered too uncommon or maybe even dangerous to put in the standard search paths.

When you start using and writing custom-built scripts and commands, you can use relative paths:

$ test/my_great_script.sh

or

$ scripts/postinstall

When you need to execute a command or script in the current working directory, you have to start the command with ./, so the shell knows to not look in the search path.

$ ./buildMyProject.sh

Remember the . is a shortcut representing the current working directory.

Tab-completion for Commands

You can use tab-completion for commands as well. This will speed up your typing and prevent typing errors.

You can use this to get a list of all the commands available in the shell. At an empty command prompt hit the tab-key twice. Then shell will warn you that there are many completions (more than a thousand, depending on your version and configuration of macOS.

You can also use this command to list all tab-completions:

$ compgen -c

Note: compgen is the command that bash runs to determine which commands are available for tab-completion. You usually would not interface with it directly.

Terminal Primer – Part 3 – Special Characters

There are a group of characters that have special meaning in bash.

| & : ; ( ) < > ~ * @ ? ! $ # [ ] { } \ / ' " `

Also whitespace characters also need to be treated with care:

space, tab, return, and newline

The space character is a legal and often used character in file names on macOS. However, in bash and other shell commands , a space character (and other whitespace characters) separates a command from an argument and the arguments from each other.

When you try to enter a filename with a space, you will get an error:

$ cd /Library/Application Support
-bash: cd: /Library/Application: No such file or directory

To convince the shell that ‘/Library/Application Support’ belongs together, you can either ‘escape’ the space character or ‘quote’ the path.

Experienced users who have worked in a UNIX environment for a long time tend to avoid these special characters in filenames. However, as a system administrator, your users will probably not heed any rules you may want to impose. You will have to deal with many possible combinations.

Escaping Characters

The escape character in bash is the backslash \. A character that follows a backslash will be treated with no special meaning:

$ cd /Library/Application\ Support

In Finder, you can name files and folders nearly any way you want. When you encounter special characters from the list above you have to escape them with backslash. For a directory named ‘Project (Old & New)’ you would type:

$ cd Project\ \(Old\ \&\ New\)

All of this further confused by the fact that the shell will happily display the path with the unescaped special characters:

$ pwd
/Users/armin/Project (Old & New)

Separation Characters

In bash (and in Unix in general) files and directory names cannot contain a forward slash / since the character is used in paths to separate directories. However, Finder lets you name a file or folder with a forward slash, e.g. ‘Yes/No’.

On the other hand, Finder does not let you name a file or folder with a colon :. The underlying HFS+ file system uses the colon as a separator.

This conflict is solved by translating a / in the Finder (and underlying file system) to a colon : in the shell and vice versa.

A folder named ‘Yes/No/Maybe’ in Finder will appear as Yes:No:Maybe in the shell and you have to escape the colons when using the path in a command:

$ cd Yes\:No\:Maybe

Note: some characters that are legal on macOS might not be on file servers, which are usually hosted by other operating systems.

Quoting

As seen above, escaping characters can make the path quite unreadable. You can also place the name or path in quotes:

$ cd 'Project (Old & New)'

In bash you can use single quotes ' or double quotes " to quote paths.

Single quotes are more effective. Any character in single quotes is used as is, with no special function. Even the backslash character has no special function. The only character you cannot use in single quotes is the single quote itself.

Double quotes " are ‘weaker’ quoting. Double quotes remove the special function from all special characters except $, `, \, and !. Within double quotes you can use the backslash to escape $, ", `, and \ (but not the !).

Escape Strategies

In general, single quotes are most useful and easiest to use. However, you cannot use single quotes when the filename contains a single quote.

Double quotes still require some characters to be escaped with the backslash and cannot deal with an exclamation mark !.

Backslash escaping works in nearly all cases, but can be tricky to type right and is quite illegible.

name (in Finder) Backslash Escape single Quotes Double Quotes
My Great Folder My\ Great\ Folder 'My Great Folder' "My Great Folder"
“New” Files \"New\"\ Files '"New" Folder' "\"New\" Folder"
‘Old’ Stuff \'Old\'\ Stuff cannot escape ' "'Old' Stuff"
Important! Important\! 'Important!' cannot escape !
Bump \m/ Bump \\m\: 'Bump \m:' "Bump \\m:"
Do@Home Do\@Home 'Do@Home' "Do@Home"
Yes/No/Maybe Yes\:No\:Maybe 'Yes:No:Maybe' "Yes:No:Maybe"
Project (Old & New) Project\ \(Old\ \&\ New\) 'Project (Old & New)' "Project (Old & New)"
Profit$$$ Profit\$\$\$ 'Profit$$$' "Profit\$\$\$"

Quoting and Tab Completion

When typing paths, always use tab completion to be safe. Tab completion uses backslash escaping by default.

$ cd Proj⇥
$ cd Project\ \(Old\ \&\ New\)/

However, when you start a quoted path, tab completion will complete in quoted form.

$ cd 'Pro⇥
$ cd 'Projects (Old & New)'/

Tab completion is even smart enough to change the approach when the strategy you chose (i.e. double quotes) cannot work:

$ cd "Imp⇥
$ cd Important\!/

Quoting and Home Path

Since you generally use quoting to avoid bash changing characters, you cannot use the ~ to get a short cut to your home directory in quotes.

However, you can leave the ~ outside of the quotes and get the best of both worlds:

$ cd ~/'Project (Old & New)'

When you use double quotes, you can also use the $HOME environment variable:

$ cd "$HOME/Project (Old & New)"

Next: Commands