Parsing dscl Output in Scripts

On macOS dscl is a very useful to access data in the local user directory or another directory the Mac is bound to. For example you can read a user’s UID with:

$ dscl /Search read /Users/armin UniqueID
UniqueID: 501

This output looks easy enough to parse, you can just use cut or awk:

$ dscl /Search read /Users/armin UniqueID | cut -d ' ' -f 2
501
$ dscl /Search read /Users/armin UniqueID | awk '{print $2;}'
501

However, dscl is a treacherous. Its output format changes, depending on the contents of an attribute. When an attribute value contains whitespace, the format of the output has two lines:

$ dscl /Search read /Users/armin RealName
RealName:
 Armin Briegel

With attributes like the UID, it is fairly safe safe to assume that there will be no whitespace in the value. With other attributes, such as RealName or NFSHomeDirectory, you cannot make that prediction with certainty. Real names may or may not have been entered with a space. A user (or management script) may have changed their home directory to something starting with /Volumes/User HD/... and your script may fail.

To remove this output ambiguity, dscl has a -plist option which will print the output as a property list:

 $ dscl -plist . read /Users/armin RealName
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>dsAttrTypeStandard:RealName</key>
    <array>
        <string>Armin Briegel</string>
    </array>
</dict>
</plist>

The resulting property list is a dict containing a key with the native attribute name and an array containing the values, even when there is only one value.

Having a property list is nice, but parsing property lists in a shell script is challenging. I have found two solutions

Xpath

You can use the xpath tool extract data from the XML output:

$ dscl -plist . read /Users/armin RealName | xpath "//string[1]/text()" 2>/dev/null
Armin Briegel

Note that the xpath output does not include a final new line character, which makes it look a bit strange.

The xpath argument in detail means:

  • //string[1]: the first of any string element
  • /text() the text contents of that stringobject

This syntax makes a lot of assumptions about the property list input. I believe they are safe with the dscl output. (Please test)

If you want to play around with xpath syntax, I recommend using an interactive tool. I used this one from Code Beautify which worked well enough, but frankly I just randomly chose one from the list of search results for ‘xpath tester’. (If you can recommend a great one, let us know in the comments.)

PlistBuddy

As I said, the xpath solution makes a lot of assumptions about the layout of the property list. A safer way of parsing property lists would be a dedicated tool, such as PlistBuddy. However, PlistBuddy does not read from stdin. At least not voluntarily.

A few weeks ago Erik Berglund shared this trick on Mac Admins Slack which makes PlistBuddy read the output from another command. We can adapt this for our use case:

$ /usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/armin RealName)
Armin Briegel

Note that you have to escape the : in the attribute name, since PlistBuddy uses the colon as a path separator.

You can use this in scripts to assign the value to a variable with

realName=$(/usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/$username RealName))

This uses nested command substitution with the $(... $(...) ...) syntax which is not possible using backticks.

Either way, you can get a safe value from dscl in shell script, whether it contains whitespace or not.

Prefs Tool

Preferences or defaults on macOS seem easy, but their subtleties can grow complex very quickly.

The main reason for confusion is that preferences can be stored in many places and on many levels. The defaults system composites all of the keys and values from all locations to a process or application.

In my book “Property Lists, Preferences and Profiles for Apple Administrators”, I list 17 possible domains or levels where preferences can be stored and read from. The most common domains are:

Domain Location
User/Application ~/Library/Preferences/identifier.plist
User/Application/Computer ~/Library/Preferences/ByHost/identifier.xyz.plist
Computer/Application /Library/Preferences/identifier.plist
Configuration Profile n/a

To add to this confusion, Apple’s documentation keeps mixing up terms like ‘domain’ and ‘identifier’. I use the term ‘domain’ to designate the level or location a setting is stored in, and ‘identifier’ for the name of the preference (i.e. com.apple.screensaver).

The defaults command, which is the proper tool to interact with preferences files, does not properly work with different levels or domains. When you run

$ defaults read com.apple.screensaver

The output will be from the User/Application domain only, i.e. the data stored in the file ~/Library/Preferences/com.apple.screensaver.plist.

But the ScreenSaver process stores more data in the ByHost domain. You can read this domain or location with defaults as well:

$ defaults -currentHost read com.apple.screensaver

However, you must remember to check and read the ByHost domain as well as the standard domain. To access the computer level domain you have to use

$ defaults read /Library/Preferences/com.apple.screensaver

(The ScreenSaver process does not use this domain, so you will get an error saying that it does not exist. However, you won’t know this domain is empty until you try.)

Defaults cannot tell you when a setting is set or overridden by a configuration profile, or what its value is in that case. You cannot get the full composited view of defaults with the defaults command.

Greg Neagle wrote a short python script a while back which could give you the effective result for an identifier and a specific key. His script will also show where the value is coming from.

I have found Greg’s script to be very useful, but I wanted it to do a bit more. My version, Prefs Tool, can now show you all keys set for a specific application identifier, including those managed by configuration profiles.

$ ./prefs.py com.apple.screensaver
idleTime <int>: 0L (User/ByHost)
CleanExit <string>: u'YES' (User/ByHost)
askForPassword <bool>: True (Managed)
askForPasswordDelay <int>: 0L (Managed)
moduleDict <dict>: {
    moduleName = iLifeSlideshows;
    path = "/System/Library/Frameworks/ScreenSaver.framework/Resources/iLifeSlideshows.saver";
    type = 0;
} (User/ByHost)
showClock <bool>: True (User/ByHost)
PayloadUUID <string>: u'AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE' (Managed)
tokenRemovalAction <int>: 0L (User/ByHost)
PrefsVersion <int>: 100L (User/ByHost)

The script has a few more tricks up its sleeve. There is also still lots of work to be done. See the Github repository and its ReadMe for details.

MacAdmins Slack – a highly opinionated guide

I love MacAdmins Slack. I am logged in nearly every day. I use it for research, solving problems, camaraderie, and just plain fun. The community there is wonderful.

There are a few things particular to this Slack and some other online forums in general that I noticed, so I thought I’d like to write this guide.

This is, as the title says, highly opionionated and from my personal perspective. I do hope it is useful for everyone.

What is Slack?

Slack is a popular message board application. It’s a cross between between a bulletin board system and a chat room.

Slack has a web interface and clients for most operating systems.

The MacAdmins Slack is a particular instance which specializes on topics relevant to Apple Administrators. You can sign up here.

The Lingo

There are a few terms particular with Slack, that might be confusing at first.

An organization can set up a Slack “Workspace.” You can be a member of and logged into multiple workspaces. You will have a different login and username for each workspace.

Slack help has a general glossary, which is helpful.

Channels

Within a workspace, Slack is separated into “channels.” Channels can be public or private. When you sign in for the first time, you are be added to some public channels by default. You can click on the “Channels” header in the sidebar to browse and search existing channels.

When you are typing and start a word with the # character Slack will treat this as a link to a channel. When you start typing the channel name Slack will suggest auto-completions. If a channel with the name exists, the word will be linked so that users can click to go to that channel.

Frustratingly, Slack’s autocompletion (for channels and users) uses the return key to confirm a selection and the tab key to jump to the next suggestion in the list. This is not how those keys are usually used on macOS and throws me off every time.

Public channels in MacAdmins Slack are either on a particular technology or software (#highsierra or #munki), regions or countries (#thenetherlands, #uk or #anzmac), events (#psumac or #wwdc) or pretty much everything else.

The language on MacAdmins is usually English, though regional channels are often held in that region’s language. Be aware that English is not every user’s main language. While this can make communication frustrating on both sides, be polite, patient and friendly about it.

To be honest, there are way #toomanychannels. The reason for this is that anyone can create a channel. Before you create a new channel, you should browse and search and maybe ask if there is already a channel for that particular purpose or topic. #general is usually a good place to ask if you can’t find something obvious in the channel browser.

In addition there are private channels, which work basically the same, but cannot be searched and only joined on invitation.

Special Channels

There are a few channels that have special roles or uses:

#general: is the “anything” channel, as long as the topic is somewhat MacAdmin related. Questions asked here may are often answered directly or you will be referred to a different channel.

#backroom: This is for ‘off-topic’ discussions. Any topic goes, as long as you follow the CoC.

#ask-about-this-slack: for technical and organizational questions to the admins about the MacAdmins Slack.

Slack is not Email

The MacAdmins Slack can get very busy. You may have the urge to keep up with every message in every channel you follow. This may be possible when you are in just a few channels. However, I have gotten used to just hitting ‘shift-escape’ (Mark all as read) in the morning and maybe again in the afternoon. I try to keep up with discussions and threads I am part of, and have learnt to be ok with missing most others.

Emojis

Emojis are an important part of Slack and there are a few ways of using them.

You can just insert an Emoji when typing with the standard macOS or iOS emoji picker. You can also type an emoji name or ‘code’ between colon characters:. So :grin: will turn into the grinning smiley. This is usually more convenient than the system pickers.

When you see an emoji, you can hover the mouse over it to learn its name or code.

Reactions

You can add an emoji to a post with the reaction button. (the smiley with the + symbol). Then the emoji will be shown attached to the post, multiple reactions by different users will be shown next to each other, and they will be counted up.

When you hover your mouse over a reaction, it will show which users added that particular reaction.

Special Emojis

Some Emojis are unique to Slack or have special meaning

:+1: will show as the ‘thumbs up’ emoji. This is commonly used to show approval or support, though some users prefer :plus1 or :heavy_plus_sign:

:protip: is used to highlight a great tip. There is a bot that gathers all post with this reaction in the #protips channel

:this: will show as an animated chevron. Used to approve or emphasize a post.

:raccoon: is used to politely notify that an ongoing discussion might be better suited for another channel (Why is is a raccoon?)

:dolphin: is sometimes used when you leave a channel to state that you are merely leaving to prune your channel list and not because something has upset you

Custom Emojis

You can create your own emojis. Or add new names for existing ones.

Create custom emoji – Slack Help Center

To Thread or not to Thread

You can reply to a post directly in a channel’s timeline or create a ‘thread’ where the replies are collapsed or sorted with the original post. Use the speech bubble icon to create a thread.

In MacAdmins Slack, most users prefer replies in the timeline, however, when you are replying to a post further up in the timeline, then threads can be quite useful. When replying in a thread, you have the option to show the reply in the channel’s main timeline as well.

Use the ‘@’ Wisely

You can ‘mention’ another user with the @ symbol and their username. With Slack’s default setting the user will get notified of a mention. When you use @scriptingosx in a post, it will notify me, even when I am not in the channel.

This can be very useful to ‘summon’ someone into a channel, because they might be interested or able to contribute to a discussion. I use it when I reply to questions or requests that happened a while ago, so that the person gets notified that there is a reply.

Other than that, you should use mentions with care. Remember that you may be ringing all of someone’s devices with it.

Set up Do not Disturb

To avoid excessive notifications, you can set Slack to ‘Do not Disturb’ mode by clicking on the bell icon next to the Workspace name. You can snooze the Slack for a certain and setup a recurring schedule to mute notifications overnight.

A user who mentions you while have the ‘Do not Disturb’ mode enabled will be informed why you may not be reacting.

You can also see a user’s local time in their profile. This might give you an idea of when they might be online or not. You can get to a user’s profile by clicking on their icon.

Manage your Notifications

Aside from the ‘Do not Disturb’ feature you can further manage the notifications Slack can send to you.

In addition to be notified when you are mentioned (@ed) you can add certain keywords that may be interesting to you. (e.g., I have keywords for my books’ titles)

Formatting Posts

You can use a simplified MarkDown-like syntax to format your posts. Enclosing a word or sentence in underscores _ will turn it italic, asterisks * will turn it bold.

If you have trouble remembering the syntax, you can also see the most common formatting options in small text under the message entry field.

Posting Code

Since MacAdmins Slack is a technical forum, posting commands or pieces of code will be fairly common. When you enclose a sequence of words with single backticks it will be shown in monospace font, which others will usually understand to be a command.

When you use triple backticks, Slack will interpret the text in between as a code block. Other special characters and white space (multiple space, tabs, new lines) will be shown as is. This is useful to share short code blocks or log sections.

To share full scripts or longer log files, you should use Slack Snippets. You can create a snippet with the big ‘+’ button next to the text entry or by just dragging a script or text file into the slack window.

Asking questions

We all use Slack to ask for help when we are stuck. The willingness to help each other out it one of the strengths. However, when you do have to ask for help, there are a few common courtesies you should follow. (These hold true for any request for help, like a support incident.)

Be Descriptive and Specific

Don’t just say “Help, XYZ is broken!” Don’t ask if “anyone knows ABC?”

Explain what you are trying to do, in which context. Show what you already tried to fix the problem. (you did try to solve it yourself first, didn’t you?)

I find, that often the act of formulating the question properly helps me figure out the solution myself, or at least get closer to a solution.

People who want to help you will follow-up with those questions, but will be more likely to help when the request is well formulated and has (most of) the necessary context.

Examples:

Bad:

My postinstall script does not work! Can anybody help?

Better:

I want to show a dialog from a postinstall script which prompts the user for the computer name. I am using osascript, but it is failing and I don’t understand why?

Even better: add the script (or a part of the script) and errors you are seeing

Keep your question relevant

Sometimes a question might just drown in another ongoing conversation. Sometimes, expecially on the less busy channels, no-one will be around to answer. Be patient before you start cross-posting to other channels.

It’s ok to repeat your question, once the ongoing discussion subsides, but don’t spam. Maybe it’s just that no-one really has an answer.

Keep in mind that everyone on the MacAdmins Slack has a job, which is not answering your questions on Slack. Helping each other out on Slack is something we all do on the side, voluntarily.

Don’t @ or DM people just because they have helped you before, unless you want to follow-up on something very specific.

MacAdmins Slack is busiest during North American office hours. Keep that in mind when posting questions as well. (There are a few admins from elsewhere in the world who will help out when they can.)

What do you want to accomplish?

Even when you ask questions properly and with detail, you may ge the the counterquestion: “What is it you actually want to accomplish?” This has turned into a sort of a meme on the MacAdmins Slack.

When you get this question, someone believes that you may be narrowing down on a dead end and a completely different approach may be more appropriate. They want to get your ‘big picture’ to understand the context.

This is the time to step back, explain your goals and let the MacAdmins community help you gain some new perspective. Don’t double down on what you are trying to do. This question has lead to some of the most interesting discussions.

Join the Slack and Enjoy!

Overall I feel the MacAdmins Slack is a great place to share and receive knowledge for MacAdmins. I you still haven’t signed up, go and do it here!

If you already are a member, I hope you learnt something useful here. If you think I missed something important, then let me know! (My user name on the MacAdmins Slack is @scriptingosx.)

BBEdit at 25 and why I still use it

BBEdit turned 25 last week! Belated Happy Birthday!

I am not shy that BBEdit is my favorite text editor. I don’t remember when exactly I got my first copy, but it must have soon after it came out. I have been using it one form or another since then.

It is usually the first application I install on a new Mac. I don’t bother to add the icon to my dock because it is open all the time. (and it doesn’t hog memory)

Recently, someone asked me why I prefer BBEdit. I was caught a bit off-guard but gave my honest answer: habit.

However, this answer doesn’t really do BBEdit much justice. Now I realize it is a habit formed over two-and-a-half decades. When BBEdit came out I was a student and used it for LaTeX documents, shell scripts and Fortran code on my PowerBook 160.

Back then Mac text files preferred a different line break character (carriage return: \r) than Unix (line feed: \n) and Windows (line feed and carriage return: \r\n). BBEdit was able to read, convert and write all of them.

BBEdit transitioned to PowerPC. It was one of the first applications to have a native Mac OS X version (the Mac/Unix text file compatibility was really valuable then). BBEdit made the Intel transition and is ready for the 64-bit only future of macOS.

All along the way, BBEdit has always stayed true to the Mac environment, however the Mac changed. When you expect native ‘Mac-like’ behavior that’s how BBEdit behaves. It supports AppleScript, but also has powerful command line tools. It will parse a binary property list to XML and it will ask for authorization to read and edit root-owned files.

But, BBEdit also has always been a great way to talk with things outside of the Mac.

When I have to edit files over ssh I will grudgingly use nano or (in desperate situations) vi, but then I remember that BBEdit has direct editing over sftp.

BBEdit has supported three version control systems I have used over the years and (so far) survived two.

I have built web pages in BBEdit, trawled through truly gigantic log files, written code in secveral languages, and documents in LaTeX, Markdown and various other file formats. Some of the files were even just plain text.

If you have read anything I have written on this website, or my books, it very likely started out as a draft in BBEdit.

Right now I am writing a book with the experimental Markua language on LeanPub. In BBEdit, of course.

The advantage of text files is that they can be opened everywhere. Trying out a new text editor has virtually no barrier. Over the years I have used and tried out uncountable different editors and many of them had some features I liked. Right now I have Atom and Sublime Text on my Mac, as well. Those are great tools with some wonderful features. However, they are not native to the Mac and that often adds friction to the workflow.

Now that I have thought about it, habit is only part of the answer. It is not just habit, it is a trust they have built and earned over more than two decades.

When Apple does their next transition, I can expect BBEdit to be right there with me, still not sucking. As long as there is still a need to edit text files on Macintosh/Mac OS/Mac OS X/OS X/macOS or whatever it is going to be called, I am looking forward to the next 25 years of working with BBEdit.

macOS Installation: Strange New World

This is a section out of my next book: “macOS Installation for Apple Administrators”

Get it on iBooks

High Sierra came with many changes for Mac administrators. Some of them were anticipated, some of them weren’t.

When Apple announced APFS would be deployed to every OS by 2018, Mac Admins thought that would be the main challenge and the reason for sweeping changes in the structure and functionality of macOS.

At the same time it was obvious that Apple liked the way MDM was in the center of all management for iOS and wanted to have it play a similar role for macOS. The extent how far Apple would push it was unclear.

While APFS has certainly come with some challenges, mainly for managed FileVault users, it was not the “Macnarök” that we were expecting.

I am not including these links to make fun of the authors – quite the opposite. At the time there was justified anxiety about the future of macOS adminstration and these posts (and many others) put this into words. I believe that because of MacAdmins blogging and presenting about their concerns and anxiety, talking with Apple Reps and filing bugs, as a community, were able to influence the course Apple is taking. It is definitely a slow, frustrating and opaque process, but not entirely fruitless.

However, other new “features” of High Sierra such as UAKEL and, later with 10.13.4, UAMDM provide their own set of hurdles administrators have to clear.

These are the main challenges for macOS deployment:

  • Imaging is Dead, Long Live the Installer
  • MDM is Essential
  • Secure Boot

Let’s look at these challenges and some solutions to them in detail.

Imaging is Dead, Long Live the Installer

Imaging has been the means to quickly and reliably install macOS to clients. It could scale from an external drive for a few clients to a NetBoot infrastructure for large deployments.

Apple has come out explicitly and said that imaging in most forms is not recommended or supported any more.

In the early days imaging meant building a ‘Golden Master’ image with all configuration and software ‘baked in,’ which then would be deployed to all the Macs. Creating this ‘Golden Master’ image was a lot of work and the result quite fragile and error-prone. Macs would usually be kept on a specific version of ‘the image’ for months or a year without updates.

Since the process of manually building the Golden Master was tedious and error-prone, administrators started to automate the process. Tool chains of packages, scripts and other tools could build the master image (or most of it) automatedly.

InstaDMG was one of the first tools to do this. AutoDMG is a direct successor of this approach and still useful for many tasks today.

The same packages and scripts that automated the configuration of the master image could be used to install and update software on ‘live’ Macs. Management tools, like Munki, Casper (now Jamf Pro) and Filewave, can deploy updates to clients automatically without user interaction. As work was moved from preparing the master image to maintaining the live system, the master image became thinner and thinner.

This resulted in the ‘thin imaging’ approach, where the image only had the minimal ‘base OS’ plus enough configuration to connect it to the management system. Once the Mac booted the management system takes over. Imaging would only be used to wipe a Mac and lay down a ‘base image.’

While this approach is slower than monolithic imaging, it is more flexible and allows to keep the system and software on the client continually updated, without having to re-image.

Some workflows even skipped the imaging entirely. When you get a Mac out of the box it already has macOS installed, all you need is to configure it to connect to your management system, which will take care of the rest.

Some administrators had already moved to ‘thin’ imaging or ‘no-imaging’ workflows before High Sierra and pronouncements that ‘Imaging is Dead!’ have been made years ago. Administrators using thin imaging or no-imaging workflows will find they are better prepared for the High Sierra installation workflows.

Mac administrators now have to build new workflows based on the Apple’s supported installation methods for High Sierra. These all rely on the macOS Installer application in one form or another.

The supported means of installing macOS High Sierra are:

  • the macOS Installer application
  • a bootable installer on an external drive
  • the macOS Recovery System
  • a NetInstall image built with Apple’s System Image Utility

Using the macOS installer application to upgrade a system from 10.12 is straightforward enough and similar to what was possible with the startosinstall command since OS X El Capitan 10.11.

The main challenge are ‘nuke-n pave’ workflows, where the system on a Mac is erased and overwritten with a new system.

This workflow is easy with traditional imaging. In fact, it was easier and faster to just image over a system, rather than upgrade.

The startosinstall --eraseinstall option added in 10.13.4 makes an installer based ‘nuke’n pave’ workflow possible with the macOS Installer application.

In the new High Sierra world of Mac deployment, we face the same challenge: how do you connect an ‘out-of-the-box’ macOS to your management system.

Apple’s answer to that is clearly DEP.

MDM is Essential

DEP will tell a freshly installed Mac which MDM server is ‘responsible’ for it. The MDM server can then use the InstallApplication MDM command to install and configure a management agent, which together with the mdmclient performs the remaining configuration and software installation.

Up until 10.13 Mac administrators could ignore and not use an MDM for Mac deployment and management. There were a few useful features, but there are also many tasks an MDM cannot do or that a local agent can do better.

With High Sierra MDM became essential to manage and support UAKEL. In the early versions of High Sierra, merely being connected to an MDM would disable UAKEL, starting with 10.13.4 UAKEL can be managed with a configuration profile which has to be pushed from a user approved MDM.

Prepare for changes to kernel extensions in macOS High Sierra – Apple Support

Like supervison for iOS devices, UAMDM adds an extra level of approval and provides an additional level of control over the macOS client device. UAMDM will be required for more and more essential functionality going forward.

However, as the name implies, a ‘user approved’ MDM needs to be interactively approved by a user at some time during deployment which creates an enormous challenge for automated workflows.

Apple claims that DEP enrolled devices are automatically user-approved. However, even the DEP enrollment still requires user interaction in the Setup Assistant.

Also DEP is not available in all regions and it may not be possible for organizations to add all existing Macs to their DEP account. DEP also requires Apple DEP servers to be accessible. So during this transition phase and for some other situations, Mac administrators have to have a non-DEP workflow solution to get a UAMDM enrolled device.

Secure Boot

Finally the iMac Pro introduces a new system controller with secure boot. By default all external means of booting an iMac Pro are disabled.

NetBoot/NetInstall is not supported on the iMac Pro at all. You can enable booting off external drives, but the process to do so is quite convoluted and cannot be automated. So for administrators, the option is fairly useless.

With the startosinstall command administrators can achieve the necessary management steps from within the local system. You can either upgrade the existing system or ‘nuke’n pave’ with the eraseinstall option.

However, the eraseinstall option is limited to systems that are already on 10.13 with an APFS file system. On the iMac Pro you can safely make that assumption, but for ‘older’ Macs and systems you usually cannot.

So administrators need a second workflow for systems and hardware that cannot run the eraseinstall option (yet).

The Recovery system, external installer volume or NetInstall are the Apple Supported means which can ‘nuke’n pave’ a Mac. NetInstall is the only means that can be automated.

However, there are other solutions such as Imagr or a modified DeployStudio workflow which can achieve the same goal using the macOS Installer application and startosinstall.

The New macOS Deployment Workflow

Deployment strategies and workflows are as varied and individual as the organizations that use them. But with the challenges listed above, I believe we can state a few pieces that are necessary for a new deployment workflow for High Sierra and hopefully going forward.

Enrolling a ‘new’ Mac

First, any deployment workflow should be designed to start with a clean ‘out-of-box’ macOS. This way the same workflow can be applied to new Macs and to Macs that were just re-installed.

At the end of the installation workflow the Mac is enrolled and user-approved with your management system, so that it can take over and perform the remaining configuration and installation. All configuration steps should be controlled from the management system and be executed independently of which deployment workflow was used.

Deployment Workflows from an out-of-box Mac

DEP is the Apple-preferred means of connecting a new device to your MDM. Even when you require a non-DEP workflow for older Macs you should implement a DEP workflow. It is likely that Apple may make DEP enrollment mandatory in the future.

However, right now in the transition phase, you will need a second workflow for machines that cannot be registered with DEP.

For the non-DEP workflow, you can choose between manual enrollment, which happens after the user has booted and logged in for first time, or some sort of ‘side-loading’ where you add the MDM and management system configuration to a system, before it even boots for the first time.

For sideloading you can boot to Recovery and install software on the system volume. (Bootstrappr)

On Macs that still support NetBoot, you can also use tools like NetInstall, Imagr or DeployStudio. Either way you should do this before first boot of the system.

UAMDM requires user interaction to approve the management in some form or another. This is a change from previous deployment workflows which could be ‘fully automated.’

You will have to plan for this user interaction. In some cases, such as individual-use devices, the approval of the management will be part of standard first setup and will be done by the end user. In other situations such as classroom setups, it will be harder to make this part of the workflow.

Some management systems let you scope software installations and configurations depending on criteria reported from the device. In this case you can set up your workflow so that installations that rely on UAMDM (third party kernel extensions) are deferred until the MDM is approved.

Either way, the end-result of each of these paths should be a Mac that is enrolled (user-approved) with your MDM where the management system is installing, configuring, maintaining the system and software.

Restoring a Mac to ‘new’

You will also need a second set of workflows that bring a Mac from any state (even a broken one) back to the ‘out-of-box’ state so that you can start over (or retire it).

Restoring a Mac to new

The first option uses the startosinstall --eraseinstall command. This is easiest to automate and works on new hardware like the iMac Pro. However, it will not work on Macs without APFS systems and older versions of macOS.

The Macs that cannot use the eraseinstall option, however, can use NetInstall or a similar NetBoot based tool. You can use the macOS Install application and startosinstall with Imagr and DeployStudio, so this combination will run the installer and can make sure the firmware is up to date as well.

Note that the workflows and tools for restoring a Mac are also used to ‘sideload’ the agent configuration on to a Mac. You can add the MDM/agent configuration to the restore workflow to save some steps here. However, since NetBoot based workflows are transition solutions to support ‘old’ Macs and systems, your deployment workflows need to work with ‘out-of-box’ systems as well as pre-configured systems.

In the transition phase until all hardware supports eraseinstall you will need both workflows available.

You should also have documentation, training, and support articles to instruct administrators, techs, and users to restore a Mac to ‘out-of-box’ state with (Internet) Recovery or an external boot drive. While this should be an option of last resort, it will be necessary in situations where a Mac cannot boot otherwise.

Obviously, these are very high level views of the workflow and the devil is very much in the details of the implementation. Even a book can only skim the surface of any implementation. Nevertheless, once you understand and accept this framework, the necessary detailed steps become obvious, though not necessarily easy.

Get it on iBooks

Many thanks and credit need to go to Anthony Reimer, whose excellent presentation at MacAD.UK 2018 helped me sort through my own thoughts on this topic.

PLIST Editor

I recently stumbled over a Property List Editor on the Mac AppStore that I had not seen before.

PLIST Editor

My favorite graphical property list editor so far is PlistEdit Pro from FatCat Software. True to its name, PlistEdit Pro has a few nice “pro” features that PLIST Editor lacks: Preference Browsing, Browser windows, AppleScript, Plist Structure definitions

However, PLIST Editor comes in at a much lower price point (US$3.99 on the Mac AppStore). Its feature set covers all the necessities. It can open property list files with file extensions other than .plist (for those pkginfo or recipe files you want to edit) and supports drag and drop and undo and even the macOS versioning system.

PLIST Editor can open and save XML and binary property lists and can open legacy ASCII/Openstep property lists. To convert from binary to XML or vice versa you have to duplicate a file.

It does not have a command line tool to quickly open a property list file from Terminal. This is a limitation imposed by being on the AppStore. However, you can use the open command:

$ open -a 'PLIST Editor' sample.plist
You can add an alias to your shell profile to simplify this: `alias pacifist='open -a "PLIST Editor"'`

Overall it seems like a useful tool that serves its purpose well. It is being actively updated. So, go and check it out!

Installing and Using Command Line Tools

There are many command line tools and scripts you can download and install that are very useful for Mac Admins.

(Just a representative list, certainly not complete.)

Some of these tools provide installer packages that deploy the tool in the proper directory – usually /usr/local/bin so that you and other users can use it. (/usr/local/bin is in the macOS default PATH.)

However, many of these tools, such as munkipkg or my own quickpkg just come as a git project folder, with no or few instructions to get it set up. The assumption is, that when you use these tools you are familiar enough with the shell to make them work.

There are actually several approaches to getting these tools to actually work for you, most with different upsides and downsides. This post will talk about a few of them.

Getting the Tool

Before you can choose a method to run the tool, you need to get it. Many admins share their scripts and tools through a hosted service like Github. My quickpkg tool, for example, is a python script hosted as an open Github repository.When you follow that link you will see the main project page. The page has a menu area up top, a file list in the middle and below an area where an introduction to the project (the ReadMe file) is shown. It is worth to read the ReadMe in case they have special installation instructions.

Download the Release

Git is a version management tool that lets you track changes throughout the coding process. Github is one popular service to host these projects online. Contributors of a project have the option of marking steps of the project as a ‘release.’ Releases are considered a tested and stable stop in between less reliable developmental steps.

Releases will be shown in the project’s ‘releases’ page (link in the middle of the page, above the file list). (quickpkg releases page

On the releases page you will see a list of releases with the newest on top. At the very least each release will have a snapshot of the project’s code as a zip or tar.gz archive. Some projects provide other archives or installers such as dmg or pkg as well.

Download the Current Project

Some projects do not manage releases. (You will see ‘0 releases’ in the tool bar.) Then you can still download the most recent version of the project. There is a large green ‘Clone or download’ button on the right area above the project’s file list for this. When you click that button it will expand to show some more options.

‘Download ZIP’ will simply download an archive of the current state of project, much like the release download would.

When you download the archives, either through the releases page or from the ‘Download ZIP’ button, the resulting project folder will not be connected with the Github project any more. If you just want to use the current version, then that is fine and will serve you well. If you want an updated version in the future you will simply download the newer version and replace the tool you already have.

If you rather use git to download and manage the code, then you can do that here, too. However, that is a topic for another post.

Using the Tool

However you get the project you will now have a directory with the tool and any supporting files. You can already change directory to this folder in Terminal (drag the folder on to the Terminal icon to open a new Terminal already changed to it) and run the tool directly:

$ cd ~/Projects/quickpkg/
$ ./quickpkg
usage: quickpkg [-h] [--scripts SCRIPTS] [--preinstall PREINSTALL]
                [--postinstall POSTINSTALL]
                [--ownership {recommended,preserve,preserve-other}]
                [--output OUTPUT] [--clean] [--no-clean] [--relocatable]
                [--no-relocatable] [--sign SIGN] [--keychain KEYCHAIN]
                [--cert CERT] [-v] [--version]
                item_path
quickpkg: error: too few arguments
This will do for tools that you use rarely. But for tools that you want to use frequently typing the path to the tool is quite cumbersome.

Put it in the PATH

The PATH environment variable lists the directories where the shell looks for commands. You could add the project directory of the tool you just added to the PATH, but that would be tedious to manage.

An easier solution is to copy the tool to /usr/local/bin. This is the designated directory for custom commands. /usr/local/bin is also in the default macOS PATH.

However, copying the tool has some downsides. When the tool get’s updated you will have to copy the newer version, as well. Also some tools may require additional resources or libraries that reside in its project directory.

Instead of moving the tool, you can create a symbolic link to the tool in /usr/local/bin.

I keep the project folders of tools in ~/Projects so I use the command:

$ sudo ln -s ~/Projects/quickpkg/quickpkg /usr/local/bin
$ ls -al /usr/local/bin/quickpkg 
lrwxr-xr-x  1 root  wheel /usr/local/bin/quickpkg -> /Users/armin/Projects/quickpkg/quickpkg
Since symbolic links use paths, this has the advantage that when you download a newer version of the project to the same location, the link will point to the new version.

Putting links to a tool in /usr/local/bin has a few downsides (or upsides, depending on your perspective):

  • you need to have administrator privileges to change /usr/local/binlinks/tools you add to /usr/local/bin affect all users on that Mac

Set your own PATH

When you want to have the tools only affect your shell environment you need to do a bit more work.

First you need to choose a location where you tools or links should live. I have created a directory ~/bin for that purpose.

$ mkdir ~/bin
When you don’t want anyone else on the Mac to see what you are doing in that directory, you can remove everyone else’s access to it:
$ chmod 700 ~/bin
If you want you can also hide the directory in the Finder:
$ chflags hidden ~/bin
(Use the same command with nohidden to make Finder show it again.)

(To test the following properly, you need to delete the symbolic link we created earlier in /usr/local/bin. If that still exists the shell will use that, since it comes earlier in the PATH.)

You can create a symbolic link to the in ~/bin with

$ ln -s ~/Projects/quickpkg/quickpkg ~/bin
However, these still will not work, since we need to add the ~/bin directory to your personal PATH.

To do that you need to add this line to your ~/.bash_profile or ~/.bashrc:

export PATH=$PATH:~/bin
(Read more about how to create a bash profile here and here. This assumes you are using bash, the default on macOS. Other shells will have other locations where you can change environment variables.)

Then open a new Terminal window or type source ~/.bash_profile so that the new profile is loaded in the current window and try running the command.

Success!

Single Brackets vs Double Brackets

In my recent post I mentioned in passing, that you should be using double brackets [[…]] for tests in bash instead of single brackets.

This is the post where I explain why. I also talked about this briefly in my MacSysAdmin session: Scripting Bash

Double Brackets are a bashism

Double brackets were originally introduced in ksh and later adopted by bash and other shells. To use double brackets your shebang should be #!/bin/bash not #!/bin/sh.

Since sh on macOS is bash pretending to be sh, double brackets will still work with the wrong shebang, but then your script might break on other platforms where a different shell might be pretending to be sh. Consistent behavior across platforms is the main point why sh is still around, so don’t use double brackets in sh (or use bash to use double brackets).

I go into detail on why to use bash over sh in this post: On the Shebang

Side note on syntax

In shell scripts you usually use tests in if or while clauses. These are tedious to write in the interactive shell. The ‘and’ operator && will execute the following statement only if the preceding statement returns 0 (success). So you can use && to write simple if … then … clauses in a single line.

if [ -d Documents ]
then
    echo "found docs"
fi

and

[ -d Documents ] && echo "found docs"

have the same effect. The second is much shorter, but as soon as the test or the command gets more complex you should revert to the longer syntax.

Alternatively, the ‘or’ operator || will only execute the following statement when the previous statement returns non-zero or fails:

[ -d Documents ] || echo "no docs"

is the same as

if [ ! -d Documents ]
then
    echo "no docs"
fi

What’s wrong with the single brackets?

The single bracket [ is actually a command. It has the same functionality as the test command, except that the last argument needs to be the closing square bracket ]

$ [ -d Documents && echo "found docs"
-bash: [: missing `]'
~ $ [ -d Documents ] && echo "found docs"
found docs
$ test -d Documents  && echo "found docs"
found docs

Note: in bash on macoS both test and [ are built-in commands, but as usual for built-in commands there are also executables /bin/test and /bin/[.

A single bracket test will fail when one of its arguments is empty and gets substituted to nothing:

$ a="abc"
$ b="xyz"
$ [ $a = $b ] || echo "unequal"
unequal
$ unset a
$ [ $a = $b ] || echo "unequal"
-bash: [: =: unary operator expected
unequal

You can prevent this error by quoting the variables (always a prudent solution).

$ [ "$a" = "$b" ] || echo "unequal"
unequal

Double brackets in bash are not a command but a part of the language syntax. This means they can react more tolerantly to ‘disappearing’ arguments:

$ [[ $a = $b ]] || echo "unequal"
unequal

You will also get an error if one of the arguments is substituted with a value with whitespace with single brackets, while double brackets can deal with this.

$ a="a"
$ b="a space"
$ [ $a = $b ] || echo "unequal"
-bash: [: too many arguments
unequal
$ [[ $a = $b ]] || echo "unequal"
unequal

Note: the = operator in sh and bash is for string comparison. To compare numerical values you need to use the -eq (equals), -ne (not equals), -gt (greater than), -ge (greater than or equal), -lt (less than), -le (less than or equal) operators. With double brackets you can also use two equals characters == for a more C like syntax. (or, better, use ((…)) syntax for arithmetic expressions)

Also, when using the = to assign variables, you cannot have spaces before and after the =, while the spaces are required for the comparison operator (both with single and double brackets):

a="a"           # no spaces
b="b"           # no spaces
[ "$a" = "$b" ] # spaces!
[[ $a = $b ]]   # spaces!

Since the single bracket is a command, many characters it uses for its arguments need to be escaped to work properly:

$ [ ( "$a" = "$b" ) -o ( "$a" = "$c" ) ]
-bash: syntax error near unexpected token `"$a"'
$ [ \( "$a" = "$b" \) -o \( "$a" = "$c" \) ]

You could alternatively split this example into two tests: [ "$a" = "$b" ] || [ "$a" = "$c" ].

Double brackets interpret these characters properly. You can also use the (again more C like) && and || operators instead of -a and -o.

 [[ ( $a = $b ) || ( $a = $c ) ]]

In general, you can work around most of the issues with single bracket syntax, but the double bracket syntax is more straight forward and hence more legible and easier to type.

Double bracket features

Aside from the cleaner syntax, there are a few ‘bonus’ features you gain with double brackets.

With double brackets you can compare to * and ? wildcards, and bracket globbing […]:

$ a="Documents"
$ [[ $a = D* ]] && echo match
match
$ a=hat
$ [[ $a = ?at ]] && echo match
match
$ [[ $a = [chrp]at ]] && echo match
match

You can also use < and > to compare strings lexicographically:

$ a=cat
$ b=hat
$ [[ $a < $b ]] && echo sorted
sorted

And you get an operator =~ for regular expressions:

$ a=cat
$ b="the cat in the hat"
$ [[ $a =~ ^.at ]] && echo match
match
$ [[ $b =~ ^.at ]] && echo match

Note that you should not quote the globbing patterns or the regex pattern.

Summary

  • you should use bash for shell scripting on macOS
  • when using bash, you should use double brackets instead of single brackets
  • double brackets are safer, easier to type and read, and also add few neat features

defaults – the Plist Killer

Last week, fellow MacAdmin Kyle Crawford has discovered that on macOS High Sierra the defaults will delete a property list file with invalid plist/XML syntax, even when you just attempt to read data. Erik Holtham has a more detailed OpenRadar bug.

Patrik Wardle has found the relevant code, which deletes the file when it doesn’t validate.

This is bad. Thanks to all involved for finding, documenting and sharing this.

This is new behavior in High Sierra. I am not yet sure which version of High Sierra this new behavior was introduced. The behavior makes sense in the context of an application attempting to read a settings file, but the defaults tool deleting arbitrary files is, of course, dangerous.

Update: This behavior has been fixed in 10.13.4. However, it is still good practice to avoid defaults for anything other than actual preferences files.

What to do?

As usual, don’t panic. This will only affect Macs running High Sierra and corrupt or broken files. However, if you have a script that accidently points the defaults command at a different file, it will delete that. So you have to use it with care.

It is probably a good practice to verify a file before you attempt to modify it with the defaults command in a script with the plutil command:

if ! plutil -lint path/to/file.plist; then
    echo "broken plist"
    exit 1
else
    defaults path/to/file …
fi

Alternatives to defaults

Alternatively, you can and should use plutil or PlistBuddy to read and modify property list files.

Learn more about plutil, PlistBuddy and other tools to read and write property lists in my book: “Property Lists, Preferences and Profiles for Apple Administrators”

plutil is unfortunately not really useful to read a single value from a property list file. The extract verb will show any value as its own plist. The plutil command is useful to edit existing plist files. (Read details on the plutil command.)

PlistBuddy, however is very useful for both reading an writing values to a property list file:

$ /usr/libexec/PlistBuddy berry.plist -c "print size"
$ /usr/libexec/PlistBuddy berry.plist -c "set size enormous"

PlistBuddy has the additional advantage of allowing to add or edit values nested deep in dict or array structures.

You can get more information on PlistBuddy in its man page or my book.

The interactive mode of PlistBuddy is also very useful.

So the defaults command is dead?

No.

Apple has been warning us to not use defaults for generic property list file editing and parsing for quite a while in the defaults command’s man page:

WARNING: The defaults command will be changed in an upcoming major release to only operate on preferences domains. General plist manipulation utilities will be folded into a different command-line program.

As this warning states, the defaults tool reads and writes data to plist files through macOS’s preferences system. This has the advantage that the tool gets (and changes) the current value whether it is cached in memory or not. When an application is listening for notifications that a preference has changed (not many do) then it will be notified.

Files for preference domains are usually stored in /Library/Preferences/, ~/Library/Preferences or their ByHost subfolders. Sandboxed applications will have their preference plist files in their container.

There, however, many other files that are property list files which are not part of the user defaults system: launchd files, configuration profiles and AutoPkg` recipes to name just a few.

Mac Admins commonly use the defaults tool, despite Apple’s warning, to create, read and edit generic plist files. As mentioned above, plutil, PlistBuddy or direct manipulation through Obj-C, Python or Swift, are better choices for generic plist files.

You can learn about all these options in my book: “Property Lists, Preferences and Profiles for Apple Administrators”

And now Server.app, too!

I have written a book which expands on this topic and is regularly updated. Please check it out: “macOS Installation for Apple Administrators

There is a common understanding that celebrity deaths come in groups of three. Maybe Apple was aiming for that, too. After killing off Imaging and NetBoot/NetInstall, now there is a new support article:

Prepare for changes to macOS Server – Apple Support.

In this article Apple announces they will change the macOS Server app “to focus more on management of computers, devices, and storage on your network.” All other services will be deprecated.

The article lists the deprecated services and provides links to some open source alternatives.

  • Calendar
  • Contacts
  • DHCP
  • DNS
  • Mail
  • Messages (Jabber)
  • NetInstall (NetBoot)
  • VPN
  • Websites (Apache)
  • Wiki

In the beginning these services will remain available when you upgrade from an older version where they are activated, but will be hidden from new installations. In some unspecified future version of macOS Server, the services will be removed.

There are few services not listed here. They were already deprecated or moved to the ‘normal’ macOS in the last Server release. Open Directory and Software Update Server were deprecated and automatically hidden in Server 5.4 (the version which was released with macOS High Sierra). At the same time, Content Caching (Caching Server), File Sharing and Time Machine services moved from the Server app to the Sharing preference pane on macOS (and are available on every Mac, without having to purchase macOS Server). Xcode Server has moved into Xcode 9.

If you are using macOS Server for one of the above solutions, what should you do?

Don’t Panic

Apple is not killing off these services immediately. Server 5.5, which was released together with macOS 10.13.3 still has all the ‘normal’ services. Apple will hide the services in the UI to discourage their use in a future release. For the time being you can continue to use them. However, you need to start planning your move away from macOS Server.

While many Mac administrators would argue that macOS Server is not and never was a “professional” server, or even a server for any kind of deployment, it has found a niche in some small network environments. While the UI was certainly never perfect is has always been somewhat easier than messing with config files.

The replacements that Apple suggest in their article are worthy solutions if you need to maintain the services locally. Many are the open source projects that Apple used inside macOS Server themselves. While this removes the UI for monitoring and configuring the services, it also takes Apple out of the loop for updates and security patches. By getting the software directly you can get more timely updates. It also requires more maintenance and effort from the administrator, especially when you are using multiple services.

To the Cloud!

However, many of the above service are better replaced by cloud-hosted services, such as Office 365 or Google for Business/Education. These will also cover user identity management (replacing Open Directory) and file sharing with cloud storage systems.

For obvious reasons, DNS, DHCP and VPN cannot be run in the cloud. For small networks, these services are usually run on the router. However, if your router cannot run these services then you can run them on a dedicated box.

For my home network I am considering (i.e. finally found an excuse for) a Raspberry Pi.

NetBoot is still dead

Apple recommends NetSUS and BSDPy for NetBoot and NetInstall. These are certainly worthy solutions to host your nbi folders.

However, NetInstall functionality (this has been discussed before) is not present with the iMac Pro. It is to be expected that future new Mac hardware releases will follow the iMac Pro.

If you currently have a NetBoot/NetInstall based imaging or installation based workflow hosted on macOS Server, you need to be exploring alternative onboarding/setup workflows instead. DEP + MDM is the solution that Apple is pushing here.

Whatever solution you will find for your setup, it will require a lot of effort to get working smoothly. Rather than spending time and effort to move your NetBoot setup to BSDPy or NetSUS, leave it where they are for as long as they will still work and spend time on building a new supportable and supported workflow instead.

Whither macOS Server?

The Apple support article states:

macOS Server is changing to focus more on management of computers, devices, and storage on your network.

I would guess that ‘storage on your network’ means Xsan. Which some people still use. Seems weird to leave this as part of macOS Server and not split it out like other services. On the other hand it seems hard to imagine that this is some new server management feature.

What remains, is Profile Manager.

Profile Manager is considered Apple’s reference implementation of the MDM protocol. Most would not recommend using it in professional environments and few do (even fewer happily).

Now, that Apple is effectively reducing the functionality of macOS Server to Profile Manager, the question is: will it remain a mere reference implementation or will Apple finally put the resources behind Profile Manager to make it a usable, affordable and scalable solution?

Or maybe I get to write Profile Manager’s eulogy in a few years time as well. Only time will tell.

Does this mean Apple is leaving Enterprise business?

Really!? No.

In some ways Apple has never been able to enter Enterprise business with their own server products, hardware and software.

But they have been able to enter Enterprise business with their devices, Macs and iPhones and iPad. And because those devices are popular and trendy with Enterprise users, the Enterprises need to support them. That is what the MDM protocol and DEP are for.

With this step, Apple is making it clear that they are not even trying to play in the server business. They are happy to provide the MDM protocol and a reference implementation. They will support the infrastructure necessary to make DEP, MDM and VPP work. Apple is not interested in being the hardware that runs DNS, DHCP, file shares, Mail, calendaring and chat etc. Maybe not even the MDM server. Apple is very happy to leave this business to others. Apple sells devices.

macOS Server has been a neglected step child since the demise of the Xserve. I am surprised it took Apple this long to make it obvious.

I have written a book which expands on this topic and is regularly updated. Please check it out: “macOS Installation for Apple Administrators