Converting Composer dmg ‘installers’ to pkg

Jamf Composer has always had two formats to build installers. The standard pkg and the seemlingly standard (but not) dmg. The pkg option will build a standard pkg installer file, which will install with any system that can install pkg files.

The dmg option will build a standard dmg disk image file, with the payload of the installer as contents. On its own, however, this dmg cannot do anything. The Jamf Pro management system how ever will understand what to do and how to install the files from the dmg to a system. There are certain features in Jamf Pro which can install and distribute files to user directories and templates (called ‘Fill User Templates’ FUT and ‘Fill Every User’ FEU) which only work with dmg installers in Jamf Pro.

However, Jamf themselves have been recommending to use the standard pkg format in favor of their proprietary use of dmg. Also the Composer application is 32-bit and its future is uncertain.

Luckily there are plenty of great other third-party tools to build installer packages. I cover many of them in my book: Packaging for Apple Administrators

In general, it is probably preferable to re-visit your imaging process and rebuild any installer you still may have in dmg format from scratch. However, in some cases that might not be possible or necessary.

Since the Composer generated dmgs contain all the files for the payload in the proper folder structure you can just use the entire mounted volume as your payload root for pkgbuild. You can easily convert a Composer generated installer dmg to a standard pkg with these commands:

1) mount the dmg:

$ hdiutil attach /path/to/Sample.dmg

this will output a bunch of info, the very last bit is the mount point of the dmg /Volumes/Sample (the name will depend on the dmg)

2) build a pkg with the contents of the mounted dmg as a payload:

$ pkgbuild --root /Volumes/Sample --version 1.0 --identifier com.example.sample --install-location / Sample-1.0.pkg

This will create Sample-1.0.pkg in your current working directory. (I like to include the version in the pkg file name, but that is entirely optional.)

3) cleanup: unmount the dmg

$ hdiutil detach /Volumes/Sample

Obviously this will not work well with other dmgs, such as Full System dmgs, or dmgs downloaded from the web, which contain an app that should be dragged to /Applications to install (use quickpkg for those dmgs).

Demystifying `root` on macOS, Part 1

As Mac Adminstrators, we often have to deal with user privileges for files and processes. While doing that we will use administrator privileges and sudo without as much as a second thought.

However, a proper understanding of what these privileges and processes actually do and mean, can help prevent many problems when managing Macs.

  • Part 1: Demystifying root (this post)
  • Part 2: The sudo Command
  • Part 3: root and Scripting
  • Part 4: The Authorization Database

Some History

macOS is based on BSD Unix, which stems back to a time where large mainframes were so expensive they had to be shared among many users. Users and their access privileges control what any user can read, write, or change in the system. These rules prevent conflicts and data loss or theft. When managing these users and their access privileges, there had to be a first, ‘top’, or ‘super user’ which has access to anything.

In Unix and Unix-like systems this user account is traditionally called root. In macOS this user is often also called ‘System Administrator’.

Classic Mac OS, in contrast, had no concept of multiple users built-in to the system. Any person sitting down at a Mac (and any process launched on that Mac) could access and change anything on that system. There were some attempts at adding multi-user functionality to classic Mac OS, but they were ‘added on, not built-in’ and fairly easy to circumvent when a user knew what to do.

User and process management was one of the main benefits Apple touted for the various ‘next generation’ systems Apple introduced in the 90s to succeed classic Mac OS. When Apple bought NeXT and with it the NeXTStep operating system it inherited the unix model of doing so.

Even though the concept of sharing your computer is now relegated to some classroom labs and supercomputer clusters, this model still is present in every macOS and iOS device today. On iOS it is completly invisible to the user, unless a jailbreak is applied. On macOS, however, users and especially admins have to deal with it every day.

Users on macOS

To create a new user on macOS you have go to the ‘Users & Groups’ Preference Pane in System Preferences. Before you can add a new user, you have to unlock the preference pane by clicking the lock icon in the lower left corner. Then the system will prompt for an username and password with adminstrative privileges.

When the account you are logged in as has admin privileges, its name will be pre-filled. When the account is a standard user the username field will be empty and you can enter another user’s name and password.

Once the pane is unlocked, you can hit the ‘+’ icon under the user and will be offered four choices for a new user (from the popup menu next to ‘New Account’:

  • Administrator
  • Standard (the default)
  • Managed with Parental Controls
  • Sharing Only

There are three types of users not present in the popup list

  • Guest
  • System Administrator
  • system services users

The difference between Administrator and Standard accounts is that Administrator accounts are members of the ‘Administrators’ or admin group. This sound simple, but membership in this group bestows many additional benefits.

In day-to-day use Administrator accounts and Standard accounts behave the same. However, there are many situations and workflows on macOS which require authenticating as an Administrator account. As a general rule, a user can affect all the files (and applications) in their home directory and in /Users/Shared, but as soon as you want to change another user, another user’s files or settings that affect all users on a system you need to authenticate as an Administrator account.

The first user created on an unmanaged Mac out of the box will always be an Adminstrator user. Most Mac users use an Administrator account. Many of the workflows built-in to macOS assume an adminstrator account. One example is setting up a new printer.

With an Administrator account you can install third party software. You can also install malicious software. Often malicious software will trick users into installing by masquerading as or hiding in an installer for something useful.

Many consider it a ‘best practice’ to run your everyday work on your Mac with a standard account and only use an administrative account when you have to. However, since you get prompted to authenticate even with an administrative account, the better advice is to take these prompts very seriously and consider what confirming this prompt will really do or install.

The only difference you get when using a standard account is that you need to enter a different username and password in an authentication box instead of just the password. If this helps you pause and consider what you are actually doing, then great! Then this is the proper workflow for you.

However, I suspect that most users would be just as non-considerate of this dialog with a separate username and password as they would otherwise.

macOS Administrator Accounts

The only difference between Adminstrator accounts and Standard accounts is the membership of the admins group.

You can check whether a given user is a member of the admin group with the dseditgroup tool:

$ dseditgroup -o checkmember -u armin admin
yes armin is a member of admin

You can also use this tool to add or remove a user from the admin group:

$ dseditgroup -o edit -n . -a username -t user admin    # add username
$ dseditgroup -o edit -n . -d username -t user admin    # add username

This membership comes with many privileges. Admin users can (after authentication):

  • unlock System Preferences and change system settings
  • install Apple and third party software and installer packages
  • create, change, and delete files owned by other users
  • change access privileges and ownership of files of folders in Finder
  • run and stop (kill) processes owned by other users
  • use sudo in Terminal

and many things more.

On macOS these privileges are controlled mainly by two mechanisms:

  • sudo and the sudoers file
  • the authorization database

sudo is used to gain root privileges in the shell (Terminal). The authorization database controls access privileges everywhere else.

What root can do

The ‘System Administrator’ or the root account controls the system. Mainly the root account can read, update, delete all local user accounts. It can control file and folder privileges and ownership. It can start system services running in the background and assign system network ports (with a port number lower than 1000). Most of this is managed by a process called launchd which is the first process to run on macOS.

Many commands require to be run as root or with elevated root privileges.

What root cannot do: System Integrity Protection (SIP)

On macOS, however, there are limits to what the root account can do. System Integrity Protection is a mechanism which protects important parts of the OS from mnodification, even with root permissions.

Only certain processes signed by Apple are allowed to modify these protected files and directories. Usually this means Apple signed installer pkgs for software and security updates.

Apple Support: About System Integrity Protection on your Mac

Apple lists a set of top-level directories that are protected. However, the list is a bit more detailed. You can use the -O (capital letter ‘O’, not a zero) to see if a file or directory is protected by SIP:

$ ls -lO /usr/
total 0
drwxr-xr-x  978 root  wheel  restricted 31296 Mar 30 18:21 bin
drwxr-xr-x  294 root  wheel  restricted  9408 Mar 30 18:21 lib
drwxr-xr-x  238 root  wheel  restricted  7616 Mar 30 18:21 libexec
drwxr-xr-x    8 root  wheel  sunlnk       256 Dec 28 13:48 local
drwxr-xr-x  248 root  wheel  restricted  7936 Mar 30 18:21 sbin
drwxr-xr-x   46 root  wheel  restricted  1472 Mar 30 18:21 share
drwxr-xr-x    5 root  wheel  restricted   160 Oct  3  2017 standalone

Files and Folders marked with restricted are protected by SIP. Sometimes folders inside a protected folder may not be protected, as the /usr/local/ directory in this example is.

SIP provides more protection than just certain parts of the file system, it also protects changing the boot volume and some other aspects of the OS.

While these limitations on even the root account can be annoying, they provide a level of security that parts of the OS have not been tampered with or changed by other software.

Enabling (and Disabling) root

On macOS the root account exists with a UID of ‘0’. However, it is set up so you cannot log in to a Mac as ‘System Administrator’ or root. (A terrible bug in early 10.13 provided a brief exeception to that rule.)

Note: login as root is disabled for security purposes. It is highly recommended that you leave the root account disabled on macOS and rely on sudo to gain temporary super user privileges when necessary.

If, for some reason, you do need to log in as root, then you can enable the root and provide it with a password. You can do so in either the ‘Directory Utility’ application. After unlocking with your administrator password, you choose ‘Enable Root User’ from the ‘Edit’ menu. You can also change the root account’s password here or disable it again later.

Apple Support: How to enable the root user on your Mac or change your root password

From the command line, you can also use the dsenableroot command:

$ dsenableroot

will enable and/or update the root account. It will interactively ask for admin credentials and for a new password for the root account. Read the command’s man page for details.

$ dsenableroot -d

will interactively disable the root account again.

Becoming root

Different environments and tools have different means of gaining super user or root privileges. While the sudo command should be the preferred means of gaining temporary super user privileges, it is important to know and understand the other options.

LaunchDaemons

Scripts and tools executed from LaunchDaemons run as root unless a different user is specified in the UserName key in launchd property list.

LaunchAgents on the other hand will be executed as the user logging in.

You can get details on how to set up and use LaunchDaemons here.

Run as root in ARD

When you prepare a ‘UNIX command’ to be sent to remote computers in Apple Remote Desktop, you have the option of running the command as the currently logged in user or as a specific user. When you specify root as the user, the script will execute with super user privileges. Since the ARD agent process runs as root on the client, no extra authentication or enabled root account is necessary.

Management Systems

The agent software of most management systems (Jamf, Munki, etc.) is installed to run with root privileges. Therefore, scripts executed by management systems run with root privileges as well.

Installation Scripts

Installation packages also perform their task with root privileges. They also require administrator authentication to start. Any installation scripts (pre-/postinstall scripts) will also run with root privileges.

set-UID bits

There is a special bit you can set on an executable’s mode (or privileges) which tells the system to run this script as the file owner, no matter who actually runs the executable. If the executable file is owned by root it will run with root privileges.

This flag is the “set-user-ID-on-execution bit”, also called the “Set-User-ID”-bit or just “s-bit”.

In the long ls format or with the stat command the set-user-ID bit is shown as an ‘s’ in place of the user’s x bit. One example is the ps command:

$ stat -l /bin/ps
-rwsr-xr-x 1 root wheel 51200 00:57 /bin/ps

Use chmod's u+s to set the set-user-ID bit and u-s to remove it:

$ chmod +x importantcommand [rwxr-xr-x]
$ chmod u+s importantcommand    [rwsr-xr-x]
$ chmod u-s importantcommand    [rwxr-xr-x]

Warning: Obviously it is very important that this executable is not modifiable by other users. They would be able to replace the command with their own code and execute anything with root privileges. Most system commands that have the s-bit set on macOS are protected with SIP.

The sudo command

As mentioned before, the recommended way of gaining super user privileges from the command line in macOS is the sudo command. The name means ‘super user do’ and will perform the following command with root privileges after verifying the user running sudo has the permission to do so.
 
We will look at the sudo command in detail in the next post.

  • Part 1: Demystifying root (this post)
  • Part 2: The sudo Command
  • Part 3: root and Scripting
  • Part 4: The Authorization Database

macOS 10.13.4 Spring Update for Mac Admins

With the recent release of 10.13.4 (the ‘spring update’) a few things have changed for the deployment of macOS. The initial premise is still unchanged: Imaging is still dead.

(NetInstall got a bit of a life extension, though.)

Quick Recap

High Sierra came with many features for both users and admins. However, for Mac admins it brought the support article HT208020: “Upgrade macOS on a Mac at your institution”. In this article Apple lists the supported means of installing and upgrading macOS and explicitly states that ‘monolithic imaging’ is not recommended and will not ensure that the firmware of a Mac is of the correct version to run the OS image that was just laid down. (I posted an article on this back in October.)

Then, with the release of the iMac Pro in December, it became clear that NetBoot and NetInstall, will not be supported with the new hardware. The assumption is, that will also be true for all new Macs with the T2 or a newer system controller. (My post on that, from December.)

NetBoot and NetInstall are used by many administrators to provide a centralized workflow to (re-)deploy or re-purpose a Mac. Common tools are macOS Server’s NetInstall, DeployStudio, AutoCasperNBI, or Imagr.

On top of that some NetInstall features, such as automated installation and adding custom packages were broken in earlier releases of 10.13.

Also booting off external drives became much harder on the iMac Pro, since its default security prohibits external boot and you have to go through the entire installation process before you can re-enable it.

Also Apple announced that the macOS Server app will lose most features, including NetBoot/NetInstall in a future release later this year. (My post on that, from January.)

What changed in 10.13.4?

The ‘spring update’ macOS 10.13.4 brought a few welcome changes.

We got a glimpse of some of these in February when HT208020 was briefly updated with new information. Interestingly, now that 10.13.4 is released, HT208020 has not been updated.

However, we got new detailed information in this article:

Enterprise content:
 - No longer disables User Approved Kernel Extension Loading on MDM-enrolled devices. For devices with DEP-initiated or User Approved MDM enrollment, administrators can use the Kernel Extension Policy payload.
- Improves Spotlight search results for files stored on network mounts.
- Properly evaluates ACLs on SMB share points.
- Adds the --eraseinstall flag to the startosinstall command in the macOS Installer app at Contents/Resources/startosinstall. Use this flag to erase and install macOS on a disk. For details, run startosinstall with the --usage flag.
- Updates System Image Utility to allow creating NetInstall images that erase and install macOS to a named target volume.

Robert Hammen posted a great summary on Slack.

Not documented here, but 10.13.4 also fixes the bug that the defaults command deletes non-plist files.

UAKEL

The first one is really important. Apple introduced a new security feature called “User Approved Kernel Extensions” (UAKEL) in 10.13. This means that third party kernel extensions have to be approved by the user at the Mac (within 30 minutes after installing) before they can be loaded.

Prepare for changes to kernel extensions in macOS High Sierra – Apple Support

In 10.13.0–10.13.3 Apple simplified the life of Mac admins by disabling UAKEL on Macs which were enrolled with an MDM. In 10.13.4 Apple added a Kernel Extension Policy profile payload which allows Mac admins to whitelist certain Kernel Extensions centrally from the MDM.

This is a useful addition and allows Mac admins to manage Kernel Extensions before they are installed and without the necessity of user interaction.

However, 10.13.4 also changes the previous behavior for UAKEL on MDM managed Macs. Since admins now have a way of whitelisting Kernel Extensions, UAKEL will be enabled, even on MDM managed Macs.

If you were installing a Kernel Extension on a managed Mac from 10.13.0 to 10.13.3 it would work, since MDM disabled UAKEL. Once you upgrade that Mac to 10.13.4, UAKEL will turn active and it will block the Kernel Extension from loading, unless there is a profile whitelisting the extension!

When the extension was previously manually approved on 10.13 or grandfathered in when the Mac was upgraded from 10.12, then it will still run under 10.13.4. While all of this has some internal logic, it will lead to strange situations were Kernel Extensions will load on some clients and not load on others.

The best way to avoid the confusion is to have the Kernel Extension Policy profile ready and in place before your clients update to 10.13.4. In other words: now!

User Approved MDM

However, (yes, there is another ‘however’ here). The Kernel Extension Policy profile has to “be delivered via a user approved MDM server.” This is another level of security introduced to keep the user in the loop.

Apparently there is malware/trojans that tricks users into accepting MDM profiles to connect their iOS devices or Macs to malicious MDMs. I am not convinced these new measures will be effective against this kind of trickery, though.

Macs deployed with the Device Enrollment Program (DEP) are considered “user approved” by default. Otherwise the user has to approve the MDM profile when it is installed. Again, this cannot be automated or approved over remote control.

This can throw a wrench into non-DEP installation workflows (sometimes also called user-initiated enrollment). Some management systems will let the user download a pkg that installs the MDM profile and necessary certificates. Some solutions install the MDM through an agent software (which is still necessary for many tasks that the mdm client software cannot perform). Either of these workflows will require the user to go to the Security pane in system preferences within 30 minutes after installation.

Note: Jeremy Baker found a creative way that uses User-interface scripting to click the approve button. However, his script requires Accessibility access for the script, which also cannot be provided in an automated fashion. (The database in question is protected by SIP.)

However, in 10.13.4, when a User installs the MDM profile directly by double-clicking it, they will be prompted to approve the MDM as part of the profile installation dialogs, streamlining the process.

Rich Trouton has documented the new workflows for the user in these articles:

For Jamf you will need to upgrade your Jamf Pro server to version 10.3 to get the new workflow. Other management solutions may already implement this or also need to be on the latest version to work well with these new requirements.

startosinstall

The next interesting new feature is for startosinstall which gains a new --eraseinstall option.

Graham Pugh has already documented this very well:

Erase All Contents And Settings – erase and reinstall macOS in situ

This allows for automated workflows where you can wipe and re-install macOS, add a few custom packages to the installation process with --installpackage arguments, which configure your management system. Then after first boot your management system takes over and installs/configures the rest.

It is important to note that startosinstall uses some APFS volume creation trickery to make --eraseinstall work. This means that you cannot run --eraseinstall on a Mac with a HFS+ system volume. You have to already be on a 10.13 system with APFS to use this option.

Nevertheless,--eraseinstall is a welcome and necessary addition to startosinstall. However, what struck me most about this is that this is the first time Apple is even mentioning the startosinstall command in any documentation. Since this tool is central to many approaches to automate the installation process, I am happy it is finally getting recognized as ‘official’.

NetInstall

The last feature isn’t really new. Apple fixed a bug that has been around since 10.13.0. In previous versions of 10.13, when you built a NetInstall set with System Image Utility and chose the option for “Automated Erase Install” on a certain volume (by name), the installation would just stall at a grey screen. Now, this option works as expected, when you build the NetInstall set from 10.13.4.

The days of NetInstall are still numbered because the iMac Pro (and presumably future Macs with similar controllers) cannot NetBoot at all, and macOS Server is loosing NetInstall along with many other services. Nevertheless, this provides another Apple-supported workflow for automated erase and re-install which you can customize with your own packages.

NetInstall (with or with out the erase) is also a good workflow to upgrade Macs with older versions of macOS to 10.13. (In this case we don’t care that it doesn’t work for iMac Pros, since they already come with 10.13.)

Since, currently, in most deployments the iMacs Pro will be a significant minority (if present at all), this allows administrators to deploy a well known workflow (NetInstall) for the existing fleet this year, while figuring out new workflows (startosinstall + DEP?) for future Macs.

There are, however, still a few kinks left with NetInstall workflows:

More Options

There was no Spring Update resurrection: Imaging is still dead.

It is obvious, that while Apple may not be going in exactly the direction that we would like them to, they are listening to criticism and providing solutions, albeit slowly. The additions to startosinstall and the fix to NetInstall now allow for automated wipe and install workflows. These were not really possible before.

However, installation workflows, however you start them, are much slower than block copying a prepared image. This is an important consideration for education deployments, which usually re-image dozens, hundreds, or even thousands of Macs over the summer break.

But this update now provides a few useful options for High Sierra:

  • use NetInstall Automated Erase to wipe and upgrade existing Macs from earlier macOS versions to 10.13
  • use startosinstall to upgrade and update Macs (including iMac Pros) to the latest version of 10.13
  • use startosinstall with --eraseinstall to ‘wipe and install’ existing 10.13 Macs
    • this will only work with 10.13 SSD Macs with APFS. Since Fusion drives and HDs cannot get APFS yet, you will have to use NetInstall to convert/upgrade these Macs, when (if ever) they support APFS
    • iMac Pros cannot use automated NetInstall, however, since they definitely have APFS 10.13 already installed, you can use startosinstall --eraseinstall
  • if fast restore times are important (like a loaner MacBooks scenario, where devices need to get reset to a well-known state quickly), use any of the above to get the Macs up to the latest 10.13, then you can still use imaging to quickly lay down a ‘fresh’ image of the same macOS version
    • you will need to put in extra effort to keep the image system version in sync with the version installed (or upgraded) on the target Macs

Now is the time to start testing, testing, testing to get your workflows ready for the summer re-installation marathon and the 10.14 release in the fall. (And to give Apple another chance to fix the remaining issues in the next update.)

I have ignored the file share changes in this post. While these are certainly important, they don’t really have influence on deployment strategy.

Join the MacAdmins on the MacAdmins Slack to share experiences and solutions!

Apple’s new Upgrade/Update Strategy

There is another aspect of the Spring Update.

Apple switched to the yearly upgrade cycle for Mac OS X with 10.7. (Upgrade meaning a ‘major’ version change, i.e. 10.8 -> 10.9. Nevermind that Apple uses the second version number for the major version. More on macOS version numbers here.) Apple did summer releases for 10.7 and 10.8 and then switched to the Fall release schedule. Since 10.9 releases have been reliably in late September or October.

The rule used to be that upgrades would bring lots of new features, both visible to the user and under the hood and then Apple would release updates (the third number in the version) to fix bugs and issues. Sometimes new features would be introduced in updates, but those were rare exceptions and usually done to match with iOS or iCloud (dotMac, Mobile Me) features.

The rule of thumb was that the first two or three releases were for ‘early adopters’ only and it would be fairly safe to migrate to the new major version by the third or fourth update. Admins could join the developer program to get access to the developer releases of the next upgrade after WWDC, but getting your hand on early releases of the updates was more difficult.

iOS, on the other hand, has had a different pattern. Apple released iOS 4.2 in the spring with new features to support the then new iPad. Since the iPad and iPhone hardware releases rarely synced to the same time of year, iOS has had a pattern with a major new iOS release in the Fall (usually with a new iPhone) and a ‘Spring Update’ with new features to match with a new iPad.

Even though the Mac hardware follows yet other cycles. We now see this Fall Upgrade/Spring Update pattern with macOS as well.

Apple has added new features in 10.13.4. Some are visible to users (eGPU support, Business Chat, new privacy dialogs), some are for client management (UAMDM, UAKEL profiles, new configuration profile documentation).

We now also have a public beta program for iOS and macOS which covers not just the major upgrades, but ‘minor’ updates as well. The beta versions for iOS 11.4 and macOS 10.13.5 were released right after 11.3 and 10.13.4. And it looks like they will contain yet more new features (iMessages on iCloud and AirPlay 2).

The fact that Apple is willing to add new features or change existing fucntionality at any time during the update cycle is a win to users. It is also a reaction to competitors more-cloud based solutions that can be updated at any time.

However, for us admins this means change:

  • the notion of a ‘stable’ release is a thing of the past.

Features might be added or changed at any time during the upgrade cycle. Different parts of the deployment and workflow will be in different stages of ‘maturity’. Additionally, parts of the workflow (DEP, MDM, and VPP) exist in the cloud and might also change at any time (Apple School Manager was not introduced with a major iOS release and it looks like Apple Business Manager will not sync with a major iOS or macOS release, either).

macOS and iOS don’t stand alone. Apple (and third parties) are building networks of operating systems, software, devices and cloud-based services. Scheduling these releases in to a yearly major update cycle must be nearly impossible. It is also not necessary as distribution of software has become reliable, secure, and fast enough to push frequent updates.

In some ways Apple is reacting to their more cloud-based competitors, which can and will push incremental updates to their systems continually.

  • ‘permanent beta’ mode

The beta versions for iOS 11.4 and macOS 10.13.5 were released hours after the release 11.3 and 10.13.4. If you are concerned how the new updates will work (or not) in your environment, you have to be testing now.

This is being a pro-active adminstrator. Rather than waiting for problems to occur and trying to fix them, you are anticipating problems and trying to pre-empt them. The traditional release cycle of the past allowed us to switch between the two roles over the year. Now, we are either in permanent beta-test or in permanent break/fix.

I don’t think any one likes it, but it is the situation we have to deal with and I don’t see that changing any time soon. You will have to adapt your and your organization’s workflows to adapt to this new situation.

  • Apple is listening and communicating the changes

Provide feedback (bugreports) to Apple. Not all the issues you find will be fixed (some might be).

However, this gives you time to document issues for your users and allow you to implement management strategies to ameliorate them. (Even if all you can do is write knowledge base articles along the line of “we know this is broken” you are saving some people a lot of time and nerves.)

I find it very interesting and encouraging that we are learning about these changes in offical support articles. Also then changes to UAKEL and UAMDM were based on user feedback, mainly adminstrators who complained that the feature as it was initially implemented was unmanageable.

Of course there are many other challenges and issues which have not been fixed (yet). But it is encouraging to see this kind of feedback work.

MacAD.UK Terminal Witchcraft and Wizardry Session

The fine people from MacAD.UK have started putting the sessions from this year’s conference online for every one to use.

The video of my presentation titled “The macOS School of Terminal Witchcraft and Wizardry” was uploaded earlier today. Go watch it!

You can find the slides and some notes on my session here.

 

Setting the PATH in Scripts

A discussion that comes up frequently on MacAdmin Slack and other admin discussions is:

Should commands in scripts have their full path hardcoded or not?

Or phrased slightly differently, should you use /bin/echo or just echo in your admin scripts?

I talked about this briefly in my MacSysAdmin session: Scripting Bash

Why can’t I just use the command?

When you enter a command without a path, e.g. echo, the shell will use the PATH environment variable to look for the command. PATH is a colon separated list of directories:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The shell will look through these directories in order for the given command.

You can read more detail about the PATH and environment variables in these posts:

PATH is Unreliable

The example PATH above is the default on macOS on a clean installation. Yours will probably look different – mine certainly does:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/munki:/Users/armin/bin

Third party applications and tools can and will modify your PATH. You yourself might want to change your PATH in your shell profile.

But on top of that, the PATH will be different in different contexts. For example, open the Script Editor application, make a new script document, enter do shell script "echo $PATH" and run the script by hitting the run/play button.

The small AppleScript we just built runs the shell command echo from the AppleScript context. The result is

/usr/bin:/bin:/usr/sbin:/sbin

Note how the PATH in this context is different from the default macOS PATH in Terminal and also different from your PATH.

I also built a simple payload-free installer package which only runs the echo "Installer PATH: $PATH" command. You will have to search through /var/log/install.log to get the output:

installd[nnn]: ./postinstall: Installer PATH: /bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec

Which is yet another, different PATH.

Solutions to the PATH confusion

The PATH may be different in different contexts that your script may run in.

That means we cannot be consistently certain if a command will be or which command will be found in a script in different contexts. This is, obviously, bad.

Mac system adminstrative scripts which can run in unusual contexts, such as the Login Window, NetInstall, the Recovery system or over Target Disk Mode and usually run with root privileges. You really want to avoid any uncertainty.

There are two solutions to make your scripts reliable in these varying contexts:

  1. hardcode the full path to every command
  2. set the PATH in the script

Both are valid and have upsides and downsides. They are not exclusive and can be both used in the same script.

Going Full PATH

You can avoid the unreliability of the PATH by not using it. You will have to give the full path to every command in your script. So instead of

#!/bin/bash

echo "hello world"

you have to use:

#!/bin/bash

/bin/echo "hello world"

When you do not know the path to a command you can use the which command to get it:

$ which defaults
/usr/bin/defaults

Note: the which command evaluates the path to a command in the current shell environment, which as we have seen before, is probably different from the one the script will run in. As long as the resulting PATH starts with one of the standard directories (/usr/bin, /bin, /usr/sbin, or /sbin) you should be fine. But if a different PATH is returned you want to verify that the command is actually installed in all contexts the script will run in.

Using full paths for the commands works for MacAdmin scripts because Mac administrative scripts will all run on some version of macOS (or OS X or Mac OS X) which are very consistent in regard to where the commands are stored. When you write scripts that are supposed to run on widely different falvors of Unix or Linux, then the location of certain commands becomes less reliable.

Also note that some elements of shell scripting are commands, even if they don’t look like it. For example the square bracket [ is actually a command:

$ which [
/bin/[

so if you are using full command paths, you should use if /bin/[ … instead of if [ ….

(On the other hand, you should be using double brackets [[ instead of single brackets for tests, since they are safer and more flexible, but that is a topic for another post.)

Choosing your own PATH

The downside of hardcoding all the command paths is that you will have to memorize or look up many command paths. Also, the extra paths before the command make the script less legible, especially with chained (piped) commands.

If you want to save effort on typing and maintenance, you can set the PATH explicitly in your script. Since you cannot rely on the PATH having a useful value or even being set in all contexts, you should set the entire PATH.

This should be the first line after the shebang in a script:

#!/bin/bash
export PATH=/usr/bin:/bin:/usr/sbin:/sbin

echo "hello world"

Note: any environment variable you set in a script is only valid in the context of that script and any sub-shells or processes this script calls. So it will not affect the PATH in other contexts.

This has the added benefit of providing a consistent and well known PATH to child scripts in case they don’t set it themselves.

The downside of this is that even with a known PATH you cannot be entirely sure which tool will be called by the script. If something installed a modified copy of echo in /usr/bin it would be called instead of the expected /bin/bash.

However, on macOS the four standard locations (/usr/bin, /bin, /usr/sbin, /sbin, as well as the less standard /usr/libexec) are protected by System Integrity Protection (SIP) so we can assume those are ‘safe’ and locked down.

/usr/local/bin is a special case

But notice that I do not include /usr/local/bin when I set the PATH for my scripts, even though it is part of the default macOS PATH. The PATH seen in the installer context does not include /usr/local/bin, either.

/usr/local/bin is a standard location where third party solutions can install their commands. It is convenient to have this directory in your interactive PATH under the assumption that when you install a tool, you want to use it easily.

However, this could create conflicts and inconsistent results for administrative scripts. For example, when you install bash version 4, it will commonly be installed as /usr/local/bin/bash, which (with the standard PATH) overrides the default /bin/bash version 3.

Since you chose to install bash v4, it is a good assumption that you would want the newer version over the older one, so this is a good setting for the interactive shell.

But this might break or change the behavior of administrative scripts, so it is safe practice to not incluse /usr/local/bin in the PATH for admin scripts.

Other Tools

When you use commands from other directories (like /usr/libexec/PlistBuddy, or third party tools like the Munki or Jamf tools) then it is your choice whether you want to use full path for these commands or (when you use the commands frequently in a script) add their directory to the PATH in your script:

E.g. for Munki

export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki

or Jamf

export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/jamf/bin

Since the third party folders are not protected by SIP, it is safer to append them to the PATH, so they cannot override built-in commands.

Commands in Variables

Another solution that is frequently used for single commands with long paths is to put the entire path to the command in a variable. This keeps the script more readable.

For example:

#!/bin/bash

# use kickstart to enable full Remote Desktop access
# for more info, see: http://support.apple.com/kb/HT2370

kick="/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart"

#enable ARD access
$kick -configure -access -on -users remoteadmin,localadmin -privs -all
$kick -configure -allowAccessFor -specifiedUsers
$kick -activate

Note: that you need to quote the variable when the path to the command contains spaces or other special characters.

Summary

As a system administrator it is important to understand the many contexts and environments that a script might be run in.

Whether you choose to write all command paths or explictly set the PATH in the script is a matter of coding standards or personal preference.

You can even mix and match, i.e. set the PATH to the ‘default four’ and use command paths for other commands.

My personal preference is the solution where I have to memorize and type less, so I set the PATH in the script.

Either way you have to be aware of what you are doing and why you are doing it.

All the Articles

While organizing links for my upcoming MacAD.UK presentation, I noticed that there are quite a few series of articles I have written over the past year or so. Some are weren’t quite intended as series but turned into loose sequels. Some were intended as series from the start, but usually turned out longer than intended.

Anyway, while I was organizing links anyway I also created a new page on this site that organizes the series:

Article Series on Scripting OS X

I intend to keep it updated as new articles and series are added.

Enjoy!

Show the Installer Log During High Sierra Installation

In previous versions of macOS you could show the installer log during system installation with the keyboard shortcut ⌘L. Since macOS Sierra, much of the system installation process happens on a black screen with a white Apple logo and a progress bar.

On that screen, you can use the keyboard shortcut ⇧⌃⌥⌘W, and it will switch back to traditional UI with the grey window. There you can then use ⌘L to show the log window again.

(Double mouse pointers are because I was running the installer in a VM to record the screen.)