Demystifying `root` on macOS, Part 1

As Mac Adminstrators, we often have to deal with user privileges for files and processes. While doing that we will use administrator privileges and sudo without as much as a second thought.

However, a proper understanding of what these privileges and processes actually do and mean, can help prevent many problems when managing Macs.

Some History

macOS is based on BSD Unix, which stems back to a time where large mainframes were so expensive they had to be shared among many users. Users and their access privileges control what any user can read, write, or change in the system. These rules prevent conflicts and data loss or theft. When managing these users and their access privileges, there had to be a first, ‘top’, or ‘super user’ which has access to anything.

In Unix and Unix-like systems this user account is traditionally called root. In macOS this user is often also called ‘System Administrator’.

Classic Mac OS, in contrast, had no concept of multiple users built-in to the system. Any person sitting down at a Mac (and any process launched on that Mac) could access and change anything on that system. There were some attempts at adding multi-user functionality to classic Mac OS, but they were ‘added on, not built-in’ and fairly easy to circumvent when a user knew what to do.

User and process management was one of the main benefits Apple touted for the various ‘next generation’ systems Apple introduced in the 90s to succeed classic Mac OS. When Apple bought NeXT and with it the NeXTStep operating system it inherited the unix model of doing so.

Even though the concept of sharing your computer is now relegated to some classroom labs and supercomputer clusters, this model still is present in every macOS and iOS device today. On iOS it is completly invisible to the user, unless a jailbreak is applied. On macOS, however, users and especially admins have to deal with it every day.

Users on macOS

To create a new user on macOS you have go to the ‘Users & Groups’ Preference Pane in System Preferences. Before you can add a new user, you have to unlock the preference pane by clicking the lock icon in the lower left corner. Then the system will prompt for an username and password with adminstrative privileges.

When the account you are logged in as has admin privileges, its name will be pre-filled. When the account is a standard user the username field will be empty and you can enter another user’s name and password.

Once the pane is unlocked, you can hit the ‘+’ icon under the user and will be offered four choices for a new user (from the popup menu next to ‘New Account’:

  • Administrator
  • Standard (the default)
  • Managed with Parental Controls
  • Sharing Only

There are three types of users not present in the popup list

  • Guest
  • System Administrator
  • system services users

The difference between Administrator and Standard accounts is that Administrator accounts are members of the ‘Administrators’ or admin group. This sound simple, but membership in this group bestows many additional benefits.

In day-to-day use Administrator accounts and Standard accounts behave the same. However, there are many situations and workflows on macOS which require authenticating as an Administrator account. As a general rule, a user can affect all the files (and applications) in their home directory and in /Users/Shared, but as soon as you want to change another user, another user’s files or settings that affect all users on a system you need to authenticate as an Administrator account.

The first user created on an unmanaged Mac out of the box will always be an Adminstrator user. Most Mac users use an Administrator account. Many of the workflows built-in to macOS assume an adminstrator account. One example is setting up a new printer.

With an Administrator account you can install third party software. You can also install malicious software. Often malicious software will trick users into installing by masquerading as or hiding in an installer for something useful.

Many consider it a ‘best practice’ to run your everyday work on your Mac with a standard account and only use an administrative account when you have to. However, since you get prompted to authenticate even with an administrative account, the better advice is to take these prompts very seriously and consider what confirming this prompt will really do or install.

The only difference you get when using a standard account is that you need to enter a different username and password in an authentication box instead of just the password. If this helps you pause and consider what you are actually doing, then great! Then this is the proper workflow for you.

However, I suspect that most users would be just as non-considerate of this dialog with a separate username and password as they would otherwise.

macOS Administrator Accounts

The only difference between Adminstrator accounts and Standard accounts is the membership of the admins group.

You can check whether a given user is a member of the admin group with the dseditgroup tool:

$ dseditgroup -o checkmember -u armin admin
yes armin is a member of admin

You can also use this tool to add or remove a user from the admin group:

$ dseditgroup -o edit -n . -a username -t user admin    # add username
$ dseditgroup -o edit -n . -d username -t user admin    # delete username

This membership comes with many privileges. Admin users can (after authentication):

  • unlock System Preferences and change system settings
  • install Apple and third party software and installer packages
  • create, change, and delete files owned by other users
  • change access privileges and ownership of files of folders in Finder
  • run and stop (kill) processes owned by other users
  • use sudo in Terminal

and many things more.

On macOS these privileges are controlled mainly by two mechanisms:

  • sudo and the sudoers file
  • the authorization database

sudo is used to gain root privileges in the shell (Terminal). The authorization database controls access privileges everywhere else.

What root can do

The ‘System Administrator’ or the root account controls the system. Mainly the root account can read, update, delete all local user accounts. It can control file and folder privileges and ownership. It can start system services running in the background and assign system network ports (with a port number lower than 1000). Most of this is managed by a process called launchd which is the first process to run on macOS.

Many commands require to be run as root or with elevated root privileges.

What root cannot do: System Integrity Protection (SIP)

On macOS, however, there are limits to what the root account can do. System Integrity Protection is a mechanism which protects important parts of the OS from mnodification, even with root permissions.

Only certain processes signed by Apple are allowed to modify these protected files and directories. Usually this means Apple signed installer pkgs for software and security updates.

Apple Support: About System Integrity Protection on your Mac

Apple lists a set of top-level directories that are protected. However, the list is a bit more detailed. You can use the -O (capital letter ‘O’, not a zero) to see if a file or directory is protected by SIP:

$ ls -lO /usr/
total 0
drwxr-xr-x  978 root  wheel  restricted 31296 Mar 30 18:21 bin
drwxr-xr-x  294 root  wheel  restricted  9408 Mar 30 18:21 lib
drwxr-xr-x  238 root  wheel  restricted  7616 Mar 30 18:21 libexec
drwxr-xr-x    8 root  wheel  sunlnk       256 Dec 28 13:48 local
drwxr-xr-x  248 root  wheel  restricted  7936 Mar 30 18:21 sbin
drwxr-xr-x   46 root  wheel  restricted  1472 Mar 30 18:21 share
drwxr-xr-x    5 root  wheel  restricted   160 Oct  3  2017 standalone

Files and Folders marked with restricted are protected by SIP. Sometimes folders inside a protected folder may not be protected, as the /usr/local/ directory in this example is.

SIP provides more protection than just certain parts of the file system, it also protects changing the boot volume and some other aspects of the OS.

While these limitations on even the root account can be annoying, they provide a level of security that parts of the OS have not been tampered with or changed by other software.

Enabling (and Disabling) root

On macOS the root account exists with a UID of ‘0’. However, it is set up so you cannot log in to a Mac as ‘System Administrator’ or root. (A terrible bug in early 10.13 provided a brief exeception to that rule.)

Note: login as root is disabled for security purposes. It is highly recommended that you leave the root account disabled on macOS and rely on sudo to gain temporary super user privileges when necessary.

If, for some reason, you do need to log in as root, then you can enable the root and provide it with a password. You can do so in either the ‘Directory Utility’ application. After unlocking with your administrator password, you choose ‘Enable Root User’ from the ‘Edit’ menu. You can also change the root account’s password here or disable it again later.

Apple Support: How to enable the root user on your Mac or change your root password

From the command line, you can also use the dsenableroot command:

$ dsenableroot

will enable and/or update the root account. It will interactively ask for admin credentials and for a new password for the root account. Read the command’s man page for details.

$ dsenableroot -d

will interactively disable the root account again.

Becoming root

Different environments and tools have different means of gaining super user or root privileges. While the sudo command should be the preferred means of gaining temporary super user privileges, it is important to know and understand the other options.

LaunchDaemons

Scripts and tools executed from LaunchDaemons run as root unless a different user is specified in the UserName key in launchd property list.

LaunchAgents on the other hand will be executed as the user logging in.

You can get details on how to set up and use LaunchDaemons here.

Run as root in ARD

When you prepare a ‘UNIX command’ to be sent to remote computers in Apple Remote Desktop, you have the option of running the command as the currently logged in user or as a specific user. When you specify root as the user, the script will execute with super user privileges. Since the ARD agent process runs as root on the client, no extra authentication or enabled root account is necessary.

Management Systems

The agent software of most management systems (Jamf, Munki, etc.) is installed to run with root privileges. Therefore, scripts executed by management systems run with root privileges as well.

Installation Scripts

Installation packages also perform their task with root privileges. They also require administrator authentication to start. Any installation scripts (pre-/postinstall scripts) will also run with root privileges.

set-UID bits

There is a special bit you can set on an executable’s mode (or privileges) which tells the system to run this script as the file owner, no matter who actually runs the executable. If the executable file is owned by root it will run with root privileges.

This flag is the “set-user-ID-on-execution bit”, also called the “Set-User-ID”-bit or just “s-bit”.

In the long ls format or with the stat command the set-user-ID bit is shown as an ‘s’ in place of the user’s x bit. One example is the ps command:

$ stat -l /bin/ps
-rwsr-xr-x 1 root wheel 51200 00:57 /bin/ps

Use chmod's u+s to set the set-user-ID bit and u-s to remove it:

$ chmod +x importantcommand [rwxr-xr-x]
$ chmod u+s importantcommand    [rwsr-xr-x]
$ chmod u-s importantcommand    [rwxr-xr-x]

Warning: Obviously it is very important that this executable is not modifiable by other users. They would be able to replace the command with their own code and execute anything with root privileges. Most system commands that have the s-bit set on macOS are protected with SIP.

The sudo command

As mentioned before, the recommended way of gaining super user privileges from the command line in macOS is the sudo command. The name means ‘super user do’ and will perform the following command with root privileges after verifying the user running sudo has the permission to do so.

We will look at the sudo command in detail in the next post.

Installing and Using Command Line Tools

There are many command line tools and scripts you can download and install that are very useful for Mac Admins.

(Just a representative list, certainly not complete.)

Some of these tools provide installer packages that deploy the tool in the proper directory – usually /usr/local/bin so that you and other users can use it. (/usr/local/bin is in the macOS default PATH.)

However, many of these tools, such as munkipkg or my own quickpkg just come as a git project folder, with no or few instructions to get it set up. The assumption is, that when you use these tools you are familiar enough with the shell to make them work.

There are actually several approaches to getting these tools to actually work for you, most with different upsides and downsides. This post will talk about a few of them.

Getting the Tool

Before you can choose a method to run the tool, you need to get it. Many admins share their scripts and tools through a hosted service like Github. My quickpkg tool, for example, is a python script hosted as an open Github repository.When you follow that link you will see the main project page. The page has a menu area up top, a file list in the middle and below an area where an introduction to the project (the ReadMe file) is shown. It is worth to read the ReadMe in case they have special installation instructions.

Download the Release

Git is a version management tool that lets you track changes throughout the coding process. Github is one popular service to host these projects online. Contributors of a project have the option of marking steps of the project as a ‘release.’ Releases are considered a tested and stable stop in between less reliable developmental steps.

Releases will be shown in the project’s ‘releases’ page (link in the middle of the page, above the file list). (quickpkg releases page

On the releases page you will see a list of releases with the newest on top. At the very least each release will have a snapshot of the project’s code as a zip or tar.gz archive. Some projects provide other archives or installers such as dmg or pkg as well.

Download the Current Project

Some projects do not manage releases. (You will see ‘0 releases’ in the tool bar.) Then you can still download the most recent version of the project. There is a large green ‘Clone or download’ button on the right area above the project’s file list for this. When you click that button it will expand to show some more options.

‘Download ZIP’ will simply download an archive of the current state of project, much like the release download would.

When you download the archives, either through the releases page or from the ‘Download ZIP’ button, the resulting project folder will not be connected with the Github project any more. If you just want to use the current version, then that is fine and will serve you well. If you want an updated version in the future you will simply download the newer version and replace the tool you already have.

If you rather use git to download and manage the code, then you can do that here, too. However, that is a topic for another post.

Using the Tool

However you get the project you will now have a directory with the tool and any supporting files. You can already change directory to this folder in Terminal (drag the folder on to the Terminal icon to open a new Terminal already changed to it) and run the tool directly:

$ cd ~/Projects/quickpkg/
$ ./quickpkg
usage: quickpkg [-h] [--scripts SCRIPTS] [--preinstall PREINSTALL]
                [--postinstall POSTINSTALL]
                [--ownership {recommended,preserve,preserve-other}]
                [--output OUTPUT] [--clean] [--no-clean] [--relocatable]
                [--no-relocatable] [--sign SIGN] [--keychain KEYCHAIN]
                [--cert CERT] [-v] [--version]
                item_path
quickpkg: error: too few arguments
This will do for tools that you use rarely. But for tools that you want to use frequently typing the path to the tool is quite cumbersome.

Put it in the PATH

The PATH environment variable lists the directories where the shell looks for commands. You could add the project directory of the tool you just added to the PATH, but that would be tedious to manage.

An easier solution is to copy the tool to /usr/local/bin. This is the designated directory for custom commands. /usr/local/bin is also in the default macOS PATH.

However, copying the tool has some downsides. When the tool get’s updated you will have to copy the newer version, as well. Also some tools may require additional resources or libraries that reside in its project directory.

Instead of moving the tool, you can create a symbolic link to the tool in /usr/local/bin.

I keep the project folders of tools in ~/Projects so I use the command:

$ sudo ln -s ~/Projects/quickpkg/quickpkg /usr/local/bin
$ ls -al /usr/local/bin/quickpkg 
lrwxr-xr-x  1 root  wheel /usr/local/bin/quickpkg -> /Users/armin/Projects/quickpkg/quickpkg
Since symbolic links use paths, this has the advantage that when you download a newer version of the project to the same location, the link will point to the new version.

Putting links to a tool in /usr/local/bin has a few downsides (or upsides, depending on your perspective):

  • you need to have administrator privileges to change /usr/local/binlinks/tools you add to /usr/local/bin affect all users on that Mac

Set your own PATH

When you want to have the tools only affect your shell environment you need to do a bit more work.

First you need to choose a location where you tools or links should live. I have created a directory ~/bin for that purpose.

$ mkdir ~/bin
When you don’t want anyone else on the Mac to see what you are doing in that directory, you can remove everyone else’s access to it:
$ chmod 700 ~/bin
If you want you can also hide the directory in the Finder:
$ chflags hidden ~/bin
(Use the same command with nohidden to make Finder show it again.)

(To test the following properly, you need to delete the symbolic link we created earlier in /usr/local/bin. If that still exists the shell will use that, since it comes earlier in the PATH.)

You can create a symbolic link to the in ~/bin with

$ ln -s ~/Projects/quickpkg/quickpkg ~/bin
However, these still will not work, since we need to add the ~/bin directory to your personal PATH.

To do that you need to add this line to your ~/.bash_profile or ~/.bashrc:

export PATH=$PATH:~/bin
(Read more about how to create a bash profile here and here. This assumes you are using bash, the default on macOS. Other shells will have other locations where you can change environment variables.)

Then open a new Terminal window or type source ~/.bash_profile so that the new profile is loaded in the current window and try running the command.

Success!

Single Brackets vs Double Brackets

In my recent post I mentioned in passing, that you should be using double brackets [[…]] for tests in bash instead of single brackets.

This is the post where I explain why. I also talked about this briefly in my MacSysAdmin session: Scripting Bash

Double Brackets are a bashism

Double brackets were originally introduced in ksh and later adopted by bash and other shells. To use double brackets your shebang should be #!/bin/bash not #!/bin/sh.

Since sh on macOS is bash pretending to be sh, double brackets will still work with the wrong shebang, but then your script might break on other platforms where a different shell might be pretending to be sh. Consistent behavior across platforms is the main point why sh is still around, so don’t use double brackets in sh (or use bash to use double brackets).

I go into detail on why to use bash over sh in this post: On the Shebang

Side note on syntax

In shell scripts you usually use tests in if or while clauses. These are tedious to write in the interactive shell. The ‘and’ operator && will execute the following statement only if the preceding statement returns 0 (success). So you can use && to write simple if … then … clauses in a single line.

if [ -d Documents ]
then
    echo "found docs"
fi

and

[ -d Documents ] && echo "found docs"

have the same effect. The second is much shorter, but as soon as the test or the command gets more complex you should revert to the longer syntax.

Alternatively, the ‘or’ operator || will only execute the following statement when the previous statement returns non-zero or fails:

[ -d Documents ] || echo "no docs"

is the same as

if [ ! -d Documents ]
then
    echo "no docs"
fi

What’s wrong with the single brackets?

The single bracket [ is actually a command. It has the same functionality as the test command, except that the last argument needs to be the closing square bracket ]

$ [ -d Documents && echo "found docs"
-bash: [: missing `]'
~ $ [ -d Documents ] && echo "found docs"
found docs
$ test -d Documents  && echo "found docs"
found docs

Note: in bash on macoS both test and [ are built-in commands, but as usual for built-in commands there are also executables /bin/test and /bin/[.

A single bracket test will fail when one of its arguments is empty and gets substituted to nothing:

$ a="abc"
$ b="xyz"
$ [ $a = $b ] || echo "unequal"
unequal
$ unset a
$ [ $a = $b ] || echo "unequal"
-bash: [: =: unary operator expected
unequal

You can prevent this error by quoting the variables (always a prudent solution).

$ [ "$a" = "$b" ] || echo "unequal"
unequal

Double brackets in bash are not a command but a part of the language syntax. This means they can react more tolerantly to ‘disappearing’ arguments:

$ [[ $a = $b ]] || echo "unequal"
unequal

You will also get an error if one of the arguments is substituted with a value with whitespace with single brackets, while double brackets can deal with this.

$ a="a"
$ b="a space"
$ [ $a = $b ] || echo "unequal"
-bash: [: too many arguments
unequal
$ [[ $a = $b ]] || echo "unequal"
unequal

Note: the = operator in sh and bash is for string comparison. To compare numerical values you need to use the -eq (equals), -ne (not equals), -gt (greater than), -ge (greater than or equal), -lt (less than), -le (less than or equal) operators. With double brackets you can also use two equals characters == for a more C like syntax. (or, better, use ((…)) syntax for arithmetic expressions)

Also, when using the = to assign variables, you cannot have spaces before and after the =, while the spaces are required for the comparison operator (both with single and double brackets):

a="a"           # no spaces
b="b"           # no spaces
[ "$a" = "$b" ] # spaces!
[[ $a = $b ]]   # spaces!

Since the single bracket is a command, many characters it uses for its arguments need to be escaped to work properly:

$ [ ( "$a" = "$b" ) -o ( "$a" = "$c" ) ]
-bash: syntax error near unexpected token `"$a"'
$ [ \( "$a" = "$b" \) -o \( "$a" = "$c" \) ]

You could alternatively split this example into two tests: [ "$a" = "$b" ] || [ "$a" = "$c" ].

Double brackets interpret these characters properly. You can also use the (again more C like) && and || operators instead of -a and -o.

 [[ ( $a = $b ) || ( $a = $c ) ]]

In general, you can work around most of the issues with single bracket syntax, but the double bracket syntax is more straight forward and hence more legible and easier to type.

Double bracket features

Aside from the cleaner syntax, there are a few ‘bonus’ features you gain with double brackets.

With double brackets you can compare to * and ? wildcards, and bracket globbing […]:

$ a="Documents"
$ [[ $a = D* ]] && echo match
match
$ a=hat
$ [[ $a = ?at ]] && echo match
match
$ [[ $a = [chrp]at ]] && echo match
match

You can also use < and > to compare strings lexicographically:

$ a=cat
$ b=hat
$ [[ $a < $b ]] && echo sorted
sorted

And you get an operator =~ for regular expressions:

$ a=cat
$ b="the cat in the hat"
$ [[ $a =~ ^.at ]] && echo match
match
$ [[ $b =~ ^.at ]] && echo match

Note that you should not quote the globbing patterns or the regex pattern.

Summary

  • you should use bash for shell scripting on macOS
  • when using bash, you should use double brackets instead of single brackets
  • double brackets are safer, easier to type and read, and also add few neat features

On the macOS Version

In many administration scripts, you need to check the version of macOS, to make sure you don’t try to access features that are not present on other versions.

Getting the Version

On macOS you can get details on the version of the currently running system with the sw_vers command:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.13.1
BuildVersion:   17B48

You can add the -productVersion argument to get just the product version number:

$ sw_vers -productVersion
10.13.1

The first part of the version number (“major” version) is always 10 (so far). The second part (“minor” version) denotes the version of macOS (11: El Capitan, 12: Sierra, 13: High Sierra, etc.) and the third (“patch” or update version) the update number. (Useful list of macOS versions and names on Wikipedia.)

Note: because the first part of the version for macOS is always 10, some call the second number the major version of macOS and the third the minor version. It does not really matter as long as you are explicit what you mean and remain consistent. Documentation in code helps avoid confusion.

You can test for a specific version with string comparison:

if [[ $(sw_vers -productName) == "10.12.6" ]]; then
    echo "Sierra"
fi

This will get tedious when you need to check for many patch versions. The [[ operator let’s you use the asterisk * as a wildcard in string comparisons:

os_ver=$(sw_vers -productVersion)
if [[ "$os_ver" == 10.13.* ]]; then
    echo "macOS High Sierra"
elif [[ "$os_ver" == 10.12.* ]]; then
    echo "macOS Sierra"
else
    echo "(Mac) OS X something"
fi

But even that can get tedious when you have a range of minor versions. Often you want to check for the minor version to be higher than a certain number (i.e. “Mavericks and later”)

Splitting the Version

You could split the version into its parts with awk:

echo "minor version:" $(sw_vers -productVersion | awk -F. '{ print $2; }')
echo "patch version:" $(sw_vers -productVersion | awk -F. '{ print $3; }')

This works well, but is a bit unwieldly.

You can also split the version string into a bash array:

os_ver=$(sw_vers -productVersion)

IFS='.' read -r -a ver <<< "$os_ver"

echo "minor version: ${ver[1]}"
echo "patch version: ${ver[2]}"

The read command splits the string into a bash array with the periods as a separator. There is a lot going in this command, so let’s have a look at the pieces.

First we assign the ‘.’ to the IFS environment variable. Characters in the IFS (“Internal Field Separator”) variable are used by bash to split strings into their pieces. The normal IFS variable consists of the whitespace characters: space, tab and newline. However, we want to change this so that the read command splits at the periods in our version string. This syntax with the variable assignment directly followed by the command without a separator tells bash to change the IFS variable just for the next command. This way the standard IFS variable is not affected.

The read command splits the input into an array and assigns it to the ver variable. Then we can get the elements in the ver array using the ${ver[index]} notation. (bash arrays are zero-indexed, so the first element ${ver[0]} will always be 10.

Once you have split out the parts of the version string, you can then use them for numerical comparisons:

if [[ "${ver[1]}" -ge 9 ]]; then
    echo "somewhere in California"
elif [[ "${ver[1]}" -ge 2 ]]; then
    echo "officially a feline"
else
    echo "secretly a feline"
fi

Get the macOS version from other Volumes

The sw_vers command only show the version for the currently booted system. System administrators often need to know the version of the OS installed on a different volume. Usually this happens in the context of installations. In installation scripts the installer system will pass the path to the target volume as the third argument $3.

Learn all about installer packages and installation scripts, read my book “Packaging for Apple Administrators

On macOS the system version information is also stored in a property list file in /System/Library/CoreServices/SystemVersion.plist. In an installation script (postinstall or preinstall you can get the target system with:

/usr/libexec/PlistBuddy -c "print :ProductVersion" "$3"/System/Library/CoreServices/SystemVersion.plist

Learn all about PlistBuddy and Property Lists in my book “Property Lists, Preferences and Profiles for Apple Administrators

macOS Installer applications have the version of macOS that will be installed in yet a different file. Tim Sutton has already documented this here.

Other Application’s versions

Obviously, you can use this approach on other version numbers as long as they follow the scheme of “numbers separated by dots.” (Not all applications do.)

To get the version of a macOS application you can read the CFBundleShortVersionString from the Info.plist file in the app bundle:

/usr/libexec/PlistBuddy -c "print :CFBundleShortVersionString" /Applications/Xcode.app/Contents/Info.plist
9.1

Some applications do not have a CFBundleShortVersionString and then you should fall back to the CFBundleVersion key.

infoPath="/Applications/Xcode.app/Contents/Info.plist"
appVersion=$(/usr/libexec/PlistBuddy -c "print :CFBundleShortVersionString" "$infoPath")
if [[ -z "$appVersion" ]]; then
    appVersion=$(/usr/libexec/PlistBuddy -c "print :CFBundleVersion" "$infoPath")
fi

Build Number

Note: update this section, because I got few things wrong. Thanks to the participants of this Slack discussion.

Aside from the numerical version number or product version, macOS also has a build number, which follows a different schema. The first part is a number which matches the “Darwin version” of macOS. Mac OS X 10.0 had a Darwin version of 4 and that number has increased with every version of Mac OS X. Currently, macOS High Sierra 10.13 has a Darwin version of 17. Then follows a single capital letter, with A being the first release of a version (i.e the 10.x.0 version), B is the first patch update (10.x.1) and so on.).

Finally the last number is the build number. The build number usually incremented during Apple internal development. You can see the build number increasing during beta releases.

However, sometimes Apple will release hardware specific versions of macOS which usually have four digit build numbers (usually starting with a 2). Also security updates and other “unexpected updates” can change the build number (usually to four digits starting with a 1) without changing the numerical version number.

Sometimes there might be an a appended to the build number. This means that there was an update to the installer, but not the code inside the installer. This frequently (but not exclusively) happens during the beta phase.

Thanks to Elliot Jordan we have a great info graphic!

Because of this it can also be useful to parse the build number out of the build version. Since the first two parts of the build version are directly mapped to the numerical version, we are only interested in the number:

build_ver=$(sw_vers -buildVersion)
if [[ "${ver[1]}" -le 5 ]]; then # 10.5 was darwin 9, 10.6 darwin 10
    build_number="${build_ver:3}" # ignore first two characters
else
    build_number="${build_ver:4}" # ignore first three characters
fi

if [[ ${build_number: -1} == 'a' ]]; then 
    build_number="${build_number:0:$((${#build_number}-1))}"
fi

echo "build number: $build_number"

You can get all this code in a sample script I posted on gist.

On the Shebang

Every script you want to run from the command line should have a shebang as the first line.

Note: I talked about this in my MacSysAdmin talk. I wanted to go into more detail here.

You can have scripts without the shebang. In this case the shell that you launch the script from will be used to interpret the script. This can, obviously, lead to complications, on other systems or with other users.

A shebang looks like this:

#!/bin/bash

The eponymous first two characters of the shebang are #! (hashbang or shebang). These two characters form a ‘magic number.’ Files are usually identified by certain codes (i.e. magic numbers) in the first few bytes of data. The hex code 23 21 converts to the ascii characters #! and tells the system that a file is script.

After the shebang comes the command that should interpret the script (the interpreter). This is either a shell, as in #!/bin/sh or #!/bin/bash or another interpreter, for example #!/usr/bin/python or #!/usr/bin/osascript. Generally, any command that can interpret text files or streams can be used in the shebang.

Since the status or value of the PATH variable is not guaranteed or certain in many contexts that scripts can be run in, the path after the shebang needs to be absolute.

The env shebang

There are environments where you cannot predict the absolute path of a given tool. For example the bash v3.2 shell on macOS is installed by default in /bin/bash. Users can also download and install bash version 4.4 onto their computers. The location for the the bash 4 binary is usually at /usr/local/bin/bash (this might be different depending on the installation method you used). Since /usr/local/bin is the first item of the default PATH on macOS the newer bash 4 will be chosen before the built-in bash 3.2 when the user types bash into their shell.

When you use the absoute path to /bin/bash in a shebang, you are ensuring that the macOS provided built-in version of bash will be used. For macOS system administrators, this should be the preferred shebang, since it provides a known, certain environment.

However, there are cases where you want scripts to be run with the user’s preferred tool, rather than a set path. You may also want your script to be able to on multiple different unix (and unix-like) systems where the location of the bash binary can be unpredictable. In this case you can use the /usr/bin/env tool as the shebang with bash (or another interpreter) as the parameter:

#!/usr/bin/env bash

This shebang means: “determine the preferred bash tool in the user’s environment and use that to interpret the script.”

Note: obviously this also presumes the absolute path to the /usr/bin/env. However, most unix and unix-like system seem to agree here.

Shebang Arguments

The shebang line can take a single argument. For weird syntactic reasons the entire line after the interpreter path is passed as a single argument to the interpreter tool.

So a shebang line that looks like this:

#!/bin/bash -x

will be executed like this:

/bin/bash "-x" /path/to/script

But if you added another variable to the shebang:

#!/bin/bash -x -e

then it would be executed as

/bin/bash/ "-x -e" /path/to/script

where "-x -e" is a single argument, making bash choke.

Some tools (like env) that are regularly used in shebangs are able to split the single argument. bash is not one of them. Read a tool’s man page and documentation and test. In general it is considered bad style to add arguments or options to the shebang.

Note: with bash you can also set these options in the script with the set command:

set -x
set -e

bash versus sh

Many Unix and unix-like systems have sh as well as bash and other shells available. sh goes back to the very early UNIX shells in the seventies. sh has survived because it serves as the lowest common standard for shell scripting. These standards are defined in the POSIX specification, though POSIX defines more than just shell scripting.

When you have to build scripts that need to run across many different flavors and versions of Unix and Linux, where you cannot rely on bash being present, then conforming to POSIX and sh might be necessary.

Note: bash is also POSIX compliant, but it has more features. When you script against sh you ensure you don’t use any of those additional features.

However, as macOS administrators, we can rely on a larger ‘common’ set. bash has been present on macOS since Mac OS X 10.0 (and earlier incarnations in NeXTSTEP). bash gives you several extra features over plain sh such as better testing with the double square bracket, ‘here documents’, parameter substitution and arrays.

As a macOS system administrator you should leverage these and always choose /bin/bash over /bin/sh.

bash versus Other Shells

The argument that bash is better than sh works for other shells as well. You can make a very good argument that other shells are better than bash, or at the very least bash version 3 included in macOS. I often hear this with zsh, so I will be using zsh as an example, but the arguments work with other shells as well.

(You can find a very good comparison of different shells here.)

zsh and other shells have many features that bash 3 (and even 4) lacks. There are two main reasons I would still recommend bash for scripting:

  • most script examples and shared code is written in bash so you need to be proficient in bash anyway (and know the quirks of bash)
  • the zsh is not included on the macOS Recovery System or other macOS installation environments, so you cannot write post-installation scripts for these contexts in zsh

When you do reach the limitations of bash for scripting, other languages such as Python and Swift are even more useful and powerful on macOS than zsh, so the step up from bash should not be another shell, but a different scripting language entirely

Interactive shell versus shell scripting

While I argue that that other shells don’t have that much edge on bash for scripting, they certainly do have an edge on bash in some of their interactive and customization features.

Two of the more interesting interactive shells for macOS are zsh and fish.

The good news here is that you can set one shell for interactive use, and still continue using and writing bash scripts and get the best of both worlds.

The default interactive shell is set in your user record, you can change this in the ‘Advanced Options’ in the Users & Groups preference pane, or with chsh command:

$ chsh -s /bin/zsh
Changing shell for armin.
Password for armin: 

The shell that interprets a script is, of course, set with the shebang.

Summary

  • the first line in a shell script starts with a shebang #! and tells the system which tool interprets the script
  • env should only be used in the shebang when the script needs to run in environments where you cannot predict the location of the actual interpreter. For admins this introduces an extra level of uncertainty
  • macOS administrators should use /bin/bash as the shebang
  • the ‘step up’ from bash scripting should be a more complex language with native macOS API access such as Python or Swift
  • you can use one shell for interactive Terminal work and another for scripting

On Distribution Packages

Distribution packages are a special format of installer packages. Distribution packages can contain one or more normal or component packages. They can also contain extra resources to customize and control the user interface in the Installer application.

In most cases administrators prefer component packages since they are easier to create and maintain. However, there are a few cases where distribution packages are necessary:

  • add a package to a custom installation in NetInstall, AutoDMG or a system installer created with createOSXinstallPkg
  • combine multiple component pkgs into a single installer
  • restrict hardware and system requirements
  • modify the interface presented in Installer.app
  • push the package with MDM’s InstallApplication command

Building Distribution Packages

You can easily convert an existing component package, built with pkgbuild to a distribution package with the productbuild command:

$ productbuild --package component.pkg dist.pkg

You can also combine multiple components into a single distribution package:

$ productbuild --package A.pkg --package B.pkg combined.pkg

You can add the --sign option to the productbuild command when the distribution package needs to be signed:

$ productbuild --sign "Installer: Armin" --package component.pkg dist.pkg

You can find valid identities with

$ security find-identity -p basic -v

The string you pass with the --sign parameter can be a partial match to the full identity name.

Note: munkipkg has a flag to build a distribitution package instead of a component package.

Extracting Component Installers from Distribution Packages

Sometimes you may want to extract a component installer pkg from a distribution package.

First you need to expand the distribution pkg with pkgutil:

$ pkgutil --expand dist.pkg dist_expanded

When you use the --expand option on a distribution package, components will also be expanded into subfolders that end in .pkg. Because of this Finder will erroneously display them as installer bundle files. This is misleading, since the components are not functional in this form.

When you want to use the component package without any modifications, you can quickly recompress or ‘flatten’ the expanded component:

$ pkgutil --flatten dist_expanded/component.pkg component.pkg

The process of expanding and flattening a component will of course remove any signature the original pkg might have had. You can re-sign the flattened package with productsign:

$ productsign --sign "Installer: Armin" component.pkg component_signed.pkg

Note: Obviously, when you are tearing a distribution package apart you need to know what you are doing. Components in a distribution package may depend on other components or on scripts and tools in other components. As always: test, test, test.

Packaging Book

You can learn more on building installer packages in my book: “Packaging for Apple Administrators”

Terminal Primer – Part 6 – Symbolic Links

Symbolic Links

When you get a detailed list of the file system root / you will see a few interesting entries: (output abbreviated)

$ ls -l /
lrwxr-xr-x@    1 root  wheel    etc -> private/etc
lrwxr-xr-x@    1 root  wheel    tmp -> private/tmp
lrwxr-xr-x@    1 root  wheel    var -> private/var

These are symbolic link files. Symbolic links are identified by the first character l on a long ls. Symbolic links are redirections to other files or directories. /etc, /tmp and /var are standard directories for certain kinds of files in Unix systems, but in this case these directories are actually located in /private.

Note: The reason for this is historical and goes back to NeXTStep. This setup could allow the /private folder to be mounted from another disk or file share separate from the rest of the system. This is not used in macOS anymore, but the directory structure remains.

Aside from the long ls you can use the readlink command to determine where a symbolic links points to:

$ readlink /etc
private/etc

or the stat -l command:

$ stat -l /etc
lrwxr-xr-x 1 root wheel 11 Nov 17 07:50:53 2016 /etc -> private/etc

A symbolic link contains a path pointing to the original file or directory. Most operations such as reading or changing are directed to the original, rather than the symbolic link. ls /tmp and ls /private/tmp will show you both the contents of the original /private/tmp.

An example for a symbolic link to a file is the file /usr/share/dict/words (a list of english words, which can be surprisingly useful to have around) which points to a file web2 in the same directory. Symbolic links can be used as a means to ‘switch’ between files without having to change around filenames and configuration files.

When you read the file /usr/share/dict/words the read command will be redirected to /usr/share/dict/web2:

$ cat /usr/share/dict/words | wc 
  235886  235886 2493109
$ cat /usr/share/dict/web2 | wc 
  235886  235886 2493109

Note: the wc command counts words, lines and bytes in a file.

Symbolic links can be relative or absolute. However, most of the time they are relative, since you do not want them pointing to different files, depending on which volumes the system is booted from. Relative paths are resolved relative to the link itself, not the current working directory. The above link for /etc points to the relative path private/etc so to the sub-directory etc in the directory private in the same location as the symbolic link itself.

To create a symbolic link use the ln -s command.

$ mkdir directory
$ touch directory/file
$ ln -s directory/file
$ readlink file
directory/file

This series of commands will create a directory, and en empty file in directory and then a link to that file in the current working directory. When you want the symbolic link to have a different name, you can give that as a second argument:

$ ln -s directory/file second_link
$ readlink second_link 
directory/file

The first argument is not really the path to a file or directory, but the path the symbolic link points to. When this path is relative it will be resolved relative to the location of the symbolic link. So if you wrote:

$ ln -s directory/file directory/link_to_file

A symbolic link named link_to_file will be created in directory but it will point to directory/directory/file instead of the file next to it. When you try to read from the symbolic link pointing to nowhere, you will get an error:

$ cat directory/link_to_file 
cat: directory/link_to_file: No such file or directory

The reference you pass when creating the symbolic link has to be relative to the where the symbolic link is created. The correct command would have been:

$ ln -s file directory/link_to_file

When the second argument is a directory itself, a link named the same as the file it refers to will be created:

$ mkdir another_dir
$ ln -s ../directory/file another_dir

will create a link named file pointing to ../directory/file. Since you have to give the path to the target relative to the where the link is created you have to add the ../ to ‘go up a level’ out of the another_dir directory and then back into directory.

Note: you can create a symbolic link that points to non-exiting path. Also when the original file or directory gets deleted, a ‘dangling’ symbolic link will remain.

When you use rm or mv on a symbolic link, only the link will be affected, not the original item.

When you run cp on a symbolic link to a file, the contents of the original will be copied:

$ cp another_dir/file filecopy
 $ stat -l filecopy
-rw-r--r-- 1 armin staff 0 Sep  4 14:51:44 2017 filecopy

However, when you recursively copy a directory tree containing symbolic links, they will be copied as symbolic links:

$ cp -R another_dir/ copy_dir
$ ls -l copy_dir/
total 8
lrw-r--r--  1 armin  staff  17 Sep  4 14:52 file -> ../directory/file

Usually the destination of a copy will be in a vastly different location and this will break the links:

$ cp -R another_dir/ /Users/Shared/copy_dir
$ ls -l /Users/Shared/copy_dir
total 8
lrwxr-xr-x  1 armin  wheel  17 Sep  4 15:19 file -> ../directory/file
$ cat /Users/Shared/copy_dir/file 
cat: /Users/Shared/copy_dir/file: No such file or directory

You can use cp‘s -L option to make the copy process resolve symbolic links and copy their contents instead:

$ cp -RL another_dir/ /Users/Shared/another_dir
$ ls -l /Users/Shared/another_dir
total 8
-rwxr-xr-x  1 armin  wheel  17 Sep  4 15:19 file

Other commands will have similar options to control the behavior when encountering symbolic links. However, their names are not standardized in anyway and you will have to consult the man page to find out the details.

Symbolic Links vs Finder Aliases

In macOS Finder, you can create aliases with the menu item ‘Make Alias’ from the ‘File’ or context menu. Finder Aliases have much the same role as symbolic links, but a few significant differences in behavior:

  • the shell and most command line tools cannot resolve Finder Aliases and treat them as files
  • Finder Aliases will ‘follow’ the original when it is moved or renamed.
  • However, when the original is deleted and replaced by an item of the same name, a Finder Alias will resolve to the new item.
  • When a Finder alias points to an item on a file share, double clicking the alias in Finder will attempt to connect to the file share, if it not already connected.

Finder will display Aliases and symbolic links with a small arrow in the corner of the icon. Both symbolic links and Finder Aliases have a ‘Show Original’ menu item in the ‘File’ menu or context menu.

There are no commands to create or resolve Finder aliases in Terminal, but you can use AppleScript with the osascript command:

$ osascript -e 'tell app "Finder" to make new alias to posix file "/Users/armin/Documents" at posix file "/Users/armin/Desktop"'

This will create a new Finder alias to ‘Documents’ in the ‘Desktop’ folder. The user who is running this command has to be logged in to the Mac, so that osascript can connect to the Finder to run this command.

To find out the original of a Finder alias, you can use:

$ osascript -e 'tell application "Finder" to get POSIX path of ( (original item of (POSIX file "/Users/armin/Desktop/Documents" as alias) ) as alias)'

Note: there is a lot of type casting (as alias) in this command. To further confuse matters the data type alias in AppleScript is not the same as a Finder alias. It is a special data type in AppleScript that references a file or folder.

Parse Binary Property Lists in Finder Metadata

For more info on plutil and everything property list related read my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’

macOS and Finder use extended attributes to store plenty of extra information about files. For example, when you download a file in Safari, it stores when a file was downloaded and which website and download URL was used.

As an example, I downloaded the latest Firefox disk image. When you look at the downloaded file in the Terminal, you see an @ after the file mode which tells us this file has additional extended attributes:

$ ls -l ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg

We can further inspect the extended attributes with the -@ option:

$ ls -l@ ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg
    com.apple.metadata:kMDItemDownloadedDate          53 
    com.apple.metadata:kMDItemWhereFroms         203 
    com.apple.quarantine          57 

This shows the three extended attributes attached to this file, their names and their sizes in bytes.

When you double-click the dmg file to mount it, the system will store the checksum and file system check result in more extended attributes:

$ ls -l@ ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg
    com.apple.diskimages.fsck         20 
    com.apple.diskimages.recentcksum          81 
    com.apple.metadata:kMDItemDownloadedDate          53 
    com.apple.metadata:kMDItemWhereFroms         203 
    com.apple.quarantine          57 

To inspect the contents of the extended attributes in further detail, we have to use the xattr command: xattr -l filename will show all extended attributes, or you can use xattr -pl attributename filename to get just a particular one:

$ xattr -pl com.apple.quarantine ~/Downloads/Firefox\ 55.0.3.dmg 
com.apple.quarantine: 0083;59a6b982;Safari;E56EFC36-29AB-4F77-89E6-F4264336060F

The contents of the quarantine attribute is a string with some numbers (presumably a hash) the application that downloaded it (Safari) and a UUID.

The downloaded date however, looks a lot different:

$ xattr -pl com.apple.metadata:kMDItemDownloadedDate ~/Downloads/Firefox\ 55.0.3.dmg 
com.apple.metadata:kMDItemDownloadedDate:
00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 BF 56 F1 02  |bplist00..3A.V..|
00000010  53 AF D9 08 0A 00 00 00 00 00 00 01 01 00 00 00  |S...............|
00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 13                                   |.....|
00000035

This attribute is some binary data (xattr automatically uses binary representation when it detects a nil byte. The first few bytes in the binary data are bplist which tells us we are working with binary plist format. Unfortunately the rest of the binary plist data is quite unreadable for humans.

We can use xxd to convert the binary representation into actual data and plutil to print the plist:

$ xattr -px com.apple.metadata:kMDItemDownloadedDate ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -p -
[
  0 => 2017-08-30 13:11:30 +0000
]

Note that the options for xattr changed from -pl to -px which forces the output to be binary data only.

And the same command for the ‘WhereFroms’:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -p -
[
  0 => "https://download-installer.cdn.mozilla.net/pub/firefox/releases/55.0.3/mac/en-US/Firefox%2055.0.3.dmg"
  1 => "https://www.mozilla.org/en-US/firefox/new/?scene=2"
]

This uses plutil’s -p option to just print the data in a human readable form. You can also have plutil convert the plist data into XML:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -convert xml1 -o - -
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
    <string>https://download-installer.cdn.mozilla.net/pub/firefox/releases/55.0.3/mac/en-US/Firefox%2055.0.3.dmg</string>
    <string>https://www.mozilla.org/en-US/firefox/new/?scene=2</string>
</array>
</plist>

or JSON:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Download.0.3.dmg | xxd -r -p | plutil -convert json -r -o - -
[
  "https:\/\/download-installer.cdn.mozilla.net\/pub\/firefox\/releases\/55.0.3\/mac\/en-US\/Firefox%2055.0.3.dmg",
  "https:\/\/www.mozilla.org\/en-US\/firefox\/new\/?scene=2"
]

For more info on plutil and everything property list related read my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’

Terminal Primer – Part 5 – Managing Files

Managing Files

We already know how to navigate and read the file system with cd and ls.
Now we want to actually do something to the files.

Create an Empty File

Sometimes it can be useful to quickly create an empty file. You can use the touch command to do this.

$ touch emptyfile

If you use the touch command on a file that already exists it will update the file’s modification date and change nothing else.
macOS (and other Unix-like operation systems) sometimes uses the existence of a file in a certain directory as a flag for configuration.
For example, when a file named .hushlogin exists at the root of a user’s home directory, the ‘Last Login: …’ message when you open a new shell (terminal window) is suppressed.

$ touch ~/.hushlogin

will create this file and subsequent new Terminal windows will not show this message. To return to showing the message, you will have to delete the file.

Deleting Files

To delete a file use the rm (remove) command:

$ rm document.txt

You have to use special care with the rm command.

In the Finder, deleted files are moved to the Trash, which is actually the invisible directory ~/.Trash. There the file will remain until the user chooses ‘Empty Trash’. Only then is the file removed from disk.
The command line has no such safety net.

When you delete a file (or directory) with the rm command it is gone. You have to be especially careful with files with special characters. If a filename has a space in it, and you run the rm command without escaping or quotes, then you will get an error or even worse, might delete the wrong file.

For example:

$ rm My Important Document.txt

Will delete the three files My, Important, and Document.txt, if they exist. If they do not exist it will show errors.

Use escape sequences or quotation marks to protect from spaces and other special characters in and directory names:

$ rm 'My Important Document.txt'

Tab-completion will also protect from improperly typed or escaped file names. If the tab-completion will not work, even though you believe you have the right file or path then something went awry and you have to step back and verify your working directory and paths.

To delete the .hushlogin file we created above, you use rm

$ rm ~/.hushlogin

Once the file is removed, new terminal windows will show the ’Last Login: …” message again.

You can add the -i option to the rm command which will ask for confirmation before actually deleting a file.

$ rm -i ~/.hushlogin 
remove /Users/armin/.hushlogin? y

Creating Directories

To create a new empty directory (or folder) you use the mkdir command.

$ mkdir scratchspace

you can give the mkdir command multiple arguments when you want or need to create multiple directories at once

$ mkdir folder1 folder2 folder3

When you create a nested directories, all the directories in between already have to exist:

$ mkdir LevelA
$ mkdir LevelA/LevelB/LevelC
mkdir: LevelA/LevelB: No such file or directory 

When you need to create nested directory hierarchies like this, you can use mkdir’s -p option:

$ mkdir -p LevelA/LevelB/LevelC

This will create all three folders at once, if they do not already exist.

Moving and Renaming

You can move a file or directory using the mv command. The mv command needs two arguments: the source and the destination.

$ touch testfile
$ mkdir testdir
$ mv testfile testdir
$ ls testdir
testfile

This mv command reads as ‘move the file testfile to the directory testdir’. To move it back to the current working directory you can use the ‘.’ short cut.

$ mv testdir/testfile .

The mv command can also rename a file:

$ mv testfile samplefile

Moving and renaming is considered the same in the shell.

Warning: When a file already exists in the destination, mv will mercilessly overwrite the destination file. There is no way to retrieve an overwritten file.

You have to take care to type the proper paths in the shell. It is a very unforgiving environment.
To make mv a bit safer add the -i option which will prompt to confirm when it will overwrite a file:

Warning: when you use mv to move between volumes, the source file will be removed after it is moved to the destination. This is different from the behavior in Finder, where the default drag action between volumes is copy.

Filename Extensions

On macOS and other operating systems it is common to denote the file type with an extension. The extension is a standard alphanumeric code separated from the rest of the file’s name by a dot or period. E.g. .txt, .pdf or .mobileconfig.

In bash, the filename extension is part of the filename. There is no special treatment for the extension.
On macOS, however, Finder usually hides the file extension from the user. You can control the display of the file extension in the ‘Advanced’ tab of Finder Preferences. You can also control this setting for each individual file in its Info panel.

Finder will also warn when you attempt to change the file extension, since it might change which application is used to open a file. (You can also disable this in the Finder Preferences.) bash has no such warning mechanism.

$ mv hello.txt hello.md

Note: A feature specific to macOS is that some directories will have filename extensions and Finder will display them as if they were files, not folders. These folders are called ‘packages’ or ‘bundles.’ The most common example are applications with the .app extension.
Packages and bundles are used to hide complex file and data structures from users. Another example is the ‘Photos Library’ (or ‘iPhotos Library’ on older systems) which hides a big and complex folder structure.
You can choose ‘Show Package Contents’ from the context menu in Finder to drill down further into the internal structure of a package or bundle.
bash and other shells are not really aware of packages or bundles and treat them like normal directories.
You can read more detail on Bundles and Packages on the Apple Developer Page.
Note to the Note: the name ‘packages’ is also used for package installer files (with the .pkg extension). These are different uses of the same word.

Copying

The command to copy is cp. It follows similar syntax as the mv command:

$ cp source destination

So you can copy the samplefile we created earlier:

$ cp samplefile newsamplefile

You can also copy to a directory:

$ cp samplefile testdir

Warning: the cp command will mercilessly overwrite existing files with the same name!

When you run cp again it will overwrite the existing copy:

$ cp samplefile testdir

As with the rm command overwritten files are lost. There is no way to retrieve overwritten files.

The -i option shows a prompt to confirm whenever a file will be overwritten:

$ cp -i samplefile testdir
overwrite testdir/samplefile? (y/n [n]) n
not overwritten

When you try to copy a directory, you will get an error message:

$ cp testdir newdir
cp: testdir is a directory (not copied).

Since the command to copy a file or directory would look exactly the same, cp expects an extra option to be certain you know what you are doing. The -R option (for recursive) will tell cp to recursively copy all files and sub-directories (and their contents) of a folder.

$ cp -R testdir newdir

This will create a copy of testdir and all its contents with the name newdir.

Warning: the option for recursive copying is -R (uppercase R). There is a legacy option -r (lowercase r) which seems to do the same thing. However, there is a difference in behavior mentioned in the cp man page:

Historic versions of the `cp` utility had a `-r` option.  This implementation supports that option; however, its use is strongly discouraged, as it does not correctly copy special files, symbolic links, or fifo's.
If the destination directory already exists the way you write the path of the source directory will influence the behavior of cp.

When the path to the source does not end with a /, cp will create a copy of the directory in the destination directory:

$ mkdir dirA
$ cp -R testdir dirA
$ ls dirA
testdir

When the path of the source directory ends with a /, cp will copy all the contents of the source directory to the destination folder:

$ ls testdir
samplefile
$ mkdir dirB
$ cp -R testdir/ dirB
$ ls dirB
samplefile

Warning: when you use tab-completion to complete paths to directories the / is always appended! You will need to consider whether you want to keep the trailing / or not.

You can add more source arguments to a cp command, the last argument will be the destination:

$ cp -R samplefile otherfile hello.txt bigfolder

Wildcards (Globbing)

Note: In early versions of Unix wildcard substitution was the responsibility of a program called glob, short for ‘global command.’ Because of this the action of replacing wildcards with actual paths and filenames was and still is called globbing.

When you have to address or manage many files at once, it can be slow, tedious and ineffective to address each file individually. bash provides wildcard characters to make that easier.

There are two commonly used wildcard characters: * and ?

The asterisk or star * will match zero or more characters. It can be placed anywhere in a path.

The question mark ? will match any character, but there has to be a character.

By default filenames that start with a period ‘.’ are not matched, unless you specifically start the string with a dot .*

It is important to keep in mind that bash will build a list of filenames that match the wildcards and substitute this list in place of the name with the wildcard(s) before executing the command.

When you enter

$ cd ~
$ ls D*

The D* will be replaced with the list of filenames that match (Desktop Documents Downloads) and then executed:

$ ls Desktop Documents Downloads

This can lead to some unforeseen consequences. For example say you are in a folder with some files and directories:

$ ls -F
dirA/  dirB/  dirC/  file1  file2  file3

And you run

$ cp file? dir?

The wildcards will be expanded to

$ cp file1 file2 file3 dirA dirB dirC

Which means that the three files as well as dirA and dirB will be copied into dirC, since treats the last argument as the destination and all previous arguments will be copied.

You can use wildcards in paths, so /Users/*/ will expand into all directories in the /Users folders.

However, /Users/*/Desktop will expand into a list of all users’ Desktop folders. Note that the first list contains /Users/Shared while the second does not contain /Users/Shared/Desktop, because that directory does not exist!

Warning: Wildcards can be extremely useful, but also very dangerous. They have to be handled with utmost caution, especially with potentially destructive commands such as mv, cp, and rm.

You can always test the result of wildcard expansion with the echo command:

$ echo /Users/*/
/Users/Guest/ /Users/Shared/ /Users/armin/
$ echo /Users/*/Desktop
/Users/Guest/Desktop /Users/armin/Desktop

You can also hit the escape key twice and bash will show the expansion, if there are any:

$ echo /Users/*<esc><esc>
Guest/  Shared/ armin/

Finally, bash has a third globbing or wildcard character, but it is a bit more complex. You can provide a list of possible characters between square brackets.

[bcr]at

will match

bat, cat or rat, but not Bat, Cat or Rat

Since shell commands are case-sensitive, you may have to provide both cases, if you want to match:

[bBcCrR]at

No matter how many characters are in the square brackets, they will match to exactly one character:

[bB][aei]t matches bat, Bat, bet, Bet, bit, or Bit

Deleting Directories

We have been creating and copying a lot of files. It is time to clean up. We already know the rm command to remove files. However, when you try to use rm to delete a directory you get:

$ rm newdir
rm: newdir: is a directory

There is a command rmdir which is the destructive equivalent of mkdir. However, rmdir can only remove empty directories:

$ rmdir newdir
rmdir: newdir: Directory not empty

You can use the * wildcard to delete all files in newdir:

$ rm newdir/*

Note: the * wildcard will not expand to filenames starting with a period. You may have to explicitly delete dot files as well:

$ touch newdir/.dotfile
$ rm newdir/*
$ rmdir newdir
rmdir: newdir: Directory not empty
$ rm newdir/.*
rm: "." and ".." may not be removed
$ rmdir newdir

This will work as long there are only files or empty directories in newdir. When the directory you want to delete contains an entire hierarchy of files and directories, then this approach will be cumbersome.

For this the rm command has the -R option which will recursively delete all contents and subdirectories.

$ rm -R testdir

Since there is no way to recover a file deleted by rm you should always use this command with care, especially when using the -R option.

You can add the -i option when using -R as well, but then you will be prompted for every single file and subdirectory, which can be very tedious and counter-productive.

Note: unlike the cp command, the -r and -R option for the rm command are synonyms. However, for consistency’s sake and to build muscle memory. I would recommend making a habit of using the -R syntax for both commands.

Terminal Primer – Part 4 – Commands

Arguments

So far we have use three commands: pwd, cd, and ls

These commands are already quite different.

pwd is a single word command. You enter it and it prints information (the working directory to the terminal).

cd, however, requires additional information from you: where do you want to change to? The cd command requires an argument:

$ cd ~/Documents

(You can enter cd without an argument, and it will change to your home directory, but usually you want an argument.)

The command itself cd and the argument ~/Documents are separated by a space.

Some commands can have more than one argument. In that case all arguments are separated from each other by a space. (Or more. bash doesn’t care about multiple spaces.)

This is why we have to treat spaces in paths and filenames so carefully, because otherwise the shell might interpret the path as two or more arguments.

Finally ls has an optional argument. When you just write ls. it will list the contents of the current working directory. When you give an argument it will list the contents of that path. The ls command also has several options that modify its behavior.

When a shell command is written in documentation optional arguments are usually enclosed in square brackets:

ls [-options] [path]

Mandatory arguments, on the other hand, are shown without the square brackets.
When you enter an ls command with completely wrong options (surprisingly difficult, since its options cover nearly the entire alphabet, and some extra characters as well.) it will print a “usage” line:

$ ls --a
ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

The extra ... after the optional file command tells us, that you can give ls more than one path argument:

$ ls ~/Desktop ~/Documents

Read the Manual

When you want detailed information on a command, there are a few approaches.

Because of the long and diverse history of shells, bash and macOS in particular, not all commands support all of these options. Behavior here can be very inconsistent.

First, as we just saw with ls, some commands will print a brief usage note, when you enter something that the command cannot parse.

With some commands you can provoke the usage message with the -h or --help option:

$ sw_vers -h
Usage: sw_vers [-productName|-productVersion|-buildVersion]

The usage message is commonly very brief and usually does not explain all the options.

To get more detailed in information on command you can read its man page. man pages are documentation, often very detailed, stored in an file format optimized for display in ASCII terminals.

To get the man page for a command run the man command:

$ man ls

This will take over the current Terminal window and display the information.

This special display mode is actually controlled by another command called less. There many key commands you can use for navigation in this mode.

Key
q exit and return to command line prompt
up/down arrow scroll up/down by a line
space or z scroll down by a page
w scroll up a page
g top of document
G (shift-g) end of document
/word<return> find next occurrence of word in document
n find next occurrence of search term
N find previous occurrence of search term
h help

You can also scroll in this mode with the mouse wheel or two-finger scrolling on a trackpad.

You can also open man pages in terminal from the Help menu. When you enter a shell command in the help search field of Terminal it will suggest a man page, when one is available. When you select a suggested man page, it will open in a new yellow window.

You can modify the appearance of the man page window by changing the ‘Man Page’ window profile in Terminal’s Preferences.

You can also open a man page by selecting text and choosing ’Open man page from the context menu.

Some commands are ‘built-in’ to the bash shell. These do not always have man pages. Requesting the man page for a built-in command will show the man page for builtin instead.

cd is one example for a built-in command.

You can get documentation for built-in commands with

$ command help cd

Finding commands

We just learned that some commands, like cd, are ‘built-in’ to the shell. Others are not, so what and where are they?

All commands are files in the file system. They have a special file privilege set which makes them executable. Obviously, you cannot make any file executable, it has to have some form of code which makes sense so the system can interpret it as commands.

If you want to know where a given command resides in the file you can use the which command

$ which ls
/bin/ls
$ which sw_vers
/usr/bin/sw_vers

However, you do not have to type /bin/ls every time you want to execute ls. How does the shell know where to look?

The shell has an environment variable called PATH which contains a list of directories where it will look for commands that are typed without an absolute path. You can print the contents of this variable with the echo command:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Note: commands and variable names in the shell are case-sensitive. It is convention that environment variables are written in all-caps. You have to use the correct case for the PATH variable to get or set the proper value.

When you are new to shell and bash, there is a lot to process in this simple command, so let’s take this apart piece by piece:

The echo command simply tells the shell to print something to the terminal, so

$ echo hello
hello

prints ‘hello’ back to the terminal. This alone is rarely useful, but can be used to get at the results of another process.

$ echo $(( 6 * 7 ))
42

The $(( … )) means ‘evaluate this arithmetically,’ so this command prints the result of this arithmetic to the terminal.

In general in bash the $ stands for ‘substitute contents.’ echo $PATH means: print the contents of the PATH variable.

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

When you forget the $ and just write

$ echo PATH
PATH

bash will interpret PATH as a literal string and prints it to the terminal.

The actual contents of the PATH variable is a list of directories separated by colons.

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The order of the directories in the PATH is important as the shell will stop looking when it finds a command.

When you enter a command without a path, e.g. ls, bash will start looking for the command executable in /usr/local/bin, then in /usr/bin, and then in /bin, where it will find an executable ls, stop looking and execute that.

Note: if there were another executable named ls in a later directories it would not be used, since the shell will stop looking at the first match it finds. Changing the order of the standard directories in the PATH or even inserting other directories earlier in the PATH can lead to unexpected behavior.

The PATH on your system may be different when you have extra software installed. Xcode, Server.app, Xquartz, munki, Python3 and many other software packages insert paths to their command directories in the search path.

Note: some software solutions will attempt to modify the PATH on a system to make their commands available to the shell, other will place the commands or links to the commands in /usr/local/bin to make them available (e.g. text editors like BBEdit or Atom).

We will look at strategies to on how and why to modify the search path later.

Some third party solutions will instruct you to modify the PATH to include their commands rather than doing it during the installation.

Running Other Commands

When you need to execute a command or script that is not in the PATH, you have to type the full or relative path to the command:

$ /usr/libexec/PlistBuddy
Usage: PlistBuddy [-cxh] <file.plist>

or

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport --getinfo

These are commands that are usually considered too uncommon or maybe even dangerous to put in the standard search paths.

When you start using and writing custom-built scripts and commands, you can use relative paths:

$ test/my_great_script.sh

or

$ scripts/postinstall

When you need to execute a command or script in the current working directory, you have to start the command with ./, so the shell knows to not look in the search path.

$ ./buildMyProject.sh

Remember the . is a shortcut representing the current working directory.

Tab-completion for Commands

You can use tab-completion for commands as well. This will speed up your typing and prevent typing errors.

You can use this to get a list of all the commands available in the shell. At an empty command prompt hit the tab-key twice. Then shell will warn you that there are many completions (more than a thousand, depending on your version and configuration of macOS.

You can also use this command to list all tab-completions:

$ compgen -c

Note: compgen is the command that bash runs to determine which commands are available for tab-completion. You usually would not interface with it directly.