Moving to zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

I will be giving a half-day ‘Moving to zsh’ class at our offices in Amsterdam on September 6! Visit the website for details!

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

zsh (I believe it is pronounced zee-shell, though zish is fun to say) will succeed bash as the default shell. bash has been the default shell since Mac OS X 10.3 Panther.

Why?

The bash binary bundled with macOS has been stuck on version 3.2 for a long time now. bash v4 was released in 2009 and bash v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3. bash v3 is still GPL v2.

zsh, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default. zsh has been available as on macOS for a long time. The zsh version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current zsh 5.7.1.

Is bash gone!?

No.

macOS Catalina still has the same /bin/bash (version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.

Many scripts in macOS, management systems, and Apple and third party installers rely on /bin/bash. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.

Users that have /bin/bash as their default shell on Catalina will see a prompt at the start of each Terminal session stating that zsh is now the recommended default shell. If you want to continue using /bin/bash, you can supress this message by setting an environment variable in your .bash_profile or .bashrc.

export BASH_SILENCE_DEPRECATION_WARNING=1

You can also download and install a newer version of bash yourself. Keep in mind that custom bash installations reside in a different directory, usually /usr/local/bin/bash.

Will bash remain indefinitely?

Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to bash, but didn’t really care if you remained on tcsh. In fact, tcsh is still present on macOS.

Apple’s messaging should tell us, that the days of /bin/bash are numbered. Probably not very soon, but eventually keeping a more than ten year old version of bash on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.

Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’ dash has been added to the lineup of shells. dash is designed to be a minimal implementation of the Posix standard shell sh. So far, in macOS (including Catalina),sh invokes bash in sh-compatibility mode.

As Apple’s support article mentions, Catalina also adds a new mechanism for users and admins to change which shell handles sh invocations. MacAdmins or users can change the symbolic link stored in /var/select/sh to point to a shell other than /bin/bash. This changes which shell interprets scripts the #!/bin/sh shebang or scripts invoked with sh -c. Changing the interpreter for sh should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.

All of these changes are indicators that Apple is preparing to remove /bin/bash at some, yet indeterminate, time in the future.

Do I need to wait for Catalina to switch to zsh?

No, zsh is available Mojave and on older macOS versions. You can start testing zsh or even switch your default shell already.

If you want to just see how zsh works, you can just open Terminal and type zsh:

$ zsh
MacBook%

The main change you will see is that the prompt looks different. zsh uses the % character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in bash.

If you want to already switch your default shell to zsh you can use the chsh command:

$ chsh -s /bin/zsh

This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.

How is zsh different?

Like bash (‘Bourne again shell’ ), zshderives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.

The main difference between bash and zsh is configuration. Since zsh ignores the bash configuration files (.bash_profile or .bashrc) you cannot simply copy customized bash settings over to zsh. zsh has much more options and points to change zsh configuration and behavior. There is an entire eco-system of configuration tools and themes called oh-my-zsh which is very popular.

zsh also offers better configuration for auto-completion which is far easier than in bash.

I am planning a separate post, describing how to transfer (and translate) your configurations from bash to zsh.

What about scripting?

Since zsh has been present on macOS for a long time, you could start moving your scripts from bash to zsh right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to #!/bin/zsh.

You will gain some features where zsh is superior to bash v3, such as arrays and associative arrays (dictionaries).

There is one exception where I would now recommend to use /bin/sh for your scripts: the Recovery system does not contain the /bin/zsh shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider older macOS installations where zsh is definitely not present in Recovery.

When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as Twocanoes MDS, installr or bootstrappr, then you cannot rely on /bin/zsh.

Since we now know that bash is eventually going away, the only common choice left is /bin/sh.

When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using /bin/sh as the shebang from now on.

I used to recommend using /bin/bash for everything MacAdmin related. /bin/sh is definitely a step down in fucntionality, but it seems like the safest choice for continued support.

Summary

Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’ bash with zsh, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!

This git repo has been shared by many on MacAdmins Slack: rothgar/mastering-zsh, I will certainly dive into that and share about my experiences here!

Next

In the next part we will look at the configuration files for zsh.

Show Exit Code in your bash Prompt

I prefer a minimal bash prompt. Recently, however, I saw an oh-my-zsh prompt, that I thought would be useful.

The intriguing prompt displayed a symbol indicating whether the previous command exited successful (exit code zero) or failed (non-zero exit code).

You can always get the exit code of the previous command from the $? variable, but seeing it right there and in color, is more direct.

While I find fish andzsh quite intriguing, I am still unwilling to move my setup for just a single feature. “This has to be possible in bash,” I thought… And it is, though the implementation was a bit more complex than I expected. But I learned a lot more about how the bash prompt worked.

The trick for changeable or dynamic prompts in bash is to create a bash function that assembles the PS1 variable, on every prompt. You can enable that function by setting the PROMPT_COMMAND environment variable to your custom function.

Obviously, you should not overload your function with time intensive processes, but with modern processing power, a lot can be done in a short time.

After a lot of experimentation, I settled on this setup:

Update: there was an error in the code that would prevent the prompt from ever showing a red exit code. I fixed it now, the change is in the last line. (Thanks to co-worker Mattias for pointing that out.)

You can add this code to your .bash_profile or .bashrc. (If you do not know what that means, read this post.)

I experimented with special characters and even Emoji to signify the exit code, but then settled on colors and the square root symbol (option-V on the US and international keyboard, looks like a checkmark) for success and the question mark ? with the exit code for errors.

Obviously, you can use a modified prompt command to show all kinds of other statuses as well. Enjoy!

On Smart Quotes and Terminal

Typography is a wonderful art and has a long history. When humans turned from manual typesetting to machines, type writers and then computers, some compromises had to be made. One of these compromises was to use simple straight quote symbols for opening and closing the quote, rather than different quote symbols for opening and closing.

Note: which kind of quotes are used for opening and closing dependent on the language or and some convention. English uses upper quotes “…”, German opens with a lower quote: „…“, French uses ‘guillemets:’ «…», and Japanese uses hooks: 「…」

Quotation Marks have funny names in many languages. Germans call them “Gänsefüßchen,” or “little goose feet.”

See this Wikipedia article for more details.

macOS, iOS and other modern operating systems have a feature which replaces the simple or straight quote symbols with the typographic quotes. So, you type "Hello!" and the quotes are automatically replaced with the proper (depending on localization) typographic quotes. This is called “smart quotes.”

This is pretty nice, but can be troublesome when dealing with Terminal and text editors. Scripting languages and shells always use straight quotes, and cannot deal with typographic quotes.

Now, if someone sends you a command or a script that uses quotes, and it goes through an app that replaces them with smart quotes, then bash and Terminal will fail miserably.

There is not much you can do, other than be aware of this and check pasted code carefully. There is something you can do to make this easier, though.

The default monospace font used in Terminal on macOS are ‘Menlo’ or ‘SF Mono,’ depending on the macOS version. Now these are fine typefaces, but their typographic quotes are not very curly at all, making them very hard to distinguish from the ‘dumb’ straight quotes that Terminal expects. The classic ‘Monaco’ typeface on the other hand has beautiful curly typographic quotes, making them very distinct from the the straight quote.

My favorite mono space typeface ‘Source Code Pro’ also has nice curly typographic quotes. I have built this table with many common monospace typefaces and their quotes.

Quotation Mark Comparison
Quotation Mark Comparison

Now this shouldn’t be your only criteria in choosing your Terminal font, but it may be something that helps avoid quote errors.

Swift 5 for MacAdmins

macOS 10.14.4 also includes Swift 5. The main new feature of Swift 5 is that Swift is ABI stable.

Simply said, ABI stability allows swift binaries to use a Swift library on the system instead of having to bundle the libraries with them. This will, of course reduce the size of the binaries.

For example, my desktoppr tool compiles to 6.5MB with Swift 4.2 and 56KB with Swift 5.

However, command line tools built with Xcode 10.2 now rely on the Swift library to be available on the system. macOS 10.14.4 and future versions will include the libraries, but older macOS versions did not. There is no option in Xcode 10.2 to keep the old behavior of bundling the libraries in the tool.

This means that when you re-build a tool in Xcode 10.2 with Swift it will not run on older macOS version:

$ sw_vers -productVersion
10.13.6
$ ./desktoppr 
dyld: Library not loaded: @rpath/libswiftAppKit.dylib
  Referenced from: /Users/armin/Desktop/desktoppr
  Reason: image not found
Abort trap: 6

Apple provides an installer for the Swift libraries for “earlier versions of macOS.” (The package installer declares a minimum OS version of 10.9.)

The libraries are installed in /usr/lib/swift, where your binaries can find them:

$ ./desktoppr 
/Library/Desktop Pictures/BoringBlueDesktop.png

Note: these libraries are used by compiled binaries. Installing the runtime libriaries will not allow you to run swift script files, i.e. text files starting with the #!/usr/bin/swift shebang. You still need to install Xcode or the Developer Command Line tools for that.

So far, only command line tools written and compiled in Xcode 10.2 will require the libraries. Application bundles will continue to include their own libraries.

As Swift gets updated, you will need to update the installed libraries as well. You can get the installed version of the Runtime with pkgutil:

$ pkgutil --info com.apple.pkg.SwiftRuntimeForCommandLineTools
package-id: com.apple.pkg.SwiftRuntimeForCommandLineTools
version: 10.2.0.0.1.1552586384
volume: /
location: /
install-time: 1553789052
groups: com.apple.FindSystemFiles.pkg-group 

However, on a Mac with 10.14.4 the swift libraries are present but not installed by the same installer package, they are part of the entire system:

$ pkgutil --file-info /usr/lib/swift/libswiftFoundation.dylib
volume: /
path: /usr/lib/swift/libswiftFoundation.dylib

pkgid: com.apple.pkg.update.os.10.14.4.18E226
pkg-version: 1.0.0.0.1.1553145155
install-time: 1553765876
uid: 0
gid: 0
mode: 755

If/When Apple updates the runtime libraries, this might be a challenge to track and update properly.

Note: While I used my tool desktoppr as an example, I have not updated the version available for download to Swift 5 yet. So, that is still built with the “old” Swift and Xcode and should work everywhere without the runtime libraries.

Nevertheless, it should be prudent for MacAdmins to install the Swift 5 runtime libraries on their fleet. At the very least be aware that these errors can occur and how to fix them.

Parsing dscl Output in Scripts

On macOS dscl is a very useful to access data in the local user directory or another directory the Mac is bound to. For example you can read a user’s UID with:

$ dscl /Search read /Users/armin UniqueID
UniqueID: 501

This output looks easy enough to parse, you can just use cut or awk:

$ dscl /Search read /Users/armin UniqueID | cut -d ' ' -f 2
501
$ dscl /Search read /Users/armin UniqueID | awk '{print $2;}'
501

However, dscl is a treacherous. Its output format changes, depending on the contents of an attribute. When an attribute value contains whitespace, the format of the output has two lines:

$ dscl /Search read /Users/armin RealName
RealName:
 Armin Briegel

With attributes like the UID, it is fairly safe safe to assume that there will be no whitespace in the value. With other attributes, such as RealName or NFSHomeDirectory, you cannot make that prediction with certainty. Real names may or may not have been entered with a space. A user (or management script) may have changed their home directory to something starting with /Volumes/User HD/... and your script may fail.

To remove this output ambiguity, dscl has a -plist option which will print the output as a property list:

 $ dscl -plist . read /Users/armin RealName
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>dsAttrTypeStandard:RealName</key>
    <array>
        <string>Armin Briegel</string>
    </array>
</dict>
</plist>

The resulting property list is a dict containing a key with the native attribute name and an array containing the values, even when there is only one value.

Having a property list is nice, but parsing property lists in a shell script is challenging. I have found two solutions

Xpath

You can use the xpath tool extract data from the XML output:

$ dscl -plist . read /Users/armin RealName | xpath "//string[1]/text()" 2>/dev/null
Armin Briegel

Note that the xpath output does not include a final new line character, which makes it look a bit strange.

The xpath argument in detail means:

  • //string[1]: the first of any string element
  • /text() the text contents of that stringobject

This syntax makes a lot of assumptions about the property list input. I believe they are safe with the dscl output. (Please test)

If you want to play around with xpath syntax, I recommend using an interactive tool. I used this one from Code Beautify which worked well enough, but frankly I just randomly chose one from the list of search results for ‘xpath tester’. (If you can recommend a great one, let us know in the comments.)

PlistBuddy

As I said, the xpath solution makes a lot of assumptions about the layout of the property list. A safer way of parsing property lists would be a dedicated tool, such as PlistBuddy. However, PlistBuddy does not read from stdin. At least not voluntarily.

A few weeks ago Erik Berglund shared this trick on Mac Admins Slack which makes PlistBuddy read the output from another command. We can adapt this for our use case:

$ /usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/armin RealName)
Armin Briegel

Note that you have to escape the : in the attribute name, since PlistBuddy uses the colon as a path separator.

You can use this in scripts to assign the value to a variable with

realName=$(/usr/libexec/PlistBuddy -c "print :dsAttrTypeStandard\:RealName:0" /dev/stdin <<< $(dscl -plist . read /Users/$username RealName))

This uses nested command substitution with the $(... $(...) ...) syntax which is not possible using backticks.

Either way, you can get a safe value from dscl in shell script, whether it contains whitespace or not.

Demystifying `root` on macOS, Part 4 —The Authorization Database

Beyond the Shell

sudo allows you to gain super user privileges from the interactive shell. launchd, installer Packages, management systems and other tools will run scripts as root when required. This covers many cases where administrators need to influence the system on macOS.

However, for most users, macOS is exclusively the graphical interface. Users can authorize to perform certain tasks when they are administrators, like unlocking a Preference Pane or running an installer package.

These tasks are not controlled by sudo but by a separate mechanism. The data for that mechanism is stored in the authorization database.

Here there be Dragons!

As mentioned in earlier parts of this series, many pieces of macOS assume users have administrator accounts. This might not be a possible or useful configuration in your environment. But you still need to provide access to some privileges without giving a user full administrator accounts and access.

However, if you find yourself frequently editing the authorization database, you should re-evaluate your approach. Future macOS updates might change privileges or rules and then your heaviliy modified setup will need to be adpated. Small modifications or granting users admin privileges will be the the least fragile configuration going forward with future macOS releases.

What is it?

The active authorization database lives in sqlite3 database files in /var/db/auth.db.

You can use the sqlite3 command or another tool to read the database directly.

$ sudo sqlite3 /var/db/auth.db .dump

You can use this database access to change the settings directly. However, this is not recommended. The only way sanctioned by Apple to access and change the authorization database is through the security authorization command.

This layer of abstraction allows Apple to change the underlying data store, while keeping tools and frameworks that access the data same.

The authorization database is intialized from the property list file /System/Library/Security/authorization.plist. In older versions of macOS administrators could replace this file with a modified version and delete the database files to have the modified property list initialize the database with the new settings. However, in current versions of macOS this file is protected by SIP, so this strategy is no longer useful.

However, the authorization.plist file is still useful to look at the default values and get an idea of how the authorization database is configured and works. Since there is quite a lot of data in this file, it is best to open it in a graphical property list editor such as Xcode or PlistEdit Pro.

The authorization property list consists of two main dictionaries: rights and rules. There are also comment fields distributed all through the file, which provide some context to what the individual elements are for.

rights designate a certain context that permits a certain action, group of actions or access to configure some part of the os. The names of the rights follow a hierarchy and are denoted in reverse DNS notation.

This website has an overview of all the rights and rules from the authorization.plist. More importantly it shows which rights and rules are available in which version of macOS.

How it works

Within the dictionary for a given right you can find the requirements that a user needs to fulfill to gain the right. There are two main classes: user and rule.

(There is also a third possible value for the class: evaluate-mechanisms will test multiple mechanisms in order. This is used for more complex processes such as the login window.)

The user class will verify whether the user asking to gain the right is in a particular user group. Usually this is the admin group. for example you can inspect the configuration of the system.preferences.datetime right with:

$ security authorizationdb read system.preferences.datetime
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>allow-root</key>
    <true/>
    <key>authenticate-user</key>
    <true/>
    <key>class</key>
    <string>user</string>
    <key>comment</key>
    <string>Checked by the Admin framework when making changes to the Date &amp; Time preference pane.</string>
    <key>created</key>
    <real>522083506.22018802</real>
    <key>group</key>
    <string>admin</string>
    <key>modified</key>
    <real>522083506.22018802</real>
    <key>session-owner</key>
    <false/>
    <key>shared</key>
    <false/>
    <key>timeout</key>
    <integer>2147483647</integer>
    <key>tries</key>
    <integer>10000</integer>
    <key>version</key>
    <integer>1</integer>
</dict>
</plist>
YES (0)

(The YES (0) at the end of the command’s output means that retrieving the data was successful.)

You can also look for the system.preferences.datetime entry in the authorization.plist. This tells us that when a user clicks on the lock in the ‘Date & Time’ preference pane the system will ask for authentication (authenticate-user is true) to check if the user a member of the admin group.

Read the com.apple.configurationprofiles.userprofile.trustcert right to see an example for a rule based right:

$ security authorizationdb read com.apple.configurationprofiles.userprofile.trustcert
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>class</key>
    <string>rule</string>
    <key>comment</key>
    <string>Install user configuration profile with certificate requiring trust change.</string>
    <key>created</key>
    <real>529239529.937994</real>
    <key>modified</key>
    <real>529239529.937994</real>
    <key>rule</key>
    <array>
        <string>authenticate-session-owner-or-admin</string>
    </array>
    <key>version</key>
    <integer>0</integer>
</dict>
</plist>
YES (0)

Rather than defining the approving criteria in the right itself, this right references the authenticate-session-owner-or-admin rule. This name is already quite self-explanatory. However, we can also read the definition of the rule:

$ security authorizationdb read authenticate-session-owner-or-admin
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>allow-root</key>
    <false/>
    <key>authenticate-user</key>
    <true/>
    <key>class</key>
    <string>user</string>
    <key>comment</key>
    <string>Authenticate either as the owner or as an administrator.</string>
    <key>created</key>
    <real>522083506.22018802</real>
    <key>group</key>
    <string>admin</string>
    <key>modified</key>
    <real>522083506.22018802</real>
    <key>session-owner</key>
    <true/>
    <key>shared</key>
    <false/>
    <key>timeout</key>
    <integer>2147483647</integer>
    <key>tries</key>
    <integer>10000</integer>
    <key>version</key>
    <integer>0</integer>
</dict>
</plist>
YES (0)

Requests to grant a right are logged. When you are unsure what the name of right is you can perform/unlock the right or task in the interface and then read the log with:

$ sudo log show --style syslog --predicate 'subsystem == "com.apple.Authorization" && eventMessage CONTAINS[c] "validating credential"' --last 1h

This will list the rights (and rules) that were requested in the last hour.

(Thanks to Erik Berglund on the MacAdmins Slack for showing me how to do this.)

Changing Behavior

You can use the security authorizationdb command to change rights and rules in the authorization database.

WARNING: you can really mess up your system and make it unusable with the wrong settings in the authorization database. You should not test these commands on your (or anybody else’s) work machine. You should do your experimentation and testing on a virtual machine, where you can quickly revert the system to a known working state. You should only use these commands on production Macs when they have been thoroughly tested.

You need super user privileges to change the authorization db. I will use sudo for the interactive examples.

The easiest way to change behavior is to change the right to a different preset rule. The most permissive rule is allow which allow any user the right and not even prompt for authorization.

When you change the ‘Date & Time’ right with:

$ sudo security authorizationdb write system.preferences.datetime allow
Password:
YES (0)

And then open the ‘Date & Time’ preference pane with any user. You will see the pane is unlocked by default.

This may be just a bit too permissive. A more useful rule to use is the authenticate-session-owner-or-admin rule, which will prompt for authentication, and accept the currently logged in user (session-owner) or any admin. This provides for some security so that other people cannot walk up and change the current user’s settings.

$ sudo security authorizationdb write system.preferences.datetime authenticate-session-owner-or-admin
YES (0)

This will allow the any user to unlock the preference pane in their own session. Note that the dialog to unlock does not prefill the user’s name and the text in the dialog still asks for an ‘administrator’s name and password’. So this may be confusing for users. Proper documentation for the user’s affected will help.

Another use case for this rule is to allow administrator account to unlock a locked screen in another user’s session (particularly useful in lab or classroom settings).

$ sudo security authorizationdb write system.login.screensaver authenticate-session-owner-or-admin
Password:
YES (0)

Now, any admin user can unlock any other user’s locked screen. This will of course drop them in that user’s login session so the admin’s have to be responsible with this privilege.

When you take a look through the rules, you will notice that some start with authenticate- and other with is-. The difference is that the authenticate- rules will prompt for user name and password, while the is- rules will grant the privilege when the user satisfies the rule with out a prompt.

There are also some rules whose names end in -nonshared. These have the shared key set to false. This means that these rules will not share credentials with other requests for authentication. Shared credentials will not prompt multiple times for the same credentials in a certain time (the timeout, usually 300 seconds/5 minutes). More sensitive settings are usually not shared.

Exporting and importing privileges

Changing the rule is a straightforward and fairly safe way to change a right’s behavior. When a rule fits your requirements, you should probably use a rule.

However, sometimes you need more fine-grained control of a right. For this you can export a right to a property list file, modify this and re-import it.

For example, say you do not want to grant every user access to a certain right but just a certain group of users even though they are not administrators. Common examples for this would be developers, teachers or lab techs. So, in our example, we create a group named techs:

$ sudo dseditgroup -o create -n . -r "Lab Techs" techs

to be safe we will nest the admin group in the techs group so that members of admin also gain all the rights of techs:

$ sudo dseditgroup -o edit -a admin -t group techs

and add the example user beth

$ sudo dseditgroup -o edit -a beth techs

You can verify that everything worked with:

$ sudo dseditgroup -o read techs
dsAttrTypeStandard:GeneratedUID -
        F9EBB33D-8A71-45EE-A65D-4DBBFF421B49
dsAttrTypeStandard:RecordName -
        techs
dsAttrTypeStandard:AppleMetaNodeLocation -
        /Local/Default
dsAttrTypeStandard:GroupMembers -
        87EE7744-5872-47C1-9574-D247D3DD4D5C
dsAttrTypeStandard:PrimaryGroupID -
        501
dsAttrTypeStandard:NestedGroups -
        ABCDEFAB-CDEF-ABCD-EFAB-CDEF00000050
dsAttrTypeStandard:RealName -
        Lab Techs
dsAttrTypeStandard:GroupMembership -
        beth
dsAttrTypeStandard:RecordType -
        dsRecTypeStandard:Groups

Now we can assign this group to a right. Our example will be the system.preferences.energysaver right. First export it to a property list file:

$ security authorizationdb read system.preferences.energysaver > energysaver.plist
YES (0)

Then change the value of the group key in the property list file to techs:

$ /usr/libexec/PlistBuddy -c "set group techs" energysaver.plist

If you want to learn more about working with property list files, I have written a book on it: ‘Property Lists, Preferences and Profiles for Apple Administrators’

And re-import the modified property list to the authorization database:

$ sudo security authorizationdb write system.preferences.energysaver < energysaver.plist

This alone is not sufficient to unlock the Energy Saver preference pane. When you look at the log with command from above, you can see that unlocking the energy saver pane requires the system.preferences right as well. You can modify this the same way:

$ security authorizationdb read system.preferences > system.preferences.plist
YES (0)
$ /usr/libexec/PlistBuddy -c "set group techs" system.preferences.plist
$ security authorizationdb write system.preferences < system.preferences.plist

Scripting

When scripting this setup you do not have to go through the export/import cycle but can directly import a prepared property list file or here doc:

Summary

There are many levels of access privileges in macOS. Most of them are controlled by different users and groups.

The super userroot exists on macOS like any other unix, but logging in as root is disabled by default as a security measure. The sudo command allows for interactive super user privileges in a shell. There are other ways administration scripts can be launched with super user privileges, such as LaunchDaemons, management systems etc.

The authorization database controls access to elevated rights in the macOS UI. You can modify it as an admin, but should do so with care.

Demystifying root on macOS: Conclusion

This ends my series on the super user in macOS. I hope it clarifies some confusing terms and configurations. As admins we have to use these privileges regularly. But, as the saying goes: “With great power comes great responsibility.”

Understanding how things work and how they affect the system will help you understand what to do when and how to avoid unexpected consequences.

Demystifying `root` on macOS, Part 3 — `root` and Scripting

sudo is very useful when working interactively in the shell. However, when you are scripting workflows then you don’t want to encounter the interactive prompt for authentication.

sudo in Scripts

You will write scripts that require root privileges to perform their tasks. Many novice scripters will simply add the sudo command to their scripts. For example:

#!/bin/bash

timeserver="time.apple.com"

echo "Setting Time Server to: $timeserver"

# note: sudo is superfluous
sudo systemsetup -setnetworktimeserver "$timeserver"
sudo systemsetup -setusingnetworktime on

When you invoke this script from the command line, it will actually work as expected, since the first sudo will prompt the password and the second sudo will use the cached credentials.

In many cases where the script already has root privileges, it will work also, because sudo when run as root will not prompt for authorization again.

However, when this script is sent with Remote Desktop or run in a different context without user interaction to authorize sudo, the script will stall and eventually fail.

In most contexts, the scripts should already be running with root privileges so sudo in the script is not necessary.

#!/bin/bash

timeserver="time.apple.com"

echo "Setting Time Server to: $timeserver"

systemsetup -setnetworktimeserver "$timeserver"
systemsetup -setusingnetworktime on

You should not use sudo in your management scripts. When commands inside a script require root privileges, you should invoke the entire script with sudo:

$ sudo ./settimeserver.sh

or you deploy and execute the script through other methods that provide root privileges. (Remote Desktop: run as user root, installation scripts, LaunchDaemons, management systems, etc.)

Testing for root Privileges in Scripts

When you write scripts that require root privileges, you may want to test whether it is actually running as root early in the script or exit gracefully with an error or warning. As usual with Unix, there are many ways to achieve this, but the recommended one is to check the EUID (effective user id) environment variable. For the root user the EUID is 0.

#!/bin/bash

# test if root
if [[ $EUID -ne 0 ]]; then
    >&2 echo "script requires super user privileges, exiting..."
    exit 1
fi

# continue with important things here
echo "I am root"

Running a Process as Another User

Most administrator scripts are run in a context where they run with super user privileges. Often they require root privileges.

On the other hand, when you need to affect settings or processes in a user’s context or login session, you may need to run commands as a specific user from a script running as root. It’s the opposite problem of gaining root privileges.

You can use sudo -u user command or su user -c command to run a command as a different user. However, the launchd man page warns us:

On Darwin platforms, a user environment includes a specific Mach boot strap subset, audit session and other characteristics not recognized by POSIX. Therefore, making the appropriate setuid(2) and setgid(2) system calls is not sufficient to completely assume the identity for a given user. Running a service as a launchd agent or a per-user XPC service is the only way to run a process with a complete identity of that user.

So it is safest use the Darwin/macOS native mechanism to launch a process as another user. Enter launchctl. You can use the the launchctl asuser verb to execute a script or command as a different user.

uid=$(id -u "$username")
launchctl asuser "$uid" /path/to/command arguments

The launchctl command takes a user’s numerical ID or UID rather than the short name as an argument. You can get a user’s UID with the id -u username command.

Note that launchctl asuser does not launch the command in the context of a new shell environment. You cannot rely on environment variables being set in that context. This is especially relevant for the PATH.

Note also that the asuser verb of launchctl is listed among the deprecated functions of launchctl. However, there is no replacement for the functionality yet.

Getting the Current User

Most of the time you will not know which user you need to run as, when you write a script, but need to run as the ‘currently logged in user’. There are many unix-y ways of determining the current user. However, once again there are edge cases in macOS where some of the traditional methods fail. (Mainly concerning Fast User Switching)

The ‘official’, method is to use the SystemConfiguration framework, and from a script this is easiest from within python:

loggedInUser=$(/usr/bin/python -c 'from SystemConfiguration import SCDynamicStoreCopyConsoleUser; import sys; username = (SCDynamicStoreCopyConsoleUser(None, None, None) or [None])[0]; username = [username,""][username in [u"loginwindow", None, u""]]; sys.stdout.write(username + "\n");')

(Enduring thanks to ‘macmule’ who documented this years ago.)

This will return the currently active user, even when multiple users are logged in with Fast User Switching. When no user is logged and the system is logging in the value returned will be "". Your scripts have to cover this case as well.

You can use this code snippet as a template:

# get the current user
loggedInUser=$(/usr/bin/python -c 'from SystemConfiguration import SCDynamicStoreCopyConsoleUser; import sys; username = (SCDynamicStoreCopyConsoleUser(None, None, None) or [None])[0]; username = [username,""][username in [u"loginwindow", None, u""]]; sys.stdout.write(username + "\n");')

# test if a user is logged in
if [[ $loggedInUser != "" ]]; then
    # get the uid
    uid=$(id -u "$loggedInUser")
    # do what you need to do
    launchctl asuser "$uid" /path/to/command arguments
fi

AppleScript

AppleScript has a special function when invoking shell commands for gaining super user privileges. When you run a shell command from AppleScript with the do shell script command you can add with administrator privileges to have the script prompt for admin user authentication and run the command as root:

do shell script "whoami"
do shell script "whoami" with administrator privileges

Will run like this:

    do shell script "whoami"
       --> "armin"
    do shell script "whoami" with administrator privileges
     --> "root"

Like with sudo the credentials for the first command run with administrator privileges will also be cached, so you will not get mulitple prompts, unless the script runs for a long time.

This is useful for providing tools and workflows from within AppleScript for interactive use. However, as with sudo you have to keep in mind that some contexts in which a script may be run does not allow for user interaction and then you have to find other means of elevating privileges.

AppleScript is often used to communicate with other applications and/or the user interface. However, when run with root privileges AppleScripts are often prohibited from connecting to other process or present user interface, such as dialogs. When you really need to do this, you have to use launchctl asuser to change the user the script is run as:

launchctl asuser "$uid" /usr/bin/osascript -e 'display dialog "Do you really want to do this?"

Beyond the Shell

sudo allows you to gain super user privileges from the interactive shell. LaunchDaemons, installer Packages, management systems and other tools will run scripts as root when required. This covers many cases where administrators need to influence the system on macOS.

However, for most users, macOS is exclusively the graphical interface. Users can authorize to perform certain tasks when they are administrator users, like unlocking a Preference Pane or running an installer package.

These tasks are not controlled by sudo but by a separate mechanism. The data for that mechanism is stored in the authorization database, which we will cover in the next post.

Demystifying `root` on macOS, Part 2 — The `sudo` Command

As mentioned before, the recommended way of gaining super user privileges from the command line in macOS is the sudo command. The name means ‘super user do’ and will perform the following command with root privileges after verifying the user running sudo has the permission to do so.

How sudo works

sudo allows a user to execute a command with super user privileges, without needing to authenticate as the super user. The user has to authenticate as themself, however, and the sudo will check whether the user is authorized to use sudo.

For example, you want to run a command that requires super user privileges:

$ systemsetup -getremotelogin
You need administrator access to run this tool... exiting!

You can then repeat the command with sudo to run it with temporary super user privileges.

$ sudo systemsetup -getremotelogin
Password:
Remote Login: On

On macOS, administator users are allowed to use sudo.

A few notes on sudo:

  • you can type sudo !! as a short cut for ‘repeat the last command with sudo’. More details on this in my post “On Bash History Substitution”.
  • the first time you run sudo with an account on a Mac it will show a longer dialog with a warning or ‘lecture’. (You can change the lecture.)
  • the system will prompt for your password when executing a command with sudo. However, there is a 5 minute (300 seconds) ‘grace period’ where the sudo system caches your credentials and you do not have to re-enter the password. This grace period is limited to a given shell session, so if you open a new Terminal window, you will have to re-enter the password.
  • on macOS sudo will not work if your administrator account’s password is empty: Using the sudo command in Terminal requires an administrator password
  • use of sudo is logged. You can find the log entries in the Console.app by searching for process:sudo:
armin : TTY=ttys000 ; PWD=/Users/armin ; USER=root ; COMMAND=/bin/echo "#iamroot"

sudo and the Environment

sudo runs the command in the current shell environment. The user (or effective user ID) the command is run as switches to root:

$ whoami
armin
$ sudo whoami
root

The sudo environment changes some of the variables, while some will be passed through from your shell. To show this, create a short script:

#!/bin/bash

echo $USER
echo $HOME
echo $EUID

and run it as yourself and then with sudo:

$ chmod +x printenvs.sh
$ ./printenvs.sh 
armin
/Users/armin
501
$ sudo ./printenvs.sh 
Password:
root
/Users/armin
0

Some tools might not read environment variables and determine the user environment through different means. This may lead to some tools writing data to root’s home directory instead of the current users when running with sudo.

The defaults command is one example:

$ sudo defaults write com.apple.loginwindow LoginHook /path/to/script
~ $ defaults read com.apple.loginwindow LoginHook
2018-04-16 15:09:00.378 defaults[69217:3291382] 
The domain/default pair of (com.apple.loginwindow, LoginHook) does not exist
~ $ sudo !!
sudo defaults read com.apple.loginwindow LoginHook
/path/to/script
~ $ sudo plutil -p /var/root/Library/Preferences/com.apple.loginwindow.plist
{
  "LoginHook" => "/path/to/script"
}

Note: This form of customizing login behavior in macOS is deprecated (but still works as of 10.13). LaunchAgents are the preferred method to run scripts or processes at login. (More info here.) If you find yourself building custom LaunchAgents and LaunchDaemons frequently, you need to check out outset.

You have to be aware that running commands with sudo results in a different environment than when you run them directly.

root Shells

In most cases running a single command with sudo is sufficient. However, sometimes it can be convenient to have an interactive shell that runs with super user privileges.

There are two ways of achieving this withsudo:

When you run sudo -s it will invoke a new shell, running as root. The shell that is run is the default shell of your account. So when you have bash set as your shell (the default on macOS) you will get a bash shell running as root. Most other environment settings will remain the same:

$ sudo -s
Password:
# whoami
root
# echo $HOME
/Users/armin
# echo $SHELL
/bin/bash
# exit
$ 

Usually, the Terminal prompt is set up to change from the $ prompt to # when you are running with super user privileges, to remind you of the power you have right now and the danger you are in.

Note: learn how to configure your shell prompt.

To leave the root shell, just type exit.

Alternatively you can use sudo -i to invoke a root shell. With the -i option, the shell will be chosen from the default shell set for root user (/bin/sh on macOS) and will be set up as if the root user were logging in, ignoring your settings, like profile files etc.

$ sudo -i
Mac:~ root# whoami
root
Mac:~ root# echo $HOME
/var/root
Mac:~ root# echo $SHELL
/bin/sh

Note that my custom minimal shell prompt changes when I switch to root with sudo -i since it creates the root shell with the root user’s environment.

In most cases sudo -s should serve well. However, when you want to avoid any customization you might have set in your user environment and work in more pristine environment then it is good to know sudo -i exists.

sudo vs su

There is a different command which allows you to change the user: su (short for ‘switch user’). The main difference between these tools is how they verify if you are authorized to switch.

su will ask for credentials of the user you are switching to. So if you run su bob you need to have Bob’s credentials.

When you run su without a username, it assumes root. But since logging in as root is disabled by default on macOS, it will fail.

$ su
Password:
su: Sorry

sudo, on the other hand, will check its configuration files to see if your account is authorized to run the command as the given user. It asks for your credentials to verify you. You do not need the credentials of the other user, whether it is root or a different user.

Since the root account login is usually disabled on macOS, you cannot use su root - or su - to get a root shell. Use sudo -s or sudo -i instead.

sudo and Scripting

sudo is very useful when working interactively in the shell. However, when you are scripting workflows then you don’t want to encounter the interactive prompt for authentication.

We will look at strategies for privilege escalation (and the opposite) in scripts in the next post.

Demystifying `root` on macOS, Part 1

As Mac Adminstrators, we often have to deal with user privileges for files and processes. While doing that we will use administrator privileges and sudo without as much as a second thought.

However, a proper understanding of what these privileges and processes actually do and mean, can help prevent many problems when managing Macs.

Some History

macOS is based on BSD Unix, which stems back to a time where large mainframes were so expensive they had to be shared among many users. Users and their access privileges control what any user can read, write, or change in the system. These rules prevent conflicts and data loss or theft. When managing these users and their access privileges, there had to be a first, ‘top’, or ‘super user’ which has access to anything.

In Unix and Unix-like systems this user account is traditionally called root. In macOS this user is often also called ‘System Administrator’.

Classic Mac OS, in contrast, had no concept of multiple users built-in to the system. Any person sitting down at a Mac (and any process launched on that Mac) could access and change anything on that system. There were some attempts at adding multi-user functionality to classic Mac OS, but they were ‘added on, not built-in’ and fairly easy to circumvent when a user knew what to do.

User and process management was one of the main benefits Apple touted for the various ‘next generation’ systems Apple introduced in the 90s to succeed classic Mac OS. When Apple bought NeXT and with it the NeXTStep operating system it inherited the unix model of doing so.

Even though the concept of sharing your computer is now relegated to some classroom labs and supercomputer clusters, this model still is present in every macOS and iOS device today. On iOS it is completly invisible to the user, unless a jailbreak is applied. On macOS, however, users and especially admins have to deal with it every day.

Users on macOS

To create a new user on macOS you have go to the ‘Users & Groups’ Preference Pane in System Preferences. Before you can add a new user, you have to unlock the preference pane by clicking the lock icon in the lower left corner. Then the system will prompt for an username and password with adminstrative privileges.

When the account you are logged in as has admin privileges, its name will be pre-filled. When the account is a standard user the username field will be empty and you can enter another user’s name and password.

Once the pane is unlocked, you can hit the ‘+’ icon under the user and will be offered four choices for a new user (from the popup menu next to ‘New Account’:

  • Administrator
  • Standard (the default)
  • Managed with Parental Controls
  • Sharing Only

There are three types of users not present in the popup list

  • Guest
  • System Administrator
  • system services users

The difference between Administrator and Standard accounts is that Administrator accounts are members of the ‘Administrators’ or admin group. This sound simple, but membership in this group bestows many additional benefits.

In day-to-day use Administrator accounts and Standard accounts behave the same. However, there are many situations and workflows on macOS which require authenticating as an Administrator account. As a general rule, a user can affect all the files (and applications) in their home directory and in /Users/Shared, but as soon as you want to change another user, another user’s files or settings that affect all users on a system you need to authenticate as an Administrator account.

The first user created on an unmanaged Mac out of the box will always be an Adminstrator user. Most Mac users use an Administrator account. Many of the workflows built-in to macOS assume an adminstrator account. One example is setting up a new printer.

With an Administrator account you can install third party software. You can also install malicious software. Often malicious software will trick users into installing by masquerading as or hiding in an installer for something useful.

Many consider it a ‘best practice’ to run your everyday work on your Mac with a standard account and only use an administrative account when you have to. However, since you get prompted to authenticate even with an administrative account, the better advice is to take these prompts very seriously and consider what confirming this prompt will really do or install.

The only difference you get when using a standard account is that you need to enter a different username and password in an authentication box instead of just the password. If this helps you pause and consider what you are actually doing, then great! Then this is the proper workflow for you.

However, I suspect that most users would be just as non-considerate of this dialog with a separate username and password as they would otherwise.

macOS Administrator Accounts

The only difference between Adminstrator accounts and Standard accounts is the membership of the admins group.

You can check whether a given user is a member of the admin group with the dseditgroup tool:

$ dseditgroup -o checkmember -u armin admin
yes armin is a member of admin

You can also use this tool to add or remove a user from the admin group:

$ dseditgroup -o edit -n . -a username -t user admin    # add username
$ dseditgroup -o edit -n . -d username -t user admin    # delete username

This membership comes with many privileges. Admin users can (after authentication):

  • unlock System Preferences and change system settings
  • install Apple and third party software and installer packages
  • create, change, and delete files owned by other users
  • change access privileges and ownership of files of folders in Finder
  • run and stop (kill) processes owned by other users
  • use sudo in Terminal

and many things more.

On macOS these privileges are controlled mainly by two mechanisms:

  • sudo and the sudoers file
  • the authorization database

sudo is used to gain root privileges in the shell (Terminal). The authorization database controls access privileges everywhere else.

What root can do

The ‘System Administrator’ or the root account controls the system. Mainly the root account can read, update, delete all local user accounts. It can control file and folder privileges and ownership. It can start system services running in the background and assign system network ports (with a port number lower than 1000). Most of this is managed by a process called launchd which is the first process to run on macOS.

Many commands require to be run as root or with elevated root privileges.

What root cannot do: System Integrity Protection (SIP)

On macOS, however, there are limits to what the root account can do. System Integrity Protection is a mechanism which protects important parts of the OS from mnodification, even with root permissions.

Only certain processes signed by Apple are allowed to modify these protected files and directories. Usually this means Apple signed installer pkgs for software and security updates.

Apple Support: About System Integrity Protection on your Mac

Apple lists a set of top-level directories that are protected. However, the list is a bit more detailed. You can use the -O (capital letter ‘O’, not a zero) to see if a file or directory is protected by SIP:

$ ls -lO /usr/
total 0
drwxr-xr-x  978 root  wheel  restricted 31296 Mar 30 18:21 bin
drwxr-xr-x  294 root  wheel  restricted  9408 Mar 30 18:21 lib
drwxr-xr-x  238 root  wheel  restricted  7616 Mar 30 18:21 libexec
drwxr-xr-x    8 root  wheel  sunlnk       256 Dec 28 13:48 local
drwxr-xr-x  248 root  wheel  restricted  7936 Mar 30 18:21 sbin
drwxr-xr-x   46 root  wheel  restricted  1472 Mar 30 18:21 share
drwxr-xr-x    5 root  wheel  restricted   160 Oct  3  2017 standalone

Files and Folders marked with restricted are protected by SIP. Sometimes folders inside a protected folder may not be protected, as the /usr/local/ directory in this example is.

SIP provides more protection than just certain parts of the file system, it also protects changing the boot volume and some other aspects of the OS.

While these limitations on even the root account can be annoying, they provide a level of security that parts of the OS have not been tampered with or changed by other software.

Enabling (and Disabling) root

On macOS the root account exists with a UID of ‘0’. However, it is set up so you cannot log in to a Mac as ‘System Administrator’ or root. (A terrible bug in early 10.13 provided a brief exeception to that rule.)

Note: login as root is disabled for security purposes. It is highly recommended that you leave the root account disabled on macOS and rely on sudo to gain temporary super user privileges when necessary.

If, for some reason, you do need to log in as root, then you can enable the root and provide it with a password. You can do so in either the ‘Directory Utility’ application. After unlocking with your administrator password, you choose ‘Enable Root User’ from the ‘Edit’ menu. You can also change the root account’s password here or disable it again later.

Apple Support: How to enable the root user on your Mac or change your root password

From the command line, you can also use the dsenableroot command:

$ dsenableroot

will enable and/or update the root account. It will interactively ask for admin credentials and for a new password for the root account. Read the command’s man page for details.

$ dsenableroot -d

will interactively disable the root account again.

Becoming root

Different environments and tools have different means of gaining super user or root privileges. While the sudo command should be the preferred means of gaining temporary super user privileges, it is important to know and understand the other options.

LaunchDaemons

Scripts and tools executed from LaunchDaemons run as root unless a different user is specified in the UserName key in launchd property list.

LaunchAgents on the other hand will be executed as the user logging in.

You can get details on how to set up and use LaunchDaemons here.

Run as root in ARD

When you prepare a ‘UNIX command’ to be sent to remote computers in Apple Remote Desktop, you have the option of running the command as the currently logged in user or as a specific user. When you specify root as the user, the script will execute with super user privileges. Since the ARD agent process runs as root on the client, no extra authentication or enabled root account is necessary.

Management Systems

The agent software of most management systems (Jamf, Munki, etc.) is installed to run with root privileges. Therefore, scripts executed by management systems run with root privileges as well.

Installation Scripts

Installation packages also perform their task with root privileges. They also require administrator authentication to start. Any installation scripts (pre-/postinstall scripts) will also run with root privileges.

set-UID bits

There is a special bit you can set on an executable’s mode (or privileges) which tells the system to run this script as the file owner, no matter who actually runs the executable. If the executable file is owned by root it will run with root privileges.

This flag is the “set-user-ID-on-execution bit”, also called the “Set-User-ID”-bit or just “s-bit”.

In the long ls format or with the stat command the set-user-ID bit is shown as an ‘s’ in place of the user’s x bit. One example is the ps command:

$ stat -l /bin/ps
-rwsr-xr-x 1 root wheel 51200 00:57 /bin/ps

Use chmod's u+s to set the set-user-ID bit and u-s to remove it:

$ chmod +x importantcommand [rwxr-xr-x]
$ chmod u+s importantcommand    [rwsr-xr-x]
$ chmod u-s importantcommand    [rwxr-xr-x]

Warning: Obviously it is very important that this executable is not modifiable by other users. They would be able to replace the command with their own code and execute anything with root privileges. Most system commands that have the s-bit set on macOS are protected with SIP.

The sudo command

As mentioned before, the recommended way of gaining super user privileges from the command line in macOS is the sudo command. The name means ‘super user do’ and will perform the following command with root privileges after verifying the user running sudo has the permission to do so.

We will look at the sudo command in detail in the next post.

Installing and Using Command Line Tools

There are many command line tools and scripts you can download and install that are very useful for Mac Admins.

(Just a representative list, certainly not complete.)

Some of these tools provide installer packages that deploy the tool in the proper directory – usually /usr/local/bin so that you and other users can use it. (/usr/local/bin is in the macOS default PATH.)

However, many of these tools, such as munkipkg or my own quickpkg just come as a git project folder, with no or few instructions to get it set up. The assumption is, that when you use these tools you are familiar enough with the shell to make them work.

There are actually several approaches to getting these tools to actually work for you, most with different upsides and downsides. This post will talk about a few of them.

Getting the Tool

Before you can choose a method to run the tool, you need to get it. Many admins share their scripts and tools through a hosted service like Github. My quickpkg tool, for example, is a python script hosted as an open Github repository.When you follow that link you will see the main project page. The page has a menu area up top, a file list in the middle and below an area where an introduction to the project (the ReadMe file) is shown. It is worth to read the ReadMe in case they have special installation instructions.

Download the Release

Git is a version management tool that lets you track changes throughout the coding process. Github is one popular service to host these projects online. Contributors of a project have the option of marking steps of the project as a ‘release.’ Releases are considered a tested and stable stop in between less reliable developmental steps.

Releases will be shown in the project’s ‘releases’ page (link in the middle of the page, above the file list). (quickpkg releases page

On the releases page you will see a list of releases with the newest on top. At the very least each release will have a snapshot of the project’s code as a zip or tar.gz archive. Some projects provide other archives or installers such as dmg or pkg as well.

Download the Current Project

Some projects do not manage releases. (You will see ‘0 releases’ in the tool bar.) Then you can still download the most recent version of the project. There is a large green ‘Clone or download’ button on the right area above the project’s file list for this. When you click that button it will expand to show some more options.

‘Download ZIP’ will simply download an archive of the current state of project, much like the release download would.

When you download the archives, either through the releases page or from the ‘Download ZIP’ button, the resulting project folder will not be connected with the Github project any more. If you just want to use the current version, then that is fine and will serve you well. If you want an updated version in the future you will simply download the newer version and replace the tool you already have.

If you rather use git to download and manage the code, then you can do that here, too. However, that is a topic for another post.

Using the Tool

However you get the project you will now have a directory with the tool and any supporting files. You can already change directory to this folder in Terminal (drag the folder on to the Terminal icon to open a new Terminal already changed to it) and run the tool directly:

$ cd ~/Projects/quickpkg/
$ ./quickpkg
usage: quickpkg [-h] [--scripts SCRIPTS] [--preinstall PREINSTALL]
                [--postinstall POSTINSTALL]
                [--ownership {recommended,preserve,preserve-other}]
                [--output OUTPUT] [--clean] [--no-clean] [--relocatable]
                [--no-relocatable] [--sign SIGN] [--keychain KEYCHAIN]
                [--cert CERT] [-v] [--version]
                item_path
quickpkg: error: too few arguments
This will do for tools that you use rarely. But for tools that you want to use frequently typing the path to the tool is quite cumbersome.

Put it in the PATH

The PATH environment variable lists the directories where the shell looks for commands. You could add the project directory of the tool you just added to the PATH, but that would be tedious to manage.

An easier solution is to copy the tool to /usr/local/bin. This is the designated directory for custom commands. /usr/local/bin is also in the default macOS PATH.

However, copying the tool has some downsides. When the tool get’s updated you will have to copy the newer version, as well. Also some tools may require additional resources or libraries that reside in its project directory.

Instead of moving the tool, you can create a symbolic link to the tool in /usr/local/bin.

I keep the project folders of tools in ~/Projects so I use the command:

$ sudo ln -s ~/Projects/quickpkg/quickpkg /usr/local/bin
$ ls -al /usr/local/bin/quickpkg 
lrwxr-xr-x  1 root  wheel /usr/local/bin/quickpkg -> /Users/armin/Projects/quickpkg/quickpkg
Since symbolic links use paths, this has the advantage that when you download a newer version of the project to the same location, the link will point to the new version.

Putting links to a tool in /usr/local/bin has a few downsides (or upsides, depending on your perspective):

  • you need to have administrator privileges to change /usr/local/binlinks/tools you add to /usr/local/bin affect all users on that Mac

Set your own PATH

When you want to have the tools only affect your shell environment you need to do a bit more work.

First you need to choose a location where you tools or links should live. I have created a directory ~/bin for that purpose.

$ mkdir ~/bin
When you don’t want anyone else on the Mac to see what you are doing in that directory, you can remove everyone else’s access to it:
$ chmod 700 ~/bin
If you want you can also hide the directory in the Finder:
$ chflags hidden ~/bin
(Use the same command with nohidden to make Finder show it again.)

(To test the following properly, you need to delete the symbolic link we created earlier in /usr/local/bin. If that still exists the shell will use that, since it comes earlier in the PATH.)

You can create a symbolic link to the in ~/bin with

$ ln -s ~/Projects/quickpkg/quickpkg ~/bin
However, these still will not work, since we need to add the ~/bin directory to your personal PATH.

To do that you need to add this line to your ~/.bash_profile or ~/.bashrc:

export PATH=$PATH:~/bin
(Read more about how to create a bash profile here and here. This assumes you are using bash, the default on macOS. Other shells will have other locations where you can change environment variables.)

Then open a new Terminal window or type source ~/.bash_profile so that the new profile is loaded in the current window and try running the command.

Success!