Category: Command Line

On viewing man pages (Ventura update)

I have written about man pages before. That post references an even older post, which is actually the second oldest post on this blog. Most of the recommendations in the posts still hold true, but there is one change relevant to macOS Ventura and one other thing that is worth adding.

Ventura’s Preview app lost the ability to render postscript or ps files. This breaks the previous, popular shell alias to open a man page in Preview. However, the amazing community in the #scripting channel of the MacAdmins Slack have figured out a replacement. They have asked me to share it.

Add this function to your shell configuration file: (bash, zsh) 

preman() {
    mandoc -T pdf "$(/usr/bin/man -w $@)" | open -fa Preview
}

Then you can run preman <command> in Terminal and the man page will render beautifully in Preview. If you want to override the man command to actually use this function instead of the built-in command, add this alias:

alias man preman

If you then need to revert to the actual man command for a test of something, just add \ before the command: \man <command>

Update: Pico has expanded this into a full blown script which caches the pdfs.

Myself, I had not noticed this change, because I prefer opening the ‘yellow’ man pages in Terminal app. You can do so by entering a command in the Help menu, or by using the x-man-page URL scheme. In these yellow terminal windows, you can scroll and search in the text with command-F. You can also do a secondary click (right/ctrl/two-finger click) on any word and it will offer to open that man page in the context menu.

If the yellow man page windows annoy you, you can change their appearance by modifying Terminal’s “Man Page” profile. I modify it to use my favorite mono spaced font at a larger size. I like the yellow background because it stands out.

In the previous posts, I had a simple function you could add to your shell configuration files, but I have since refined this to also support man page sections (the number you sometimes see after a command, you can learn what they mean in the man page for the man command).

xmanpage() {
  if [[ -z $2 ]]; then
    open x-man-page://"$1"
  else
    open x-man-page://"$1"/"$2"
  fi
}

With this, you can open the desired section in the ‘yellow’ man window with xman 2 stat. This works already with the preman function. The mandoc command knows how to deal with the extra argument, the URL scheme needs a bit of extra work.

If you want to override the normal man command you can, again, use an alias in your shell configuration file:

alias man xmanpage

This way, I can have both functions in the configuration file and choose or change which function (if any) overrides the normal man by just changing the alias.

When you work in Terminal for a while you may accumulate a lot of yellow man page windows. You can use this AppleScript/osascript one-liner to close all Terminal windows which use the ‘Man Page’ profile at once.

osascript -e 'tell application "Terminal" to close (every window where name of current settings of every tab contains "Man Page")'

And while this one-liner is succint, it is still easier to wrap this in a function for your shell configuration file:

closeman() {
  osascript -e 'tell application "Terminal" to close (every window where name of current settings of every tab contains "Man Page")'
}

If you enjoyed this excursion into macOS Terminal and command line tricks and configuration, you might like one of my books: “macOS Terminal and shell” and “Moving to zsh”.

Launching Scripts #4: AppleScript from Shell Script

In the last post, we discussed how to run shell commands and scripts from an Apple Script environment. In this post, we will look at how we can run AppleScript commands and scripts from the shell environment.

Open Scripting Architecture

The key to running AppleScript from the shell is the osascript command. OSA is short for ‘Open Scripting Architecture’ which is the framework that powers AppleScript. This framework allows AppleScript to have its native language, but also use JavaScript syntax.

The osascript command allows us to run AppleScript commands from Terminal and shell. The most common use is the user interaction commands from AppleScript, like display dialog:

osascript -e 'display dialog "Hello from shell"'

The -e option tells osascript that it will get one or more lines of statements as arguments. The following argument is AppleScript code. You can have multiple -e options which will work like multiple lines of a single AppleScript:

> osascript -e 'display dialog "Hello from shell"' -e 'button returned of result'
OK

osascript prints the value of the last command to stdout. In this case, it is the label of the button clicked in the dialog. (The ‘Cancel’ button actually causes the AppleScript to abort with an error, so no label will be returned for that.)

When you have multiple lines of script, using multiple -e statements will quickly become cumbersome and unreadable. It is easier to use a heredoc instead:

osascript <<EndOfScript
   display dialog "Hello from shell"
   return button returned of result
EndOfScript

This also avoids the problem of nested quotation marks and simplifies shell variable substitution.

Shell variables and osascript

There are a few ways to pass data into osascript from the shell.

Since the shell substitutes variables with their value before the command itself is actually executed, this works in a very straightforward manner:

computerName=$(scutil --get ComputerName)

newName=$(osascript -e "text returned of (display dialog \"Enter Computer Name\" default answer \"$computerName\")")

echo "New Name: $newName"

This works well, but because we want to use shell variable substitution for the $computerName, we have to use double quotes for the statement. That means we have to escape the internal AppleScript double quotes and everything starts to look really messy. Using a heredoc, cleans the syntax up:

computerName=$(scutil --get ComputerName)

newName=$(osascript <<EndOfScript
    display dialog "Enter Computer Name" default answer "$computerName"
    return text returned of result
EndOfScript
)

echo "New name: $newName"

I have a detailed post: Advanced Quoting in Shell Scripts.

Environment Variables

Generally, variable substitution works well, but there are some special characters where it might choke. A user can put double quotes in the computer name. In that case, the above code will choke on the substituted string, since AppleScript believes the double quotes in the name end the string.

If you have to expect to deal with text like this, you can pass data into osascript using environment variables, and using the AppleScript system attribute to retrieve it:

computerName=$(scutil --get ComputerName)

newName=$(COMPUTERNAME="$computerName" osascript <<EndOfScript
    set computerName to system attribute "COMPUTERNAME"
    display dialog "Enter Computer Name" default answer computerName
    return text returned of result
EndOfScript
)

echo "New name: $newName"

The shell syntax

VAR="value" command arg1 arg2...

sets the environment variable VAR for the process command and that command only. It is very useful.

Retrieving environment variables in AppleScript using system attribute is generally a good tool to know.

Interpret this!

osascript can also work as a shebang. That means you can write entire scripts in AppleScript and receive arguments from the shell. For example, this script prints the path to the front most Finder window:

#!/usr/bin/osascript

tell application "Finder"
    if (count of windows) is 0 then
        set dir to (desktop as alias)
    else
        set dir to ((target of Finder window 1) as alias)
    end if
    return POSIX path of dir
end tell

You can save this as a text file and set the executable bit. I usually use the .applescript extension.

> print_finder_path.applescript
/Users/armin/Documents

To access arguments passed into a script this way, you need to wrap the main code into a run handler:

#!/usr/bin/osascript

on run arguments
    if (count of arguments) is 0 then
        error 2
    end if
    return "Hello, " & (item 1 of arguments)
end

You can combine this into a longer script:

macOS Privacy and osascript

When you ran the above script, you may have gotten this dialog:

If you didn’t get this dialog, you must have gotten it at an earlier time and already approved the access.

AppleEvents between applications are controlled by the macOS Privacy architecture. Without this, any process could use AppleEvents to gather all kinds of data from any process. These dialogs are easy enough to deal with when running from Terminal. But if you put your AppleScript code (or shell scripts calling AppleScript) into other apps or solutions, it could get messy quite quickly.

Mac Admins generally want their automations to run without any user interactions. You can avoid these dialogs by creating PPPC (Privacy Preferences Policy Control) profiles that are distributed from an MDM server. In this case you have to pre-approve the application that launches the script, which can sometimes also be challenge. The other option is to find solutions that avoid sending AppleEvents altogether.

I have a longer post detailing this: Avoiding AppleScript Security and Privacy Requests

osascript and root

Management scripts often run as a privileged user or root. In this case, certain features of AppleScript may behave strangely, or not at all. I generally recommend to run osascript in the user context, as detailed in this post: Running a Command as another User

Conclusion

AppleScript’s bad reputation may be deserved, because its syntax is strange, and often very inconsistent. Nevertheless, it has features which are hard to match with other scripting languages. You can use the strategies from this and the previous posts to combine AppleScript with Shell Scripting and other languages to get the best of both worlds.

Launching Scripts #3: Shell scripts from AppleScript

In this series of posts, I am exploring the various different ways of launching scripts on macOS. In the first two posts, we explored what happens when you launch scripts from Terminal. We already explored some concepts such as the shell environment and how that affects scripts. In this post we are going to explore a different, but very common way to launch shell commands and scripts: AppleScript’s do shell script command.

do shell script

When AppleScript made the transition from Classic Mac OS 9 and earlier to Mac OS X, it gained one command that allowed AppleScripts to interact with Mac OS X’s Unix environment. The do shell script command executes a command or script in the shell and returns the output as a text to the AppleScript.

We have used this in an earlier post:

do shell script "echo $PATH"
    --> "/usr/bin:/bin:/usr/sbin:/sbin"

But you can use this to run any shell command:

do shell script "mdfind 'kMDItemCFBundleIdentifier == org.mozilla.firefox'"
    --> "/Applications/Firefox.app"

Note the use of single quotes inside the double quotes which delineate the AppleScript text. I have a post on the challenges of quoting in these mixed scripting environments.

You can assemble the command you pass into do shell script using AppleScript text operators:

set bundleID to "org.mozilla.firefox"
do shell script "mdfind 'kMDItemCFBundleIdentifier == " & bundleID & "'"
    --> "/Applications/Firefox.app"

Note that the PATH variable for AppleScripts that are run from Script Editor or as an AppleScript applet is different than the PATH in your interactive environment. Most notably, it does not include /usr/local/bin. When you want to use a command or script that is not stored in the four default directories, you will have to use the full path in the do shell script:

do shell script "/usr/local/bin/desktoppr"

(Desktoppr is a small tool I built to work with desktop pictures on macOS, you can get it here.)

When you are unsure what the full path to a command is, you can use the which command in Terminal:

> which desktoppr
/usr/local/bin/desktoppr

Errors

Keep in mind that which uses the same logic to lookup a command as the shell does when it looks up a command with no path. So, if you think you can trick AppleScript by using the which command to lookup a non-standard command, it will still fail:

do shell script "which desktoppr"
    --> error "The command exited with a non-zero status." number 1

When the command in do shell script returns a non-zero exit code, you will get an interactive dialog informing the user of the error. The AppleScript will not continue after the error. You can handle the error the same way you would handle any AppleScript error, with a try… on error block:

set filepath to "/unknown/file"

try
    do shell script "/usr/local/bin/desktoppr" & quoted form of filepath
on error
    display alert "Cannot set desktop picture to '" & filepath & "'"
end try

Files and Paths

AppleScript has its own methods of addressing files and folders. Actually, there are multiple ways, which is one of the confusing things about AppleScript. Neither of the native forms of addressing files and folder in AppleScript use the standard Unix notation with forward slashes separating folders. But there are built-in tools to convert from Unix notation to AppleScript and back.

Use the POSIX path attribute to get a Unix style file path from an AppleScript file or alias. Unix style paths used with commands need spaces and other special characters escaped. You can use the quoted form attribute to escape any AppleScript string. This is usually used directly with POSIX path:

set imagefile to choose file "Select a Desktop"
    --> alias "Macintosh HD:Library:Desktop Pictures:BoringBlueDesktop.png"
set imagepath to quoted form of POSIX path of imagefile
    --> '/Library/Desktop Pictures/BoringBlueDesktop.png'
do shell script "/usr/local/bin/desktoppr " & imagepath

You can convert a Unix style file path into an AppleScript file with the POSIX file type:

set bundleID to "org.mozilla.firefox"
set appPaths to do shell script "mdfind 'kMDItemCFBundleIdentifier == " & bundleID & "'"
    --> "/Applications/Firefox.app"

if (count of paragraphs of appPaths) = 0 then
    display alert "No app found"
else
    set appPath to first paragraph of appPaths
    -- convert the path to an AppleScript file
    set appFile to POSIX file appPath
        --> file "Macintosh HD:Applications:Firefox.app:"
    tell application "Finder" to reveal appFile
end if

Shell Scripts in AppleScript Bundles

Sometimes you are writing an AppleScript and want to use a script for some functionality which is difficult to achieve in AppleScript. When you have an AppleScript file and a shell script file that work together this way, you want to store them together and also have an easy way for one to get the location of the other.

For this, AppleScript provides the notion of script bundles and AppleScript applets (which are also bundles). A script bundle is not a flat file, but a folder which contains the script itself and other resources, such as script libraries, or shell scripts. The script can easily locate items in its bundle and call them.

For example, we want a script that needs to unzip a file. We can use the unzip command line tool to do that, but in my experience it is better to use ditto. The ditto expansion seems to be closer to how the expansion from Archive Utility works and do better with extended attributes and resource forks and other macOS specific things.

This is the shell script for our example:

#!/bin/sh

# target dir for expansion
targetdir="/Users/Shared/Script Bundle Demo"

# sanity checks for argument 1/filepath
if [ -z "$1" ]; then
    exit 2
fi

filepath=$1

# is it a file?
if [ ! -f "$filepath" ]; then
    exit 3
fi

# note: ditto seems to work better than unzip
ditto -x -k "$filepath" "$targetdir"

This is simple enough that you could just do it in a one-line do shell script, but I want to keep it simple. You could extend this shell script to use different tools to expand different types of archives, such as xar, tar, aa etc. If you want a more complex script, feel free to build one!

Now we can build the AppleScript/shell script combination. Open Script Editor, create a new script and save it right away. In the save dialog, change the ‘File Format’ to ‘Script bundle’ before saving.

After you have saved the Script as a bundle, you can see the Bundle Info in a pane on the right side of the script window. If you don’t see this pane, choose ‘Show Bundle Contents’ from the ‘View’ menu, or click the right most icon in the tool bar.

In this pane, you can set the name, identifier, version and some other data for the script bundle. You can also see a list of the ‘Resources’ which shows the contents of the Contents/Resources folder in the script’s bundle. When you find the script you save, you will see it has a scptd file extension and when you open the context menu on it in Finder, you can choose ‘Show Package Contents’ and dig into the bundle contents.

Note: AppleScript applications (or applets) work the same way. Their .app bundles have a few more sub folders, but the Resources work the same way. The difference is that AppleScript applets work on double-click, drag’n drop, and some other events that we will get to in later posts. Script bundles have to run from Script Editor.

Save the shell script from above into the script bundle’s Resources sub-directory with the name unarchive.sh. You should see it appear in the ‘Resources’ list in the script window.

This way, the AppleScript bundle can contain all the resources it might need, including shell (or other) scripts.

Now we still need to find a way to access the Resources from the script. To run our shell script, add the following code to the AppleScript in Script Editor:

-- Script Bundle Demo
set theArchive to choose file "Select a zip archive:" of type {"zip"}
set archivePath to quoted form of POSIX path of theArchive

-- assemble command
set scriptPath to quoted form of POSIX path of my (path to resource "unarchive.sh")
set commandString to scriptPath & space & archivePath

-- for debugging 
log (commandString)

do shell script commandString

First we prompt the user to choose a file with a zip extension, and the we convert the result into a quoted Unix path.

Then, we use the path to resource "unarchive.sh" to get the path to our shell script in the Resources folder in the bundle. Then we get the quoted Unix notation, and assemble the command for the do shell script. The log command there will print the commandString to the output in the script window and is useful for debugging. Then we run the command with do shell script.

Environment for do shell script

Our example script expands the archive into a subfolder of /Users/Shared. If you wanted to use a different location, you could use a second argument in the script.

There is a different way of passing data into scripts and that is environment variables.

First of all it is important to note that the shell environment for commands and scripts run with the do shell script command from an AppleScript in Script Editor or an AppleScript application is very different from the shell environment in an interactive shell in the Terminal. We have already seen that the PATH environment variable has a different value, which influences the command lookup behavior.

You can check the environment variable by running the env command. This will list all environment variables. (To be nitpicky, there is more to a shell environment than just the env variables, there are also shell options, but those will be different for each shell, sh, bash or zsh, anyway.)

do shell script "env"
    --> "SHELL=/bin/zsh
TMPDIR=/var/folders/2n/q0rgfx315273pb4ycsystwg80000gn/T/
USER=armin
COMMAND_MODE=unix2003
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/usr/bin:/bin:/usr/sbin:/sbin
__CFBundleIdentifier=com.apple.ScriptEditor2
PWD=/
XPC_FLAGS=0x0
SHLVL=1
HOME=/Users/armin
LOGNAME=armin
_=/usr/bin/env"

Interestingly, we have USER and HOME to use in this environment.

We can also add environment variables to a do shell script command:

do shell script "TARGET_DIR='/Users/Shared/Script Bundle Demo' " & scriptPath & space & filePath

You can use this to set the value of the TARGET_DIR env variable for the next command, which is our script in the script bundle.

Administrator Privileges

No matter which way you use do shell script, it has one big benefit. You can prompt the user to get the command or script to run with administrative privileges:

do shell script "systemsetup -getRemoteLogin" with administrator privileges

This will prompt for the user name and password of an administrator user. This can allow to build some simple workflows that require user interaction and administrative privileges really easily.

Conclusion

Combining Script Bundles and AppleScript Applications with shell scripts can create a powerful combination. You can use the “best of both worlds” with some of AppleScript’s user interaction commands and the shell’s strength in file manipulation and similar workflows. You can also sign AppleScript applications with a valid Apple Developer ID and pre-approve them for privacy exemptions with a PPPC profile.

If this explanation is not detailed enough for you, there is an amazing Tech Note in Apple’s Documentation Archive.

This post covered launching shell scripts from AppleScript. In the next post we will launch AppleScript code from shell scripts.

Launching Scripts #2: Launching Scripts from Finder

In this series of posts, I will explore the many ways that you can launch a script on macOS. In the previous, inaugural post, I described what happens when you launch a script from an interactive terminal shell.

There are several virtual terminal applications available for macOS. iTerm is very popular. Some text editors like Visual Studio Code and Nova, have terminals built-in. Since the actual launching of an executable is done by the shell running inside the virtual terminal, the launch process remains the same.

That said, Terminal app has a useful trick up its sleeves.

command file extension

When you change the file extension of a script to .command, double-clicking the file will open it in a new Terminal window and run it there. Any input or output the script requires will happen in that Terminal window. When the script exits, the shell session in the Terminal window will exit.

Let’s take this simple script:

#!/bin/sh
echo "Enter your name: "
read -r username
echo "Hello, $username"

When you put this this in a .command file and double-click it, you get a new Terminal window with:

/Users/armin/Desktop/hello_name.command ; exit;                                 
~ % /Users/armin/Desktop/hello_name.command ; exit;
Enter your name:

You can see that Terminal opens a new window with a new, default shell and all your configurations, then launches the script right away. The script prints its output and then waits for the user input (the read command). When you enter the name at the prompt, the script continues.

Armin
Hello, Armin

Saving session...completed.

[Process completed]

When the script ends, the shell in the Terminal window exits, as well. No more interactive prompt will be shown.

This script expects user input in the Terminal and then presents output to stdout in the same window. While you could re-write a script to use AppleScript’s display dialog to handle both in the input and the output, it would make the script significantly more complex.

Instead, you can change the file extension to .command and then a double-click will create a new Terminal window where the user interaction (input and output) takes place. For the right kind of user and workflow, this can be a sufficient solution with practically no overhead.

You can also remove the file extension completely. The behavior when you double-click such a file in Finder will be the same. Extension-less executables also get a different the icon. Either way, you need to have the executable bit set for the script.

Note: iTerm can also open .command files, but I have had some trouble with user interaction in these cases. Since I usually don’t use iTerm, maybe I have something setup wrong?

Quarantine and Gatekeeper

A Terminal window is not a user interface that many users will appreciate, but this allows you share scripts with other users in a form they understand. “Double-click this to run” is something that fits with most users’ idea of how macOS works.

Before you start creating dozens of .command scripts and share them, there is a major tripwire that macOS security has set up.

When you share an executable file through a website, email or a chat message, macOS will attach a quarantine flag. With applications, this flag triggers a GateKeeper scan before the app is launched and it will show the standard dialog, even when the app is signed and notarized and a much more “scary” warning when it is not.

When you launch a command or script from Terminal, either directly or indirectly with a double-click, and it still has the quarantine flag set, it will not launch. You will not get one of the standard Gatekeeper dialogs. Just an opaque operation not permitted error in the Terminal output.

You can check if a file has the quarantine flag set with the xattr command:

> xattr hello_name.command
com.apple.TextEncoding
com.apple.lastuseddate#PS
com.apple.quarantine

Your list list of extended attributes may be different. The quarantine flag has the label com.apple.quarantine.

The xattr command also can remove the quarantine flag:

> xattr -d com.apple.quarantine hello_name.command

Apple seems to assume that when you are using Terminal, you know what you are doing. That means you can bypass most of the mechanisms that attach a quarantine flag with command line tools. When you download something with curl it won’t get quarantined. You can install an unsigned, unnotarized pkg installer using the installer command. Because this is possible, it doesn’t mean it is always wise. Piping a curl command directly into sh or bash or any interpreter is still poor security.

Most of the time though, a script file shared to another Mac or another user will almost certainly get the quarantine flag. Users who are comfortable with using Terminal should be able to use the xattr command to disable this protection, but this is not something for ‘normal’ users. So quarantine, makes the use of the command file extensions far less effective than it could be. This is probably intentional, since executables that are opened by double-click can be an easy way to sneak malware and other unwanted software onto a system, leveraging a user’s ignorance of what is happening.

This is generally true when you move scripts and executable files between macOS systems. I have also seen the quarantine flag getting set when you store a script in a cloud sync service (especially iCloud) or when you edit an executable with a sandboxed application.

One way around the quarantine, would be to distribute and properly install the scripts with an installer pkg. Then you can properly sign and notarize the installer pkg and the scripts will not be quarantined, as they come from a trusted and verified source. This may be a good solution for some workflows, but generally feels a bit “over-designed.”

Output flashes by so quickly

There is a setting in Terminal’s preferences which determines what happens with windows when the shell exits. You can find it under the ‘Shell’ tab in the ‘Profiles’ area. This setting can be different for each profile. Under ‘When the shell exits’ there is a popup menu with the options ‘Close the window,’ ‘Close if the shell exited cleanly,’ and ‘Don’t close the window.’ The last ‘Don’t close’ is the default.

When you have this option set to ‘Close the window,’ the new Terminal window from a command file might only be active and visible for a short time. This may or may not be a good thing, depending on what you want.

Conclusion

Script files with the command file extension can be a simple, straightforward way to make scripts easily ‘launchable’ from Finder. You can also put them in the Dock or in the Login Items. The user experience is, well, a terminal, so not terribly nice, but it can be useful, and does not require any modification of the script.

When you share executable scripts, whether they have the command file extension or not, Gatekeeper quarantine on macOS can prevent the script from running. You should get familiar with the quarantine flag and the xattr command to manipulate it.

In the next post, we launch shell scripts from AppleScripts.]

On env Shebangs

There was a comment to my previous post about using the /usr/bin/env shebang to gain system portability.

Regarding shebang, a more system independent way would be to use e.g. ‘#!/usr/bin/env bash’. With this the script would also be usable on FreeBSD, where bash would be installed from FreeBSD Ports and then be available as ‘/usr/local/bin/bash’ or some Linux systems (e.g. Debian) where it is ‘/usr/bin/bash’. Unfortunately there are some other unixode systems around, where ‘env’ is not in ‘/usr/bin/’ and so the shebang needs to be adjusted.

(I replied to the comment there, but then realized this deserves its own post.)

The /usr/bin/env shebang provides a means for system portability. It has many valid use cases. However, you don’t just magically gain portability by switching to an env shebang. There are many trade-offs to consider.

The note on how the env binary may not be in /usr/bin on all platforms, hints at some of these trade-offs, but there are more.

The trade-offs are a loss of predictability and reliability, or functionality, or increased maintenance and management.

Let me elaborate…

How the /usr/bin/env shebang works

When used in the shebang, the /usr/bin/env binary will use the current environment’s PATH to lookup the interpreter binary for the script, in the same way the shell looks up commands.

As an example, let us imagine you have installed bash v5 on your Mac. Either manually or using brew or some other package management system.

This will (usually) put the bash v5 binary at /usr/local/bin/bash. The /usr/local/bin directory is a common choice for custom command line tools, because it is part of the default PATH for interactive shells on macOS and not protected by SIP/SSV. The default PATH on macOS for interactive shells is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Some installations will put the binary in a different location in the file system. Then you you have to pre-pend the directory containing the binary to your PATH variable in your shell configuration. The order of the directories in the PATH is important, because the shell and env will stop the search when they find the first match. If /usr/local/bin came after /bin in the PATH the new binary would not be ‘seen’ since the pre-installed, old /bin/bash binary is found first.

Some installations solve this by placing a symbolic link to the binary in /usr/local/bin.

When you run a script from the interactive shell with a shebang of #!/usr/bin/env bash, then env would find the bash v5 binary first in /usr/local/bin, so your script is interpreted with bash v5. This is probably what you were hoping for, when you installed bash v5 in the first place.

When you run the same script on a different Mac (same macOS version, but it doesn’t have bash v5 installed) env will pick up /bin/bash. Your script will work even though that other Mac doesn’t have /usr/local/bin/bash, so you gained portability.

However, /bin/bash is bash v3.2, so your script may behave differently. If the script uses bash v5 features that are not available in the 15-year-old bash v3.2, it will generate errors. Since you actively chose to install bash v5 on the first Mac, it is likely you needed some of these bash v5 features, so it is likely your script will fail on other Macs, which don’t have bash v5 installed.

You lost either predictability and reliability (which version and features are available? Does my script run successfully?), or you lose functionality (the features added to bash v5 since v3.2). To retain reliability, you can restrict the script to features that work in both bash versions. But then using the env shebang gives you no advantage, and you might as well use /bin/bash as the shebang.

Some solutions

One alternative is to use a /usr/local/bin/bash shebang for scripts which use bash v5 functionality and continue to use /bin/bash for scripts that need to run across multiple Macs, where you pay attention to using only features available in bash v3.2. You gain predictability and reliability, but your bash v5 scripts aren’t portable to other Macs. They may even fail on other Macs with bash v5 installed, if the bash v5 binary is installed in a different location.

When you use /usr/bin/env bash for a bash v5 script, it will run fine on all Macs which have bash v5 installed and the PATH configured properly to find it. (Configuring and maintaining the PATH does not happen on its own.) But the script will still fail on Macs without any bash v5. You can (and probably should) add a version check to the script, but now you are increasing code maintenance.

When you are managing a fleet of Macs, you also have the option (or in this case, I would say, the duty) to install bash v5 in a consistent location and version across all Macs in your fleet and pre-configure the proper PATH in the contexts the script will run in. Then you get predictability and functionality, but it requires extra effort in deployment and maintenance.

This requires a decently experienced MacAdmin and the proper tooling, neither of which comes for free.

Note: There are great open source solutions for macOS in this area, but I consider them ‘free, as in puppy,’ so they come with higher skill requirements and/or maintenance effort for the admin. And this isn’t supposed to imply that all commercial solutions are ‘easy to use,’ either. It’s trade-offs all the way down.

Context changes the PATH

Notice that so far I kept talking about “the default PATH for the interactive shell.”

The PATH variable may be different depending on the context, even on the same Mac with the same user. For example, when you run your script with the AppleScript do shell script command, the PATH in that context is not the same as the PATH in your interactive shell. It will be:

/usr/bin:/bin:/usr/sbin:/sbin

You can verify this by opening Script Editor and running the do shell script "echo $PATH". Other context, like scripts in installation packages will see other PATH values.

Most importantly, the PATH in these other contexts, does not contain /usr/local/bin, or any other addition you made to your PATH in the shell configuration files. An /usr/bin/env shebang will not ‘see’ a bash 5 binary you installed on the system. The same script with the same user on the same computer, will behave differently when run in a different context.

These different PATH values are an intentional choice. In these contexts, especially installation package scripts, reliability and predictability are extremely important. You do not want user and third-party installed custom binaries to interfere with the command lookup.

Sidenote on Python

With python and python3 and other run time interpreters, it gets even more difficult. There may multiple different versions installed and the behavior and functionality between versions varies more. My Mac currently has four different Python 3 binaries, each with a different version, and I not even remotely a full-time Python developer. When you call python3 on a non-developer Mac it will trigger the ‘You have to install Developer Command Line Tools’ dialog when Xcode is not installed. (Developers seem to have a hard time considering that there are Macs without Xcode installed.)

With the demise of python 2 in macOS 12.3, some developers reacted by changing the shebang in their python scripts from /usr/bin/python to /usr/bin/env python which solves nothing, when the binary goes away without replacement. Some switched to /usr/bin/env python3 which can makes things worse, by triggering the Developer Tools installation or picking a random python3 binary of the wrong version.

The only reliable solution for the mess that is python is to deploy a consistent version of Python 3 in a consistent location. You can do this by either bundling the python framework and module your tool needs together with the tool, or by deploying and maintaining the Python frameworks and modules with a management system.

MacAdmin perspective

As a MacAdmin, my scripts don’t need to be portable to systems other than macOS. They usually use tools and access files and folders that only exist on macOS. Predictability and reliability, however, are paramount. Configuration and installation scripts need to run reliably on thousands of Macs across multiple versions of macOS (even future versions) regardless of what else is installed.

As MacAdmins, we also (should) have the tools and experience to deploy and maintain binaries in predictable locations. But then, like everyone, we only have limited time, and often need to prioritize business critical software over our own tooling. So, the pre-installed interpreter binaries have little ‘friction’ to use, even if they may have a reduced functionality when compared to the latest version available elsewhere.

This is the reason bash v3.2 is still present on macOS 12.3 and it will never be easy when Apple ultimately decides to remove it. So many tools and scripts rely on /bin/bash.

(I don’t expect the removal to be any time soon, but there is a limit to how long Apple will or can keep this interpreter from 2007 on the system. We got the first warning when Apple switched the default interactive shell to zsh in Catalina. There will be more warnings …I hope. With the removal of the Python 2 binary we saw that Apple can move quickly when they feel the need. They did not even wait for a major macOS release.)

In this context, there is no gain in using /usr/bin/env. The trade-offs favor the absolute shebang very strongly.

Cross-platform portability

After this rant, you may think that I recommend against using /usr/bin/env shebangs always. But there are very good use cases. Using /usr/bin/env shebangs is the solution for workflows where cross-platform portability is required.

When your scripts need to run across multiple platforms, installing the binaries in the same location in the file system may not be possible, or require an unreasonable effort. For example, the /bin and /usr/bin are protected by SIP and the Sealed System Volume on macOS, so you cannot add your tooling there without significantly impacting the integrity and security of the entire system.

In these cases, using a /usr/bin/env shebang provides the required flexibility, so your scripts can work across platforms. But the portability does not come magically from just switching the shebang.

The target platforms need the binary to be installed and the versions should match. The installation location of the binary has to be present in the PATH in the context the script runs in. To get reliable behavior, the systems you are porting between need to be well managed, with a predictable setup and configuration of the interpreter binary and environment.

When your scripts work ‘effortlessly’ across systems with the env shebang, it is thanks to the work of the platform developers and your sysadmins/devops team for creating and maintaining this consistency. Even if you are the sole developer and devops admin, maintaining all the systems, you have to put in this work. Also the platform developers put in a lot of effort to achieve much of this consistency out of the box. As the commenter noted, some platforms don’t even agree where the env binary should be.

You gain portability at the price of increased maintenance.

Trade-offs all the way down

Alternatively, you can keep the scripts simple – restricted to a subset of features common across platforms and versions – so that the differences have no impact. Then you trade for reliability and portability at the price of functionality.

This is often the trade-off with Python (restrict the use of python features to those common among versions) and one of the reasons Python 2 was kept around for so long on macOS.

Using POSIX sh, instead of bash or zsh, is another option to gain portability, but that has its own trade-offs. Most of these trade-offs will be in functionality, but also consider not all sh emulations are equal, and supporting multiple different emulators or the real common subset of functionality, requires extra effort.

Conclusion

Shebangs with absolute paths have their strengths and weaknesses, as do shebangs with /usr/bin/env. Each has their use case and you have to understand the trade-offs when using either. Neither env shebangs nor absolute path shebangs are generally ‘better.’ Either one may get you in trouble when used in the wrong context.

When someone says ‘you should be using,’ or ‘a better way would be,’ you always need to consider their context, use case, and which trade-offs they are accepting. You need to understand the tools to use them efficiently.

Some CLI updates in macOS Monterey

The other day on Twitter, I got a question about a flag for the readlink command that I was not familiar with. As it turns out, the readlink command (which tells you where a symbolic link points to) got an update in macOS 12.3 and now has a -f option. With this new option, readlink will resolve symbolic links anywhere in the path and print the ‘actual’ absolute path to the item. This is equivalent to the realpath command available on Linux and some programming languages.

I have written about this before, and then I mentioned that there is a python function to resolve the path. However, even back then I anticipated the removal of python and suggested using a zsh parameter expansion modifier instead:

resolved_path=${path_var:A}

The removal of Python 2 is the likely explanation for why Apple chose to update readlink in 12.3.

It will be nice to have the new readlink -f option available going forward, but if your script still needs to support versions of macOS older than 12.3 then you should prefer to use the zsh expansion modifier.

More Monterey Command Line Changes?

This was discovered mostly by chance. While Apple’s release notes are improving, there are still nowhere near detailed enough and missing this level of detail, even though that would be amazingly useful.

I remembered that the Kaleidoscope app team had posted a script that allows me to compare man pages between versions of macOS. They published this back when macOS 12 was released to track the changes of the plutil command line tool. With the help of this tool I determined a few more interested changes in macOS 12, the most interesting of which I will summarize here.

(Even with this script, the process was tedious. Many changes to the man pages are just reformatting whitespace and/or typos. I may have missed something. Please, let me know when you find more changes!)

cut

  • new -w option (splits fields on whitespace)

du

  • new -A option (apparent size)
  • new --si option (human-readable, in 1000 based units)
  • new -t option (only show items over a certain threshold)

aa (Apple Archive)

  • new options for encryption
  • new aea command for encrypted Apple Archives

tar

  • new encryption and compression types

find

  • new -quit primary
  • new -sparse primary (so you can find APFS sparse files)

grep

  • new rgrep, bzgrep, bzegrep, and bzfgrep
  • new --label option
  • new -M, --lzma option

hdituil

  • segment subcommand and Segmented images are deprecated
  • UDBZ dmg format (bzip2 compression) is deprecated
  • udifrez and udifderez are deprecated (this allows to embed a license in a dmg)

head

  • new -n and -c options (probably just the man page updated)

killall

  • new -I option (confirm)
  • new -v option (verbose)

ls

open

  • new -u option to open file paths as URLs

pkgbuild

  • new --large-payload option
  • new --compression option
  • new --min-os-version option

I have an article on the new pkgbuild options.

plutil

  • new -type option for extract
  • new -raw option for extract
  • new type subcommand to query type
  • new create subcommand to create a new empty plist

pwd

  • new -p option prints working directory with symbolic links resolved

readlink (12.3)

  • new -f option to resolve symbolic links

rm (12.3)

  • new -I option which prompts only when more than three files will be deleted or a directory is being removed recursively

shortcuts

  • new command to run, list, or interact with Shortcuts

smbutil

  • new multichannel and snapshot verbs

Also, the nano command now actually opens pico. (Thanks, @rgov) Most people won’t notice this, as the two are quite similar. The excision of GNU tools from macOS continues.

Scripting macOS, part 7: Download and Install Firefox

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Follow this blog or the Twitter account for updates on the book’s progress!

Download and Install Firefox

To further illustrate the progress from the idea of a workflow to a working script, let us look at another, more involved example.

To download and install the latest version of Firefox a user has to go to the Firefox website and download the latest version, which will come as a disk image (dmg) file. Then the user needs locate the dmg in the ~/Downloads folder and open it to mount the virtual disk image. Finally, they need to copy the Firefox application from the virtual disk to the Applications folder.

When we want to automate the task ‘Download and Install Firefox,’ we have the following steps:

  • download latest Firefox disk image
  • mount downloaded disk image
  • copy Firefox application to /Applications
  • unmount disk image

From this list of steps, we can build the first ‘frame’ of our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

This breaks the workflow into smaller pieces, that we will now tackle individually.

Download from the Command Line

You can use the curl command to download data in the command line. The curl command is very complex and has many options. We will only discuss the few options that we require for our task here. As always, you can find a detailed description of the curl command and its options in the curl man page.

The URI to download the latest Firefox is
https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US

However, when you try to curl this URI, you only get the following:

> curl "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US"
<a href="https://download-installer.cdn.mozilla.net/pub/firefox/releases/86.0.1/mac/en-US/Firefox%2086.0.1.dmg">Found</a>.

This is a re-direction, that is commonly used to have a single URI, that is redirected to different final URIs, so that when the software updates, the same URI always returns the latest version.

We can tell curl to follow these redirections with the --location option.

By default, the curl command will output the download to standard out. To save the download to a file, we can use the --output option with a file name.

> curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" --output Firefox.dmg

This command will download the latest Firefox disk image to a file named Firefox.dmg in your current working directory. We can use this as our first step:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

Note: Like many other command line tools, curl has short and long options. The short options for –location and –output are -L and -o.
Short options are convenient in the interactive shell, as they save typing and reduce the potential for typos. But they are much less readable, and you usually have to look up their function in the documentation. For that reason, I recommend using the long, descriptive options in scripts.

Working with Disk Images

The command line tool to work with disk image (dmg) files on macOS is hdiutil. This is also a very powerful command with many verbs and options. You can find all the detail in the hdiutil man page.

To mount a disk image, use the attach verb:

> hdituil attach Firefox.dmg

This will output some information and mount the virtual disk. The last line ends with the path to the mounted virtual disk /Volumes/Firefox.

By default, you can see the mounted volume in Finder. We do not really need the disk image to appear in Finder while the script is running. We can suppress this behavior with the -nobrowse option.

Since we are only going to read from the disk image, we can tell hdiutil to mount the dmg in readonly mode with the -readonly option. This speeds things up a bit.

> hdiutil attach Firefox.dmg -nobrowse -readonly

You can unmount or eject the virtual disk with

> hdiutil detach -force /Volumes/Firefox

The -force option will unmount the disk image, even when another process is still using it.

Thehdiutil command covers two of our steps, so we can fill them in:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications

# unmount disk image
hdiutil detach /Volumes/Firefox -force

Copying the Application

When you manually install Firefox the disk image shows you a nice graphic that reminds you to drag the app to the Applications folder. Once the disk image is mounted, the cp command can be used to do this in the shell:

> cp -R /Volumes/Firefox/Firefox.app /Applications/

This provides the last missing step in our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications
echo "copying Firefox to /Applications"
cp -R /Volumes/Firefox/Firefox.app /Applications/

# unmount disk image
hdiutil detach /Volumes/Firefox/ -force

You can now test the script. If Firefox is running, you want to quit it before you run the script. You may also want to delete the existing copy of Firefox from the Applications folder, to be sure that your script is doing the work.

Lists of Commands—Conclusion

We have been able to automate a fairly complex workflow with a script of four commands.

To be perfectly honest, this script (as well as all the others we have built so far) is not complete yet.

A ‘proper’ script needs to be able to react to errors that occur. In our example, imagine the download fails. The script should be able to detect the failure before it overwrites the installed, functional Firefox application.

We will get to this kind of error handling later.

Nevertheless, this script is already useful in its current form. You can try to adapt this script to work with some other software you can download as a disk image.

You can also add extra commands that

  • delete the downloaded disk image at the end
  • open the newly installed Firefox app after installation
  • quit or kill the Firefox process before copying the new version

In the book “Scripting macOS”, you will learn more scripting techniques, and we will re-visit some of these sample scripts and keep improving them.

Follow this blog or the Twitter account for updates on the book’s progress!

Note: After using different variations of these kinds of workflows, I did put together a more generic script to download and install various kinds of software, called ‘Installomator.’ You can see the script at its open source repository on GitHub.

Scripting macOS, part 6: Turn it off and on again

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Turn it off an on again

A common strategy to solve problems with computers is ‘turning it off and on again.’ For example, when you have Wi-Fi connectivity problems, one of the first things you should try it to turn the Wi-Fi off, wait a few seconds, and then turn it on again.

Let’s write a script to do that.

Create the ‘frame’

When you start out writing a script, you will usually have only a vague idea of what needs to be done, and even less an idea of how to actually do it. In this case is helps to write out the steps that your script should perform:

  • turn off Wi-Fi
  • wait a few seconds
  • turn on Wi-Fi

With this list we have broken down the problem into smaller, more manageable steps, which we can each solve on their own.

I usually take this list of steps and build a script ‘frame’ from them:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi

# wait for a few seconds

# turn on Wi-Fi

We just copied our list of steps into the text file for our script and made them comments by adding the # character at the beginning of the line. We also added the shebang in the first line and two more comment lines at the beginning, which name the script and have short description on what it is supposed to do.

Since the script in this form only consists of the shebang and comments, it does nothing. But it provides a frame to fill in. Now we can tackle these steps one at a time.

Control Wi-Fi

When you want to control network related settings on macOS, the networksetup command is the first place you should look. The networksetup command allows you to configure the settings in the ‘Network’ pane in ‘System Preferences’— and a few more. The networksetup command has dozens of options. You can get a list and descriptions by running networksetup -help or in the networksetup man page.

To turn off Wi-Fi, you can use this option:

> networksetup -setairportpower <hwport> off

The value you need to use for <hwport> depends on what kind of Mac you are working on. For MacBooks, it will be en0 (unless you have a very old MacBook with a built-in ethernet port). For Macs with a single built-in ethernet port, it will be en1. For Macs with two built-in ethernet ports it will be en2.

You can also use the networksetup command to list all available hardware ports and their names:

> networksetup -listallhardwareports

Hardware Port: Wi-Fi
Device: en0
Ethernet Address: 12:34:56:78:9A:BC

Look for the Hardware Port named Wi-Fi. The value shown next to Device is the one you have to use. So, for me, on a MacBook, I will use:

> networksetup -setairportpower en0 off

We will use en0 in our sample script going forward. If your Wi-Fi port is different, remember to change it in your script going forward

Note: Apple used to brand Wi-Fi as ‘Airport’ and this naming still lingers in some parts of macOS. Changing networksetup’s options to match the new branding would break all existing scripts.

When you replace the off with on it will turn the Wi-Fi back on:

> networksetup -setairportpower en0 on

We have solved the first and third step for our script and tested them in the interactive shell. Now, we can fill them into our script frame:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi
networksetup -setairportpower en0 off 

# wait for a few seconds

# turn on Wi-Fi
networksetup -setairportpower en0 on

You can now save the script as reset_wifi.sh, set its executable bit and run it:

> chmod +x reset_wifi.sh
> ./reset_wifi.sh

This should already work. You should see the Wi-Fi icon in the dock switch to the animation indicating that it is re-connecting with your network.

Taking a break

We still want the script to ‘take a break’ and wait for a few seconds between the turning off and turning back on commands. This will allow other parts of system to ‘settle’ and react to the change.

In the shell you can use the sleep command to achieve this. It takes a single argument, the time in seconds it should pause before continuing:

> sleep 5

When you enter this command in the interactive shell, you should notice that it takes 5 seconds for the next prompt to appear. The sleep command is delaying progress for the given time.

When you watch the CPU load in Activity Monitor while you run the sleep command, you will not see a spike in load. The sleep command merely waits, letting other processes and the system do their thing.

Let us insert a ten second break into our script between the commands to turn Wi-Fi off and on:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again

# turn off Wi-Fi
networksetup -setairportpower en0 off 

# wait for ten seconds
sleep 10

# turn on Wi-Fi
networksetup -setairportpower en0 on

Now, when you run the script. You can see that Wi-Fi icon in the menu bar should switch to ‘disabled’ and then return to the ‘searching’ animation and eventually reconnect after ten seconds.

Feedback

While you are running this script, there is no feedback in the Terminal though. It just takes a disconcerting time for the prompt to return. We can add some output to let the user of the script know what is happening:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi
echo "Disabling Wi-Fi"
networksetup -setairportpower en0 off 

# wait for a ten seconds
echo "Waiting..."
sleep 10

# turn on Wi-Fi
networksetup -setairportpower en0 on
echo "Re-enabled Wi-Fi"

Now, the user will get some feedback in Terminal that lets them know what is going on.

>  ./reset_wifi.sh
Disabling Wi-Fi
Waiting...
Re-enabled Wi-Fi

Script Building Process

We formed an idea or a goal: ‘Turn Wi-Fi off and back on’

Then we described the steps necessary to achieve that goal:

  • turn off Wi-Fi
  • wait a few seconds
  • turn on Wi-Fi

We used these descriptive steps to create a frame or scaffolding script to ‘fill in’ with the proper commands.

Then we explored commands that would achieve these steps in the interactive terminal.

Once we determined the correct commands and options we placed them into our script ‘frame’ in the correct order.

As you add the commands, test whether the script shows the expected behavior.

Then we also added some output, to provide user feedback.

This is a really simple example script to illustrate this process. Nevertheless, we will re-visit these process steps with every script we build and you should follow this process (or a similar flow) when building your own scripts.

The process will not always be quite so clearly defined and you may have to iterate or repeat some of these tasks multiple times, before you find a combination of commands that do what you want. Less experienced scripters will naturally spend more time on ‘exploring commands.’ This is part of the normal learning process.

With larger, more complex scripts, each descriptive step may need to broken into even smaller sub-steps to manage the complexity. Again, experienced scripters will be able tackle more complex steps faster. Do not let that intimidate you when you are just beginning! Experience has to be built ‘the hard way’ by learning and doing.

In many situations you can learn from other people’s scripts that solve a similar problem. This is a well-proven learning strategy. I would still recommend to ‘explore’ the commands used in other people’s scripts before adding them to your scripts, as it will deepen your knowledge and experience with the commands. Sometimes, you may even be able to anticipate and avoid problems.

Next Post: Download and Install Firefox

Scripting macOS, part 5: Lists of Commands

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Beyond Hello, World

Now that you have a minimal working script, let’s extend it a bit.

Create a copy of the hello.sh script file, set its executable bit, and open it in your favored text editor:

> cp hello.sh hello_date.sh
> chmod +x hello_date.sh
> bbedit hello_date.sh

A script is a list of commands that the interpreter will process sequentially. Up to now, we only have a single command, the line with echo, in our script. We will add another one:

Change the text in the script file:

#!/bin/zsh

# Greetings
echo "Hello, World!"
date

We have added a line to the script with the date command.

If you are unfamiliar with this command, you can try it out in the interactive command line:

> date    
Tue Feb 23 10:23:05 CET 2021

When you invoke date with out any arguments it will print out the current date and time. The date command has many other functions for doing date and time calculations which we will not use right now, but you can read about in the date man page.

Save the modified script and execute it:

> ./hello_date.sh
Hello, World!
Tue Feb 23 10:27:06 CET 2021

As we can tell from the output, each command in the script was executed one after the other and the output of each command was shown in Terminal.

You can insert more echo commands before the date to make the output prettier:

#!/bin/zsh

# Greetings
echo "Hello, World!"

# print an empty line
echo

# print without line break
echo -n "Today is: "

date

As we have learned earlier, empty lines and lines starting with a # character will be ignored, but can serve to explain and clarify your code. I will be using comments in the example scripts for quick explanations of new commands or options.

Here I added the -n option to the third echo command. This option suppresses the new line character or line break that echo adds automatically at the end of the text. This will then result in the output of the date command to print right after the ‘Today is:’ label, rather than in a line of its own.

Take a look:

> ./hello_date.sh
Hello, World!

Today is: Tue Feb 23 10:30:44 CET 2021

The date command allows for changing the format of the date and time output. It uses the same formatting tokens as the C strftime (string format time) function. You can read the strftime man page for details. The %F format will print the output as ‘year-month-date:’

> date +%F
2021-02-23

You can combine place holders:

> date +"%A, %F"
Tuesday, 2021-02-23

You can experiment with the date formatting placeholders from the strftime man page in the interactive shell. Once you have built a formatter that you like, you can add it to the date command in your script.

#!/bin/zsh

# Greetings
echo "Hello, World!"

# print an empty line
echo

# print without line break
echo -n "Today is: "

date +"%A, %B %d"

It is very common that you will test and iterate a command and arguments in the interactive shell before you add or insert it into your script. This can be a much easier and safer means of testing variations of commands and options than changing and saving the entire script and running it repeatedly.

Scripts are basically ‘lists of commands.’ Once you know the steps to perform a workflow in the interactive terminal, you can start building a script. Let’s look at another example.

Desktop Picture Installer

When you copy an image file to /Library/Desktop Pictures, it will appear in the list of pictures in the ‘Desktop & Screen Saver’ pane in System Preferences. On macOS 10.15 Catalina and higher you may have to create that directory.

You can easily build an installer package (pkg file) that installs an image file into that location with the pkgbuild command.

First, create a directory to hold all the sub-directories files we will need. There needs to be a payload directory in the project directory. Copy the image file into the payload directory:

> mkdir BoringDesktop
> cd BoringDesktop
> mkdir payload
> cp /path/to/BoringBlueDesktop.png payload

You can then build an installer package with the following command:

> pkgbuild --root payload --install-location "/Library/Desktop Pictures/" --identifier blog.scripting.BoringBlueDesktop --version 1.0 BoringDesktop-1.0.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to BoringDesktop-1.0.pkg

This will create a file BoringDesktop-1.0.pkg. When you double-click this file, it will open the Installer application and, when you proceed with the installation, put the image file in /Library/Desktop Pictures.

Note: To learn more about using and building installer package files for macOS, read my book “Packaging for Apple Administrators.”

The pkgbuild command has a lot of arguments and when you get any of them just slightly wrong, it may affect how the installer package works. This is a very simple example, but when you build more complex installer packages, you may be building and re-building many times and you want to avoid errors due to typos.

We can copy this big command and place it in a script file:

#!/bin/zsh
pkgbuild --root payload --install-location "/Library/Desktop Pictures/" --identifier blog.scripting.BoringBlueDesktop --version 1.0 BoringDesktop-1.0.pkg

Then make the script file executable:

> chmod +x buildBoringDesktopPkg.sh

Now you can just run the script and not worry about getting the arguments ‘just right.’

But we can go one step further and make the script more readable. We can add a comment describing what the script does.

The shell usually assumes the line break to be the end of the command. But when you place a single backslash \ as the last character in a line, the shell will continue reading the command in the next line. That way, you can break this long, difficult to read command into multiple lines:

#!/bin/zsh

# builds the pkg in the current directory

pkgbuild --root payload \
         --install-location "/Library/Desktop Pictures/" \
         --identifier blog.scripting.BoringBlueDesktop \
         --version 1.0 \
         BoringDesktop-1.0.pkg

The arguments are now more readable. When you want to change one of the values, e.g. the version, it is easier to locate.

Even though this script only contains a single command, it improves the workflow significantly, as you do not have to remember a long complex command with many arguments.

Of course, you can copy and modify this script for other package building projects.

Next Post: Turning it off an on again

Scripting macOS, part 4: Running the Script

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week over the summer. Enjoy!

Running the Script

Now that we have the code for a functional (even though minimal) script, you can run it from the command line with:

> ./hello.sh

(Verify that the current directory is your script project folder.) You probably have wondered why you need the ./ before the script name. When you type the script name without the ./ it will fail:

> hello.sh
zsh: command not found: hello.sh

What does the ./ mean and why do we need it?

Current Working Directory

In the shell, the . character represents the current working directory or the directory that you ‘cd-ed’ to. Usually, the working directory is shown in the window title bar of Terminal and in the shell prompt (the status text shown before your cursor in the Terminal window). You can also have the system show your current working directory with the pwd command:

> pwd
/Users/armin/Projects/ScriptingMacOS

When you type ./hello.sh the shell will substitute the current working directory for the . character to get

/Users/armin/Projects/ScriptingMacOS/hello.sh

Your path will of course look different.

This means that when I use the ./hello.sh form, the shell knows exactly which file I want to execute and where to look for it. There is no ambiguity.

Now we know what the ./ means, but why is it necessary?

Finding Commands

A command you enter in the shell is just a string of text to begin with. The shell has to parse this text into pieces to determine what needs to be done.
We will use the chmod command from earlier as an example:

> chmod +x hello.sh

The text you entered is ‘chmod +x hello.sh’. The shell will split this text into pieces on the spaces (or tab characters). This will yield three elements:
‘chmod,’ ‘+x,’ and ‘hello.sh’

The shell is only really interested in the first element. The shell will try to interpret the first element of your entry as a command.

Note: Many programming languages start counting at zero. This first element is also called ‘argument zero’ or $0.

The remaining elements are arguments, which the shell will pass on to the command. Arguments are optional. Not all commands require or even have arguments.

Note: This is a simplified description of the parsing process in shells. The reality is quite a bit more complex. But this description is ‘close enough’ for most situations. We will explore some of the nuances later.

If you are interested, you can get all the details in the shell’s documentation:

  • Zsh Documentation: Shell Grammar
  • bash man page, search for ‘Command Execution’

When there is a ‘/‘ character anywhere in the first element, the shell will interpret the first element as a full or relative path to an executable file and attempt to run that.

This is the behavior we are using when we type ./hello.sh. Since that contains a ‘/‘ the shell will resolve the path and execute our script.

We are intentionally using the seemingly redundant ./ prefix to tell the shell to run the script from the current working directory.

When the first element does not contain a ‘/,’ the shell will check if it is one of the following:

  • a shell function
  • a shell built-in command or reserved word
  • an external command

Shell Functions

Shell functions are (mainly) customized shell behavior declared by the user. They look and work like commands.

Note: If you are interested in customizing your shell environment, you can find details in my books “Moving to zsh” and “macOS Terminal and Shell.”

Shell Built-ins

Shell built-in commands cover tasks that are either inherent to the shell or can be performed much faster within the shell than as an external command. These are commands that affect the internal state of the current shell process like cd, alias, or history, or commands that are simpler and faster to implement as built-ins like echo or read.

You rarely need to worry about whether a command is a built-in.

Many built-in commands will have an executable external command file as well. This serves as a ‘fallback’ for the rare situation that the shell built-in is not available.

You can use the (aptly, but confusingly, named) command built-in to determine if a command is built-in or external:

> command -V cd
cd is a shell builtin
> command -V sw_vers
sw_vers is /usr/bin/sw_vers

External Commands

When the command entered is neither a function, nor a built-in, and does not contain a ‘/,’ then the shell will go search for an executable file with that name.

The PATH environment variable determines the locations in the file system and the order in which to search them.

The default PATH on macOS in an interactive shell is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The PATH variable contains a colon-separated list of directories. When you have third-party software installed, or customized your shell configuration, your shell may have additional directories. The default PATH splits into the following five directories:

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

Note: Variable names in the shell are case-sensitive. The variable names MY_VAR and my_var represent different variable and values.
Zsh, however, has the concept of ‘connected variables.’ In zsh, PATH and path are connected variables. The upper-case PATH contains the colon-separated list of directory paths, while the lower-case path is an array of paths. The actual list of directories will be the same and changing one variable will change the other. Zsh has this concept to maintain compatibility with other shells with the PATH, while also allowing you to use array operators on the path.
I will use the colon-separated PATH, because it is more compatible with other shells.

When a user enters a command like this:

> system_profiler

This is neither a function or alias, nor a built-in command. It also does not contain a ‘/.’

The shell will check for the presence of an executable file with the name system_profiler in all the directories given in the PATH. It will start with /usr/local/bin, then /usr/bin, then /bin, until it finally finds a matching file in /usr/sbin.

Then the shell will attempt to execute /usr/sbin/system_profiler.

When no matching files can be found in any of the directories listed in the PATH, the shell will present a ‘command not found’ error.

> cantFindMe
zsh: command not found: cantFindMe

If you are curious which file the shell will use for a given command you can use the command or the which tool:

> command -V system_profiler
system_profiler is /usr/sbin/system_profiler
> which system_profiler
/usr/sbin/system_profiler

PATH precedence

Once the shell finds a matching file, it will stop searching the remaining paths. If there were a second matching executable in /sbin, the shell would never find and execute it. The order of the directory paths given in the PATH variable determines the precedence.

We can test this by placing an executable with a file name matching an existing command in /usr/local/bin. Since /usr/local/bin comes first in the default interactive PATH, the shell should prefer our executable over the default command.

> sudo ditto hello.sh /usr/local/bin/system_profiler

You need administrator privileges to modify the contents of /usr/local/bin. The ditto command preserves all the file’s metadata (like privileges and extended attributes), which makes it preferable to cp for this task.

Then open a new Terminal window. You need to do this because every shell instance will cache or ‘remember’ the lookup for a command to speed up the process later. The shell instance in your current terminal window will remember the last lookup for the system_profiler command. A new Terminal window will start a new ‘fresh’ shell instance, forcing a new lookup:

> which system_profiler
/usr/local/bin/system_profiler
> system_profiler
Hello, World!

Obviously, overriding a system provided command this way can break your workflows and scripts. To be safe, remove our script from /usr/local/bin right away:

> sudo rm /usr/local/bin/system_profiler

There are some situations where overriding a system-provided command is desirable, though. For example, you could install the latest version of bash 5 as /usr/local/bin/bash. And then, when you invoke bash from your interactive terminal, you will launch that version, instead of the outdated version that comes with macOS.

Note: You cannot simply overwrite /bin/bash with the newer version on macOS. The /usr/bin, /bin, /usr/sbin, and /sbin folders are protected by System Integrity Protection and the read-only system volume.

When bash is invoked with the absolute path /bin/bash, the path will need to be updated to use the newer version, though. This includes the shebangs in scripts, and the UserShell attribute in a user’s account record for their default shell.

Extending your tool set

As you get more confident and experienced with scripting, you will assemble a set of scripts that you will use regularly. When you use a script often, it would be nice if the shell recognized them as commands, without having to type the path to them.

To achieve that you can add the directory containing the scripts to the PATH variable. I put my frequently used tools in ~/bin. The name is chosen to be somewhat consistent with the four standard locations for tools, but really does not matter. You can append your tool directory to the PATH with:

> export PATH=$PATH:~/bin

This reads as: replace the value of the environment variable PATH with its current value and append :~/bin. You can verify the new value with

> echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/armin/bin

We added our custom directory to the end of the PATH variable, so you cannot accidentally override system tools.

To prevent other local users from changing your command files, you should set the directory’s privileges, so that only you can access:

> chmod 700 ~/bin

Now you can copy our hello.sh script to that folder. We will rename it to just hello, so it looks more like a command:

> ditto hello.sh ~/bin/hello

Then you can run your script just by typing

> hello
Hello, World!

Most of the scripts you build will not need to be as accessible as this.

Since you do not want to manually change the PATH every time you create a new Terminal window, you should add this line to your shell configuration file:

export PATH=$PATH:~/bin

Context Matters

Leveraging PATH to find and possibly override commands can be a useful and powerful tool. However, when scripting, you have to always remember, that the interactive Terminal environment may not be the environment that your script will run in ‘production.’

This is especially important for scripts run by

  • LaunchDaemons or LaunchAgents
  • AppleScripts
  • applications other than Terminal (such as Xcode)
  • installer packages
  • management systems

When run from any of these environments, the environment the script runs in will be different from your interactive shell environment. Most likely the PATH in any of these environments will be set to the minimal four system folders:

/usr/bin:/bin:/usr/sbin:/sbin

This should work well for most commands and tools. But when you are working with third party software, especially newer versions, then you need to pay very close attention.

When you build scripts for these environments, I recommend setting the PATH at the beginning of your script explicitly to the value you need:

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

When your scripts require tools, include their locations in the PATH. This way, the PATH environment is declared explicitly at the beginning of the script and there can be no confusion.

For the scripts in this series, the default PATH will be sufficient, so we will not need to worry about this.

Next: Lists of Commands