This is an excerpt from my book “Moving to zsh” which is available for order on the Apple Books Store.
One of the advantages of zsh over bash 3 is the support of “associative arrays,” a data structure known as hash tables or dictionaries in other languages.
In associative arrays, you can store a piece of data, or value with an identifying ‘key’. For example, the associative array userinfo has multiple values, each identified with a key:
In zsh, before you can use a variable as an associative array, you have to declare it as one with
declare -A userinfo
This will tell the shell that the userinfo variable is an associative array. You can also use typeset -A as an alternative syntax. You can verify the type of the variable:
% echo ${(t)userinfo}
association
You can then set the key-value pairs of the userinfo associative array individually:
Setting the values for each key is useful in some situations, but can be tedious. You can also set the entire associative array at once. There are two syntaxes for this in zsh:
userinfo=( name armin shell zsh website scriptingosx.com )
This format follows the format ( key1 value1 key2 value2 ...). The other syntax is more verbose and expressive:
When you set the associative array variable this way, you are overwriting the entire array. For example, if you set the userinfo for ‘armin’ like above and then set it later like this, the website key and value pair will have been overwritten as well:
% userinfo=( [name]=beth [shell]=zsh )
% if [[ -z $userinfo[website] ]]; then echo no value; fi
no value
If you want to partially overwrite an existing associative array, while leaving the other key/value pairs intact, you can use the += operator:
You can also use this to loop through all the keys and values of an associated array:
for key value in ${(kv)userinfo}; do
echo "$key -> $value"
done
#output
website -> fishshell.com
shell -> fish
name -> beth
Limitations
Associative arrays have their uses, but are not as powerful as dictionaries in more powerful languages. In zsh, you cannot nest associative arrays in normal arrays, which limits their use for complex data structures.
There is also no functionality to transfer certain file formats, like XML or property lists directly in to associative arrays or back.
Shell scripting was never designed for complex data structures. When you encounter these limitations, you should move “up” to a higher level language, such as Python or Swift.
There are many solutions to get the current logged in user in macOS to use in a shell script. However, the semi-official, “sanctioned” method has always involved a rather elaborate python one-liner, originally published by Ben Toms:
There are various other solutions to get the current user which use stat, who, or last commands. These will work in most situations. But there are edge cases, mostly with Fast User Switching, where these methods don’t return the correct user.
From python to scutil
While this has worked wonderfully and more reliably than other solutions, it always looked inelegant, and added a dependency on the python binary. This used to be fine, as the python binary shipped with macOS. But in the macOS Catalina release notes, we have learned that some, yet undefined, future version of macOS will not include the Python, Ruby, and Perl interpreters any more.
With prescient timing, Erik Berglund figured out a different method, which still accesses the same system framework, but through the scutil command rather than the PyObjC bridge.
Over time, different contributors in the MacAdmins Slack have optimized this command:
Sidenote: the scutil version runs 15–20x faster than the python version. Speed is not usually a major consideration for MacAdmin scripts, but it’s a nice improvement.
While zsh is a great replacement for bash for interactive terminal use and scripting on the ‘full’ macOS system, there are environments (Recovery) where zsh is not present.
The only shell Mac Admins can rely on in all contexts is now /bin/sh.
Because of this I recommend using /bin/sh for installer scripts (pre- and postinstall scripts in pkgs).
The Posix sh standard does not include the ‘here string’ <<< used in Erik’s command. Nevertheless, when you use the above construct in sh on macOS it will still work fine. This is because sh on macOS is actually handled by bash in sh compatibility mode.
And while bash in sh compatibility mode will recreate the odd syntax and quirks of sh for compatibility, it will also happily run ‘bashims’ that don’t even exist in sh. E.g. double brackets[[...]] or here strings <<<.
As long as bash is taking care of sh scripts, you will be fine. Myself, I would not have noticed this if I had not ran a sh script through shellcheck. Shellcheck told me that Posix sh should not understand the here string:
But we’re fine on macOS, because sh is really bash right? No need to worry…
Predicting the demise of bash
We don’t yet know when, but the /bin/bash binary will eventually be removed from macOS. The version included with macOS has not been updated since 2014. It is unlikely that the twelve year old bash v3.2 will receive patches for security vulnerabilities in the future.
With the switch to zsh as the default shell in Catalina Apple is laying the groundwork to remove bash v3.2 from the system in a future release. cron and login hooks were infamously deprecated in macOS years ago and are still around, so we could be looking at years. But my suspicion is that bash is just one security vulnerability away from being removed.
With the High Sierra and Mojave updates, Apple has shown that they do not need to wait for major releases to remove or add functionality or restrictions to the system.
I believe /bin/bash has more than a few months still. However, Apple’s messaging on this switch is uncharacteristically strong. It would not be terribly surprising if the Catalina Spring Update in March 2020 removed bash.
The amount of old installer packages which would break on such a change is terrifying. Apple might be willing to pay this price, to avoid a publicized security vulnerability. A severe bash vulnerability would definitely get a lot of publicity. Remember Shellshock? That was the last time bash v3.2 was patched.
Instead of removing /bin/bash Apple might be able to replace it with zsh in bash emulation mode. So far, I have found no signs for this. While it wouldn’t be perfect, it would be a safer move.
Python 2.7 will still get updates until the end of 2019. There are also some commands and tools in the system written in python 2.7. It will be harder for Apple to migrate these to other languages, so I think Python has a longer countdown clock. But it cannot hurt to start preparing now.
To test script compatibility with Bourne-compatible shells in macOS Catalina, you can change /var/select/sh to /bin/bash, /bin/dash, or /bin/zsh. If you change /var/select/sh to a shell other than bash, be aware that scripts that make use of bashisms may not work properly.
zsh can be made to emulate sh by executing the command zsh --emulate sh.
We can actually see that Apple has two options here. sh processes could be run by zsh in emulation mode or by dash.
dash is the Debian Almquist shell, which is a minimal implementation of the Posix sh standard. dash is a newcomer to macOS on Catalina. New arrivals on a release where Apple seems intent on cleaning out ballast (32-bit, Kernel Extensions, bash v3, Python, Perl, and Ruby) are immediately of interest.
When you check on Catalina Recovery system, you can see that /bin/dash has been added, but /bin/zsh is absent. (/bin/bash is still present on Catalina Recovery and the sh binary is also still bash.)
Putting all of this together, I would hazard that Apple is planning to use dash to take over from ‘bash as sh.’ This will also bring macOS in line with most other Unix and Linux distributions where dash commonly handles sh processes.
With the symbolic link mechanism described in the support article, Apple could switch sh to dash in an update while bash is still present. This could allow Developers or Mac Admins to switch their managed systems back to bash when they need to mitigate problems.
When Apple switches to dash, sh scripts that use bashisms will break.
You can start testing this in Catalina by changing the symbolic link at /var/select/sh:
% sudo ln -sf /bin/dash /var/select/sh
(Change /bin/dash to /bin/bash to revert to default.)
When I test the above command to get the current user with dash, it returns:
Syntax error: redirection unexpected
(You can also change the shebang of a script you want to test to #!/bin/dash then you do not have to change the system sh.)
Getting the Current User in sh, Future Proof (I hope)
The solution is easy enough. Replace the here string with a pipe. For sh scripts, change the syntax to:
This will make Posix sh, dash, and shellcheck happy and will still work in bash and zsh.
This syntax (echoing into a pipe) is considered inelegant and inefficient, because it requires an additional process (echo) and an additional pipe. Here docs and here strings were introduced exactly to avoid this. Obviously, if you prefer, you can continue to use the here string form of the command for bash and zsh.
Conclusion
There is no need panic and search and replace the python command in all your scripts immediately. I would recommend replacing this line when you update a script. Keep all of this in mind when working on your administration, installation and workflow scripts going forward.
You may also find scripts in third party documentation and installers that need updating, and you should let the author or vendor know.
/bin/bash (v3.2) is deprecated and has not received updates since 2014
/usr/bin/python (v2.7) is deprecated and will not receive new updates starting in Jan 2020
both will eventually be removed from macOS
we don’t know when exactly, could be years, could be months
even when you do not use Python for scripting, you may be using Python ‘one-liners’ in shell scripts
bash scripts should be moved to zsh (general use) or sh (general and installation scripts)
sh processes in macOS will likely be handled by dash in the future
use shellcheck to find and correct bashisms in sh scripts
Someone on the MacAdmins Slack recently asked how you could assign a global keyboard short cut to open Terminal on macOS.
Note: alternative terminal applications such as iTerm2 may have this built-in.
macOS has an option to assign custom global keystrokes to pretty much anything, but it is not obvious how to get there.
First, open the Automator application. In the chooser for a new Workflow, choose ‘Quick Action’ (on Mojave) or ‘Service’ on earlier versions of macOS.
The new Workflow chooser in Mojave
In the new workflow configure the input to be ‘no input’ and the application to be ‘any application.’
Then search for ‘Launch Application’ action in the library pane on the left and add it to your workflow by double-clicking or dragging.
The popup menu where you can slect an application in the action will only show applications from the /Applications folder. Choose ‘Other…’ and select Terminal in the ’/Applications/Utilities` folder.
Configure your workflow
Save the workflow. Give it a meaningful name such as ‘Open Terminal.’ Since you chose Quick Action or Service, this workflow will be saved in ~/Library/Services.
Open System Preferences > Keyboard. Click the ‘Shortcuts’ tab and select ‘Services’ from the list on the left side. (Even on Mojave, it is still called ‘Services’.)
Scroll all the way down the list of services under the ‘General’ heading, you should find the service you just created. Select it and click ‘Add Shortcut’ to assign a global shortcut.
Keyboard Shortcut Preferences
You are done!
When the active application uses the same keystroke, the application’s definition will precede your global shortcut.
Of course, you don’t have stop at launching applications. You can assign a global keyboard shortcut to any Automator workflow this way. Since Automator workflows can include AppleScript, Python or shell scripts, you can do pretty much anything this way!
However, most Apple users don’t bother with shortcuts to launch apps. Just invoke Spotlight with command-space and start typing term and hit return.
Pro Warehouse is extending their services with training for Mac IT specialists. We call it the ‘Pro Academy.’ As part of that, we are going to offer two-day Scripting macOS classes!
The point of this class is to overcome the first hurdles of a very daunting and complex topic. You will learn the basics of bash scripting (and debugging). The goal you can start scripting and gain experience on your own, without getting into too much trouble. The class is designed specifically for Mac adminstrators, so most of the examples and exercises will have direct application for Mac system management. You will learn both scripting and some useful adminstration tools.
Later, we will also offer an “Advanced Scripting macOS” class, that builds on the first, which will go deeper and address more complex topics.
The Scripting classes are designed and held by myself. I do have to thank the entire team at Pro Warehouse for the amazing effort everybody put in to make this happen.
The Scripting classes (and our other management classes) will be offered in our new training facility at the Pro Warehouse offices in Amsterdam! Classes will be offered in English, so international participants are welcome!
In a past post, I described how path_helper works. As an example, I mentioned the installer for Python 3 which runs a postinstall script that locates and modifies the current user’s shell profile file to add the Python 3 binary directory to the PATH.
Not only is modifying a user’s file a horrible practice, but it will not achieve the desired purpose when the user installing the package is ultimately not the user using the system. This setup happens fairly frequently in managed deployment workflows.
As described in the earlier post, macOS will add the contents of files in /etc/paths.d/ to all users’ PATHs. So, all we have to do is create a file with the path to the Python 3 binary directory in /etc/paths.d/. A perfect task for a simple installer package.
The steps to create such an installer are simple:
$ mkdir -p Python3PathPkg/payload
$ cd Python3PathPkg
$ echo "/Library/Frameworks/Python.framework/Versions/3.7/bin" > payload/Python-3.7
$ pkgbuild --root payload --install-location /private/etc/paths.d --version 3.7 --identifier com.example.Python3.path Python3Path-3.7.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to Python3Path-3.7.pkg
This is not so hard. However, since the path to binary contains the major and minor version number, you will have to create a new version when Python 3 updates to 3.8 (and 3.9, etc…).
So, it makes sense to script the process. With a package this simple, you can create everything required to build the package (i.e. the payload folder with contents) from the script in a temporary directory and then discard it after building.
Note: when you modify the PATH with path_helper, your additions will be appended. The Python 3 installer prepends to the PATH. This might lead to slightly different behavior, as the Python 3 behavior overrides any system binaries. If you want to prepend for every user, you have to modify the /etc/paths file.
There are a few other simple installers where this approach makes sense. I also made a script that builds a package to create the .AppleSetupDone file in /var/db to suppress showing the setup assistant at first boot. Since I was planning to use this with the startosinstall --installpackage option, this script builds a product archive, rather than a flat component package.
You could create this package once and hold on to it whenever you need it again, but I seem to keep losing the pkg files. The script allows you to easily re-build the package in a different format or sign it when necessary. Also, dealing with the invisible file is a bit easier when you just create them on demand.
The last example creates a single invisible file .RunLanguageChooserToo, also in /var/db/. This will show an additional dialog before the Setup Assistant to choose the system language. MacAdmins might want to have this dialog for the obvious reason, but it also allows for a useful hack. When you invoke the Terminal at the Language Chooser with ctrl-option-command-T it will have root privileges, which allows some interesting workflows.
With this package the creation of the flag file happens too late to actually show the chooser. So I added the necessary PackageInfo flags to make the installer require a restart. Note that startosinstall will only respect this flag with a Mojave installer, not with High Sierra.
These three scripts can be used as templates for many similar use cases. As your needs get more complex, you should move to pkgbuild scripts with a payload folder, munkipkg, or Whitebox Packages.
As MacAdmins our goal is to automate workflows when ever possible. The advantages of automation are great. You immediately reduce the workload, but also reduce the potential for someone to make a mistake, which would mean even more work later.
In nearly all cases, we want the automation to happen “magically” in the background, without any user interaction. User interaction slows down the process as the script is waiting for the user to confirm or enter data. User input also requires validating and checking user entered data.
In most cases, when you believe you need to prompt the user for, well, anything, you should take that moment to re-think your workflow. Maybe you can come up with a different workflow that can provide the data from a source that can be automated without user interaction.
If, however, you really are convinced that user interaction is necessary, then you need be aware there are many potential pitfalls in writing shell scripts with user interaction.
macOS Mojave will introduce even more pitfalls with its increased security features.
Do you really need UI?
A common example for user interaction is to get and set the Computer Name or Asset Tag/ID/Number from manual input.
This is a task that can be fully automated in many situations. As an admin you could provide a text file mapping serial numbers to a name and/or asset tag on a server, that a script can download (curl) and parse. If you have a management system or asset database, there is probably an XML or JSON API, you can use to retrieve this information from a script. You could read or scan the serial numbers from the labels of the Mac’s boxes or even get the list with the purchase order from most vendors. With that data you can pre-fill your text files or databases.
However, in some cases, especially DEP workflows it may be difficult or impossible to predict which computer will end up on which desk, especially with zero-touch deployment workflows, where a device can be sent to a user directly.
In this case, you have to question whether you need a unique, specific computer name or asset tag. Maybe the data which your management system already gathers, such as the user who enrolled the device and its serial number will (have to) be sufficient?
In many cases, however, the reasons to prompt for and set a computer name will not be technical but stem from other, external factors that the IT department may or may not have influence on.
AppleScript’s Moment of Glory
bash was built to run in a text based terminal on many different operating systems. It has (and should have) no concept of a graphical user interface. bash alone is not useful to interact with the user, unless you want to open a terminal window.
However, AppleScript does have some (basic) commands to present dialogs and notifications to the user. We can call AppleScript command from the shell with the osascript command (OSA = open scripting architecture, the underlying framework that AppleScript uses)
There are other tools that allow to display user interface from a shell script. However, they all require an additional installation. AppleScript/osascript is simpler and built-in to the OS, so it is my first choice. That doesn’t mean it is always the appropriate choice, though.
You can go to Terminal and make a dialog appear with
$ osascript -e 'display dialog "Hello from bash!"'
The display dialog AppleScript command is documented in the “StandardAdditions” dictionary. You can see it when you choose ‘Open Dictionary…’ from the File Menu in Script Editor. Then select the “Scripting Additions.osax” dictionary and choose the “User Interaction” category.
This command has a lot of options that allow us to configure the dialog. You can experiment and test with these commands in the Script Editor application. For example, you can change the names of the buttons:
display dialog "Are you sure!?" buttons {"No", "Yes"}
display dialog "The answer is C" buttons {"A", "B", "C"} default button 3
button returned:C
The script will return which button the user clicked. However, instead of parsing the output with shell tools, you should let AppleScript do the work:
button returned of (display dialog "Are you sure!?" buttons {"No", "Yes"})
The choice variable will be No or Yes.
Adding icons
You can also add an icon to the dialog:
display dialog "Hello" with icon note
Will show the current application’s icon. You can also use stop or caution for different icons.
You can also add a path to an icns file:
display dialog "Hello!" with icon POSIX file "/Applications/Notes.app/Contents/Resources/AppIcon.icns"
Asking for Input
In some cases you want to get information back from the user. You can add a text field to the dialog with the default answer argument. The default answer can be an empty string:
display dialog "Who are you?" default answer "nobody"
display dialog "Who are you?" default answer ""
You can get the result of the dialog with the text returned property:
text returned of (display dialog "Who are you?" default answer "nobody")
You can combine the default answer argument with all the above arguments as well.
Notifications
In some case you just want to notify the user that something happened, and not stall everything while the script waits for confirmation. You can use display notification for these situations:
display notification "Hello, again" with title "Hello"
Running from Shell
You can execute AppleScript from the shell with the osascript command.
osascript -e 'display dialog "Hello!" with icon note'
This works well enough. You also use shell variable substitution:
title='Hey, there!'
osascript -e "display dialog \"What's up?\" with title \"$title\""
When you use variable substitution, you have to use double quotes for the command strings. And then you have to escape any additional double quotes in the AppleScript command. With more complex commands and arguments this will get unwieldy very quickly.
In a bash script you can use a here document instead:
title='Hey, there!'
osascript <<-EndOfScript
display dialog "What's Up?" with title "$title"
EndOfScript
You start a here document with the <<- characters followed by a limit string of your choice (I chose EndOfScript). Then all the text until the limit string is repeated will be fed into the osascript command.
Bash variables will be substituted in the here document, so $title will be substituted with its contents.
Note: I prefer the <<- syntax over the << syntax with osascript. The <<- syntax will ignore leading white space in the following lines, allowing me to properly indent the AppleScript code, making the entire script more legible.
The Trouble with root
Management scripts will run with root privileges. They may also be run in situations where no user is logged in. AppleScript requires a user to be logged in (and a window server to be present) to display the alert.
In many situations all of this will ‘just work’ even when the script is executed from a root process, in others, the script will fail. It is generally safer to always check the current user and run the osascript command as the currently logged in user. You can learn about how and why to do this in this article on ‘Root and Scripting’.
This also gives your script an option to continue silently or fail when no user is logged in.
user=$(python -c 'from SystemConfiguration import SCDynamicStoreCopyConsoleUser; import sys; username = (SCDynamicStoreCopyConsoleUser(None, None, None) or [None])[0]; username = [username,""][username in [u"loginwindow", None, u""]]; sys.stdout.write(username + "\n");')
if [[ $user != "" ]]; then
uid=$(id -u "$user")
launchctl asuser $uid /usr/bin/osascript -e "button returned of (display dialog \"Hello\")"
fi
Putting it all Together
While all of these examples are simple enough, you can already see that, once you consider all the possible combinations, everything will get fairly complex.
Over time I have put together a few bash functions that I use in my scripts. Even they only cover quite simple workflows, but can be useful as a sample for more complex needs:
System Events, Privacy and macOS Mojave
In many scripts the author wraps the display dialog command wrapped in a tell statement:
osascript -e 'tell app "System Events" to display dialog "Hello"'
The reason for this is that it will ensure that the dialog is displayed on top of all other windows. The activate command will push the targeted process to the front. Both “System Events” and “Finder” are used frequently.
In most cases this is not necessary, as the dialog will properly display on top of all other windows, even as a standalone command. There may be some weird conditions when other applications are launched in the same time frame, though. In other cases it may be better to use display notification, anyway.
In macOS Mojave, Apple Events (AppleScript Commands) can only be sent to another application with user permission.
When you try this command in macOS Mojave, the user will be prompted to allow the Terminal application to control the System Events application
If the user denies this request, the osascript command will fail with an error:
$ osascript -e 'tell app "System Events" to display dialog "Hello"'
28:50: execution error: Not authorized to send Apple events to System Events. (-1743)
Once a user has denied access, they will not be prompted again. They will have to go to the ‘Privacy’ tab in the ‘Security & Privacy’ pane in System Preferences, search for ‘Automation’ and allow access for Terminal.
However, management scripts will usually not be executed from Terminal, but from within the context of an installation script or management agent. It may be possible to pre-approve configurations with a UAMDM configuration profile, but it will be impossible to anticipate all contexts in which your scripts may run.
For macOS Mojave it will be better to either modify your script to not wrap the command in a tell statement. Alternatively, you can use a different solution for the UI altogether. (e.g. jamfHelper, Yo, Pashua or CocoaDialog)
All of these wrapped commands in scripts will break in macOS Mojave!
MacAdmins need to go through all their management scripts and check for AppleScript UI commands wrapped in tell statements.
When you use the standalone display dialog it will be the AppleScript process itself that displays the dialog. This requires no specific permission.
Even if your dialogs may not appear on top of all the windows, this is a preferable solution.
This will also affect scripts using osascript to control or receive data from other applications.
Summary
question and revisit every use of user interaction in your workflow
when you really have to, you can use osascript with the display AppleScript commands
to be safe, run the commands as the current user
increased macOS Mojave security will require you to verify all your scripts using osascript
