New iPhones! New Apple Watch! Even a new entry level iPad! The Apple iPhone event delivered mostly within expectations. I am not going to link all the detailed posts, you can go to Apple’s Newsroom for the official details.
We got release dates for iOS 13 (Sep 19), iOS 13.1 (Sep 30), iPadOS (Sep 30, unclear if this is iPadOS 13.0 or 13.1, but likely the latter) and macOS 10.15 Catalina (a vague ‘October’). (Check this MacStories post with all the dates, including hardware and services.)
So, there are a few extra weeks this year to test and prepare for the next macOS upgrade. It’ll be a busy fall either way. Make the most of it: Test, prepare, file bugs and feedback!
Eric Holtam: “Notice: Do not try and restore BridgeOS on a machine running a beta OS with Configurator 2. Configurator doesn’t allow backward updates to previous BridgeOS versions and ”bricks“ the machine. ‘AMRestoreErrorDomain error 79 – backwards update not allowed: 17P50541b -> 16PXXX)’ ”
William Smith: “UltraThin Updates take 2: For tomorrow’s Microsoft Office for Mac update, Microsoft AutoUpdate will use UltraThin updates to move 16.28 > 16.29. Should be around 176 MB to update all of Office.”
There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!
If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!
The upcoming macOS 10.15 Catalina will require more apps and tools to be notarized. Apple has somewhat loosened the requirements at last minute, but these changed limitations are only temporary, to give developers more time to adapt.
Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.
Howard Oakley has written up his experiences and that post was extremely helpful. But there were a few omissions and some steps that aren’t really necessary, so I decided to make my own write-up.
And yes, there is a script at the end…
Note: these instructions are for macOS 10.14.6 Mojave, Xcode 10.3 and Swift 5.0. It is very likely that the details will change over time.
Update 2019-09-24: Tested with Xcode 11 and it still works (the screen layout has changed for some of the options)
What do you need?
Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates)
Xcode 10.3 or 11
Developer ID Certificates (Application and Install)
Application Specific Password for your Developer account
a Command Line Tool Project that you want to sign and notarize
That’s a longish list. If you are already building command line tools in Xcode, you should have most of these covered already. We will walk through the list step-by-step:
You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.
Xcode
You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.
Developer ID Certificates
There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.
To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.
We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.
If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.
Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:
% security find-identity -p basic -v
This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.
You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"
The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.
Application Specific Password for your Developer Account
Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.
Create a new application specific password in Apple ID portal for your developer account.
You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:
Keychain Item Name: Developer-altool
Account Name: your developer account email
Password: the application-specific password you just created
This will create a developer specific password item that we can access safely from the tools.
If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.
A Command Line Tool Project
You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.
My sample project for this article will be named “hello.”
Preparing the Xcode Project
The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.
Choosing the proper signing certificates
Before you can notarize the command line tool, it needs to be signed with the correct certificates.
in Xcode, select the blue project icon in the left sidebar
select the black “terminal” icon with your project’s name under the “Targets” list entry
make sure the ‘General’ tab is selected
under ‘Signing’ disable ‘Automatically manage signing’
under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Setting the certificates
Enable Hardened Runtime
Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.
from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
click on ‘All’ to show all available settings
enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
set the value in the project column (blue icon) to YES
Enable Hardened Runtime
Change the Install Build Location
If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:
in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
double click on the value in the Project column (blue icon), this will open a popup window
change the value to $SRCROOT/build/pkgroot
Change the Installation Build location
If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)
Build the Binary
You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:
open Terminal and change directory to the project folder
% xcodebuild clean install
This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.
After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.
/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.
Build the pkg
Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.
We can use the pkgroot directory as our payload to build the installer package:
I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.
This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.
If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”
Notarizing the Installer Package
Xcode has a command line tool altool which you can use to upload your tool for notarization:
The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.
The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)
This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:
Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.
Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.
Completing the Process
You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.
If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.
You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:
% xcrun stapler staple build/hello-1.0.pkg
You can also use stapler to verify the process went well:
% xcrun stapler validate build/hello-1.0.pkg
But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:
Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.
The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.
You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.
Links and Videos
These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.
Notarization is a key part of Apple’s security strategy going in macOS.
As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.
If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.
On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.
Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.
You can tell from the various articles and support posts that release day is definitely getting closer.
iOS 10.13.1 beta 2 was release and it is questionable if we will see another beta of either iOS, iPadOS or macOS before release. Apple has loosened notarization requirements for macOS Catalina temporarily (until Jan 2020) to help developer handle the transition.
Rich Trouton’s and Charles Edge’s “Apple Device Management: A Unified Theory of Managing Macs, iPads, iPhones, and AppleTVs” book is available for pre-order on Amazon US, UK, DE (Affiliate Links)
Mike Boylan:
“This is true. Age != skill (in both directions). Am fortunate that most people have treated me with respect.” (Thread)
Jason Fried: “When Google puts 4 paid ads ahead of the first organic result for your own brand name, you’re forced to pay up if you want to be found. It’s a shakedown. It’s ransom. But at least we can have fun with it. Search for Basecamp and you may see this attached ad.” (Image)
Firefox 69.0 “For Enterprise system administrators that manage macOS computers, we begin shipping a Mozilla signed PKG installer to simplify your deployments.” (Pkg download here.)
There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!
If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!
There are many solutions to get the current logged in user in macOS to use in a shell script. However, the semi-official, “sanctioned” method has always involved a rather elaborate python one-liner, originally published by Ben Toms:
There are various other solutions to get the current user which use stat, who, or last commands. These will work in most situations. But there are edge cases, mostly with Fast User Switching, where these methods don’t return the correct user.
From python to scutil
While this has worked wonderfully and more reliably than other solutions, it always looked inelegant, and added a dependency on the python binary. This used to be fine, as the python binary shipped with macOS. But in the macOS Catalina release notes, we have learned that some, yet undefined, future version of macOS will not include the Python, Ruby, and Perl interpreters any more.
With prescient timing, Erik Berglund figured out a different method, which still accesses the same system framework, but through the scutil command rather than the PyObjC bridge.
Over time, different contributors in the MacAdmins Slack have optimized this command:
Sidenote: the scutil version runs 15–20x faster than the python version. Speed is not usually a major consideration for MacAdmin scripts, but it’s a nice improvement.
While zsh is a great replacement for bash for interactive terminal use and scripting on the ‘full’ macOS system, there are environments (Recovery) where zsh is not present.
The only shell Mac Admins can rely on in all contexts is now /bin/sh.
Because of this I recommend using /bin/sh for installer scripts (pre- and postinstall scripts in pkgs).
The Posix sh standard does not include the ‘here string’ <<< used in Erik’s command. Nevertheless, when you use the above construct in sh on macOS it will still work fine. This is because sh on macOS is actually handled by bash in sh compatibility mode.
And while bash in sh compatibility mode will recreate the odd syntax and quirks of sh for compatibility, it will also happily run ‘bashims’ that don’t even exist in sh. E.g. double brackets[[...]] or here strings <<<.
As long as bash is taking care of sh scripts, you will be fine. Myself, I would not have noticed this if I had not ran a sh script through shellcheck. Shellcheck told me that Posix sh should not understand the here string:
But we’re fine on macOS, because sh is really bash right? No need to worry…
Predicting the demise of bash
We don’t yet know when, but the /bin/bash binary will eventually be removed from macOS. The version included with macOS has not been updated since 2014. It is unlikely that the twelve year old bash v3.2 will receive patches for security vulnerabilities in the future.
With the switch to zsh as the default shell in Catalina Apple is laying the groundwork to remove bash v3.2 from the system in a future release. cron and login hooks were infamously deprecated in macOS years ago and are still around, so we could be looking at years. But my suspicion is that bash is just one security vulnerability away from being removed.
With the High Sierra and Mojave updates, Apple has shown that they do not need to wait for major releases to remove or add functionality or restrictions to the system.
I believe /bin/bash has more than a few months still. However, Apple’s messaging on this switch is uncharacteristically strong. It would not be terribly surprising if the Catalina Spring Update in March 2020 removed bash.
The amount of old installer packages which would break on such a change is terrifying. Apple might be willing to pay this price, to avoid a publicized security vulnerability. A severe bash vulnerability would definitely get a lot of publicity. Remember Shellshock? That was the last time bash v3.2 was patched.
Instead of removing /bin/bash Apple might be able to replace it with zsh in bash emulation mode. So far, I have found no signs for this. While it wouldn’t be perfect, it would be a safer move.
Python 2.7 will still get updates until the end of 2019. There are also some commands and tools in the system written in python 2.7. It will be harder for Apple to migrate these to other languages, so I think Python has a longer countdown clock. But it cannot hurt to start preparing now.
To test script compatibility with Bourne-compatible shells in macOS Catalina, you can change /var/select/sh to /bin/bash, /bin/dash, or /bin/zsh. If you change /var/select/sh to a shell other than bash, be aware that scripts that make use of bashisms may not work properly.
zsh can be made to emulate sh by executing the command zsh --emulate sh.
We can actually see that Apple has two options here. sh processes could be run by zsh in emulation mode or by dash.
dash is the Debian Almquist shell, which is a minimal implementation of the Posix sh standard. dash is a newcomer to macOS on Catalina. New arrivals on a release where Apple seems intent on cleaning out ballast (32-bit, Kernel Extensions, bash v3, Python, Perl, and Ruby) are immediately of interest.
When you check on Catalina Recovery system, you can see that /bin/dash has been added, but /bin/zsh is absent. (/bin/bash is still present on Catalina Recovery and the sh binary is also still bash.)
Putting all of this together, I would hazard that Apple is planning to use dash to take over from ‘bash as sh.’ This will also bring macOS in line with most other Unix and Linux distributions where dash commonly handles sh processes.
With the symbolic link mechanism described in the support article, Apple could switch sh to dash in an update while bash is still present. This could allow Developers or Mac Admins to switch their managed systems back to bash when they need to mitigate problems.
When Apple switches to dash, sh scripts that use bashisms will break.
You can start testing this in Catalina by changing the symbolic link at /var/select/sh:
% sudo ln -sf /bin/dash /var/select/sh
(Change /bin/dash to /bin/bash to revert to default.)
When I test the above command to get the current user with dash, it returns:
Syntax error: redirection unexpected
(You can also change the shebang of a script you want to test to #!/bin/dash then you do not have to change the system sh.)
Getting the Current User in sh, Future Proof (I hope)
The solution is easy enough. Replace the here string with a pipe. For sh scripts, change the syntax to:
This will make Posix sh, dash, and shellcheck happy and will still work in bash and zsh.
This syntax (echoing into a pipe) is considered inelegant and inefficient, because it requires an additional process (echo) and an additional pipe. Here docs and here strings were introduced exactly to avoid this. Obviously, if you prefer, you can continue to use the here string form of the command for bash and zsh.
Conclusion
There is no need panic and search and replace the python command in all your scripts immediately. I would recommend replacing this line when you update a script. Keep all of this in mind when working on your administration, installation and workflow scripts going forward.
You may also find scripts in third party documentation and installers that need updating, and you should let the author or vendor know.
/bin/bash (v3.2) is deprecated and has not received updates since 2014
/usr/bin/python (v2.7) is deprecated and will not receive new updates starting in Jan 2020
both will eventually be removed from macOS
we don’t know when exactly, could be years, could be months
even when you do not use Python for scripting, you may be using Python ‘one-liners’ in shell scripts
bash scripts should be moved to zsh (general use) or sh (general and installation scripts)
sh processes in macOS will likely be handled by dash in the future
use shellcheck to find and correct bashisms in sh scripts
Things are heating up for September release season. We got new betas for iOS and iPadOS 13.0 and macOS Catalina beta7. Surprisingly, there also is a beta for iOS 13.1, which includes some features shown at WWDC, which were removed from the 13.0 beta. While I generally approve of this flexibility, it makes testing confusing and more complex.
We also got a second supplemental update for 10.14.6, fixing some more bugs and security holes. The newest 10.14.6 has the build number 18G95. The silver lining in all this version chaos is, that we will likely not end Mojave on a forked hardware build.
Apple sent out the invitations to the iPhone event on September 10, 10am PDT (19:00 CEST). If previous years can be used as a guide, this could mean the iOS 13.0 release will be on September 13 or 20 and the macOS 10.15.0 Catalina release on September 20 or 27.
There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!
If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!
The release notes for Catalina also tell us that other built-in scripting runtimes, namely Python, Perl, and Ruby. Will not be included in future macOS releases (post-Catalina) any more.
This means, that if you want to use bash, Python, Perl, or Ruby on macOS, you will have to install, and maintain your own version in the future.
However, scripts in installation packages, cannot rely on any of these interpreters being available in future, post-Catalina versions of macOS. Installer pkgs can be run in all kinds of environments and at all times, and you would not want them to fail, because a dependency is missing.
The good news is that we still have time. All the runtimes mentioned above are still present in Catalina, so the packages will continue to work for now. But if you are building installation scripts, you need to check if any of the installation scripts use one of these interpreters and fix them.
I recommend to use /bin/sh for installation scripts, since that will run in any macOS context, even the Recovery system.
If you are using third-party installer packages, you may also want to check them for these interpreters, and notify the developer that these packages will break in future versions of macOS.
To check a flat installer package, you would expand it with pkgutil --expand and then look at script files in the Scripts folder. This will work fine for a package or two, but gets tedious really quickly, especially with large distribution pkgs with many components (e.g. Office).
So… I wrote a script to do it. The script should handle normal component pkgs, distribution pkgs and the legacy bundle pkgs and mpkgs.
Once I had written the code to inspect all these types of pkgs, I realized I could grab all other kinds of information, as well. The pkgcheck.sh script will check for:
Signature and Notarization
Type of Package: Component, Distribution, legacy bundle or mpkg
Identifier and version (when present)
Install-location
for Distribution and mpkg types, shows the information for all components as well
for every script in a pkg or component, checks the first line of the script for shebangs of the deprecated interpreters (/bin/bash, /usr/bin/python, /usr/bin/perl, and /usr/bin/ruby) and print a warning when found
How to run pkgcheck.sh
Run the script with the target pkg file as an argument:
% ./pkgcheck.sh sample.pkg
You can give more than one file:
% ./pkgcheck.sh file1.pkg file2.pkg ...
When you pass a directory, pkgcheck.sh will recursively search for all files or bundle directories with the pkg or mpkg extension in that directory:
% ./pkgcheck.sh SamplePkgs
Features and Errors
There are a few more things that I think might be useful to check in this script. Most of all, I want to add an indicator whether a component is enabled by default or not. If you can think of any other valuable data to display, let me know. (Issue or Pull Request or just ping me on MacAdmins Slack)
I have tested the script against many pkgs that I came across. However, there are likely edge cases that I haven’t anticipated, which might break the script. If you run into any of those, let me know. (File an Issue or Pull Request.) Having the troublesome pkg would of course be a great help.
Note: the script will create a scratch directory for temporary file extractions. The script doesn’t actually expand the entire pkg file, only the Scripts sub-archive. The scratch folder will be cleaned out at the beginning of the next run, but not when the script ends, as you might want to do some further inspections.
Sample outputs
This is a sample pkg I build in my book, it has pre- and postinstall scripts using a /bin/bash shebang:
Summer vacation time is coming to an end and the beta season for macOS Catalina and iOS 13 is heating up! Another Catalina beta dropped this week, and third parties are now starting to announce their own betas and/or preliminary support.
SimpleMDM: “Bootstrap Tokens, new to macOS 10.15 Catalina, are available in SimpleMDM. Begin testing your Automated Enrollment (DEP) flows with SecureToken configuration today.”
William Smith: “Excited for this release from Microsoft because it’ll be the first app they’ve ever released for Mac that’s fully manageable from day 1. There’s been a lot of outreach the past few years from them to MacAdmins to make something like this possible.”
Mike Stern: “SF Mono is now available! Also, SF Pro and SF Compact have been updated to reflect some additions and changes to SF Symbols. developer.apple.com/fonts”
There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!
If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!
This is the final article in this series. (If I ever announce an eight part series again, please somebody intervene!) However, I am quite sure it will not be the last post on zsh
All the previous posts described how zsh works as an interactive shell. The interactive shell is of course the most direct way we use a shell and configuring the shell to your taste can bring a huge boost in usefulness and productivity.
The other, equally, important aspect of a shell is running script files. In the simplest perspective, script files are just series of interactive commands, but of course they will get complex very quickly.
sh, bash, or zsh?
Should you even script in zsh? The argument for bash has been that it has been pre-installed on every Mac OS X since 10.2. The same is true for zsh, with one exception: the zsh binary is not present on the Recovery system. It is also not present on a NetInstall or External Installation System, but these are less relevant in a modern deployment workflow, which has to work with Secure Boot Macs.
If you plan to run a script from Recovery, such as an installr or bootstrappr script or as part of an MDS workflow, your only choices are /bin/sh and /bin/bash. The current /bin/bash binary is 12 years old, and Apple is messaging its demise. I would not consider that a future proof choice. So, if your script may run in a Recovery context, i would recommend /bin/sh over either /bin/bash or /bin/zsh
Since installation packages can be run from the Recovery context as well, and you cannot really always predict in which context your package will be used, I would extend the recommendation to use /bin/sh for all installation scripts as well.
While sh is surely ubiquitous, it is also a ‘lowest common denominator’, so it is not a very comfortable scripting language to work in. I recommend using shellcheck to verify all your sh scripts for bashisms that might have crept in out of habit.
When you can ensure your script will only run on a full macOS installation, zsh is good choice over sh. It is pre-installed on macOS, and it offers better and safer language options than sh and some advantages over bash, too. Deployment scripts, scripts pushed from management systems, launch daemons and launch agents, and script you write to automate your admin workflows (such as building packages) would fall in this category.
You can also choose to stick with bash, but then you should start installing and using your own bash 5 binary instead of the built in /bin/bash. This will give you newer security updates and features and good feeling that when Apple does eventually yank the /bin/bash binary, your scripts will keep working.
Admins who want to keep using Python for their scripts are facing a similar problem. Once you choose to use a non-system version of bash (or python), it is your responsibility to install and update it on all your clients. But that is what system management tools are for. We will have to get used to managing our own tools as well as the users’ tools, instead of relying on Apple.
Shebang
To switch your script from using bash to zsh, you have to change the shebang in the first line from #!/bin/bash to #!/bin/zsh.
If you want to distinguish your zsh script files, the you can also change the script’s file extension from .sh to .zsh. This will be especially helpful while you transfer scripts from bash to zsh (or sh). The file extension will have no effect on which interpreter will be used to run the script. That is determined by the shebang, but the extension provides a visible clue in the Finder and Terminal.
zsh vs bash
Since zsh derives from the same Bourne shell family as bash does, most commands, syntax, and control structures will work just the same. zsh provides alternative syntax for some of the structures.
zsh has several options to control compatibility, not only for bash, but for other shells as well. We have already seen that options can be used to enable features specific for zsh. These options can significantly change how zsh interprets your scripts.
Because you can never quite anticipate in which environment your particular zsh will be launched in, it is good practice to reset the options at the beginning of your script with the emulate command:
emulate -LR zsh
After the emulate command, you can explicitly set the shell options your script requires.
The emulate command also provides a bash emulation:
emulate -LR bash
This will change the zsh options to closely emulate bash behavior. Rather than relying on this emulation mode, I would recommend actually using bash, even if you have to install and manage a newer version yourself.
Word Splitting in Variable Substitutions
Nearly all syntax from bash scripts will ‘just work’ in zsh as well. There are just a few important differences you have to be aware of.
The most significant difference, which will affect most scripts is how zsh treats word splitting in variable substitutions.
Recap: bash behavior
In bash substituted variables are split on whitespace when the substitution is not quoted. To demonstrate, we will use a function that counts the number of arguments passed into it. This way we can see whether a variable was split or not:
#!/bin/bash
export PATH=/usr/bin:/bin:/usr/sbin:/sbin
function countArguments() {
echo "${#@}"
}
wordlist="one two three four five"
echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 5
echo "substitution with quotes"
countArguments "$wordlist"
# -> 1
In bash and sh the contents of the variable split into separate arguments when substituted without the quotes. Usually you do not want the splitting to occur. Hence the rule: “always quote variable substitutions!”
zsh behavior: no splitting
zsh will not split a variable when substituted. With zsh the contents of a variable will be kept in one piece:
#!/bin/zsh
emulate -LR zsh # reset zsh options
export PATH=/usr/bin:/bin:/usr/sbin:/sbin
function countArguments() {
echo "${#@}"
}
wordlist="one two three four five"
echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 1
echo "substitution with quotes"
countArguments "$wordlist"
# -> 1
The positive effect of this is that you do not have to worry about quoting variables all the time, making zsh less error prone, and much more like other scripting and programming languages.
Splitting Arrays
The wordlist variable in our example above is a string. Because of this it returns a count of 1, since there is only one element, the string itself.
If you want to loop through multiple elements of a list
In bash this happens, whether you want to or not, unless you explicitly tell bash not to split by quoting the variable.
In zsh, you have to explicitly tell the shell to split a string into its components. If you do this naïvely, by wrapping the string variable in the parenthesis to declare and array, it will not work:
wordlist="one two three"
wordarray=( $wordlist )
for word in $wordarray; do
echo "->$word<-"
done
#output
->one two three<-
Note: the for loop echoes every item in the array. I have added the -> characters to make the individual items more visible. In the subsequent examples, I will not repeat the for loop, but only show its output. So the above example will be shortened to:
wordarray=( $wordlist )
->one two three<-
There are few options to do this right.
Revert to sh behavior
First, you can tell zsh to revert to the bash or sh behavior and split on any whitespace. You can do this by pre-fixing the variable substitution with an =:
Note: if you find yourself using the = frequently, you can also re-enable sh style word splitting with the shwordsplit option. This will of course affect all substitutions in the script until you disable the option again.
This option can be very useful when you quickly need to convert a bash script to a zsh script. But you will also re-enable all the problems you had with unintentional word splitting.
Splitting Lines
If you want to be more specific and split on particular characters, zsh has a special substitution syntax for that:
Switching your scripts from bash to zsh requires a bit more work than merely switching out the shebang. However, since /bin/bash will still be present in Catalina, you do not have to move all scripts immediately.
Moving to sh instead of zsh can be safer choice, especially for package installation scripts.
In zsh, there always seems to be some option to disable or enable a particular behavior.
This concludes my series on switching to zsh on macOS. I hope you found it helpful.
After having worked with zsh for a few weeks, I already find some of its features indispensable. I am looking forward to discovering and using more features over time. When I do, I will certainly share them here.
Denise Yu: “I illustrated my three favourite binaries”
Pepijn Bruienne: “To my surprise we recently updated the Authorization Plugin documentation. Surprise because it’s not seen a lot of change in recent times, but you get TWO new callbacks, RemoveHintValue and RemoveContextValue!”
Mr. Macintosh: “2019–004 for High Sierra 10.13, Sierra 10.12 and BridgeOS have been re released! I will update the main article and reply to this post when I have the new build numbers.”
Rosyna Keller: “The updated Hardened Runtime docs are out! The overview includes more information on how to enable it in Xcode and explains that the hardened runtime is designed to stop certain classes of exploits.” (Thread)
Ronald Oussoren: “The version 6 branch of PyObjC has finally caught up with the current beta: tests pass in macOS 10.15 beta 5 with Python 3.8b3, including bindings for all new frameworks.”
Edward Marczak: “Support local meetups! Particularly in tech: we need more discussion between disparate teams. There’s undoubtedly a meetup near you that matches your style. Support can mean presenting, but also just showing up. Without you, these gatherings go away. With you, they get stronger.”
Timo Elliott: “How to evaluate your existing enterprise infrastructure…” (click for cartoon)
Ed Marczak: “We have a long way to go in exploring this medium. ” (read linked thread)
Support
There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!
If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!
