Notarize a Command Line Tool

There is an updated version of this post for the new tools in Xcode 13.

The upcoming macOS 10.15 Catalina will require more apps and tools to be notarized. Apple has somewhat loosened the requirements at last minute, but these changed limitations are only temporary, to give developers more time to adapt.

Notarizing Mac Application bundles has its pitfalls, but is overall fairly well documented. However, I have been working on some command line tools written in Swift 5 and figured out how to get those properly signed and notarized.

Howard Oakley has written up his experiences and that post was extremely helpful. But there were a few omissions and some steps that aren’t really necessary, so I decided to make my own write-up.

And yes, there is a script at the end…

Note: these instructions are for macOS 10.14.6 Mojave, Xcode 10.3 and Swift 5.0. It is very likely that the details will change over time.

Update 2019-09-24: Tested with Xcode 11 and it still works (the screen layout has changed for some of the options)

What do you need?

  • Apple Developer Account (Personal or Enterprise, the free account does not provide the right certificates)
  • Xcode 10.3 or 11
  • Developer ID Certificates (Application and Install)
  • Application Specific Password for your Developer account
  • a Command Line Tool Project that you want to sign and notarize

That’s a longish list. If you are already building command line tools in Xcode, you should have most of these covered already. We will walk through the list step-by-step:

Apple Developer Program Account

You need either the paid membership in the Apple Developer Program or be invited to an Apple Developer Enterprise Program team with access to the proper certificates.

You cannot get the required certificates with a free Apple Developer account, unless you are member of a team that provides access.

Xcode

You can download Xcode from the Mac App Store or the developer download page. When you launch Xcode for the first time, it will prompt for some extra installations. Those are necessary for everything to in the article to work.

Developer ID Certificates

There are multiple certificates you can get from the Developer Program. By default you get a ‘Mac Developer’ certificate, which you can use for building and testing your own app locally.

To distribute binaries (apps and command line tools) outside of the App Store, you need a ‘Developer ID Application’ certificate. To sign installer packages for distribution outside of the Mac App Store, you need a ‘Developer ID Installer’ certificate.

We will need both types of Developer ID certificates, the first to sign the command line tool and the second to sign and notarize the installer package.

If you have not created these yet, you can do so in Xcode or in the Developer Portal. If you already have the certificates but on a different Mac, you need to export them and re-import them on the new Mac. Creating new certificates might invalidate the existing certificates! So beware.

Once you have created or imported the certificates on your work machine, you can verify their presence in the Terminal with:

% security find-identity -p basic -v

This command will list all available certificates on this Mac. Check that you can see the ‘Developer ID Application’ and ‘Developer ID Installer’ certificates. If you are a member of multiple teams, you may see multiple certificates for each team.

You can later identify the certificates (or ‘identities’) by the long hex number or by the descriptive name, e.g. "Developer ID Installer: Armin Briegel (ABCD123456)"

The ten character code at the end of the name is your Developer Team ID. Make a note of it. If you are a member of multiple developer teams, you can have multiple Developer ID certificates and the team ID will help you distinguish them.

Application Specific Password for your Developer Account

Apple requires Developer Accounts to be protected with two-factor authentication. To allow automated workflows which require authentication, you can create application specific passwords.

Create a new application specific password in Apple ID portal for your developer account.

You will only be shown the password when you create it. Immediately create a ‘New Password Item’ in your Keychain with the following fields:

  • Keychain Item Name: Developer-altool
  • Account Name: your developer account email
  • Password: the application-specific password you just created

This will create a developer specific password item that we can access safely from the tools.

If you want, you can also store the app specific password in a different password manager, but the Xcode tools have a special option to use Keychain.

A Command Line Tool Project

You may already have a project to create a command line in Xcode. If you don’t have one, or just want a new one to experiment, you can just create a new project in Xcode and choose the ‘Command Line Tool’ template from ‘macOS’ section in the picker. The template creates a simple “Hello, world” tool, which you can use to test the notarization process.

My sample project for this article will be named “hello.”

Preparing the Xcode Project

The default settings in the ‘Command Line Tool’ project are suitable for building and testing the tool on your Mac, but need some changes to create a distributable tool.

Choosing the proper signing certificates

Before you can notarize the command line tool, it needs to be signed with the correct certificates.

  1. in Xcode, select the blue project icon in the left sidebar
  2. select the black “terminal” icon with your project’s name under the “Targets” list entry
  3. make sure the ‘General’ tab is selected
  4. under ‘Signing’ disable ‘Automatically manage signing’
  5. under ‘Signing (Debug)’ choose your Team and choose ‘Developer ID Application’ as the certificate
  6. under ‘Signing (Release)’ choose your Team and choose ‘Developer ID Application’ as the certificate
Setting the certificates
Setting the certificates

Enable Hardened Runtime

Enabling the ‘Hardened Runtime’ will compile the binary in a way that makes it harder for external process to inject code. This will be requirement for successful notarization starting January 2020.

  1. from the view where you changed the signing options, click on ‘Build Settings’ in the upper tab row
  2. click on ‘All’ to show all available settings
  3. enter ‘enable hardened’ in the search field, this will show the ‘Enable Hardened Runtime’ setting
  4. set the value in the project column (blue icon) to YES
Enable Hardened Runtime
Enable Hardened Runtime

Change the Install Build Location

If we want to automate the packaging and notarization, we need to know where Xcode builds the binary. The default location is in some /tmp subdirectory and not very convenient. We will change the location for the final binary (the ‘product’) to the build subdirectory in the project folder:

  1. in the same view as above, enter ‘Installation Build’ in the search field, this will show the ‘Installation Build Products Location’ setting
  2. double click on the value in the Project column (blue icon), this will open a popup window
  3. change the value to $SRCROOT/build/pkgroot
Change the Installation Build location
Change the Installation Build location

If you manage your code in git or another VCS, you want to add the build subdirectory to the ignored locations (.gitignore)

Build the Binary

You can use Xcode to write, test, and command line tool debug your. When you are ready to build and notarize a pkg installer, do the following:

  1. open Terminal and change directory to the project folder
  2. % xcodebuild clean install

This will spew a lot of information out to the command line. You will see a build subdirectory appear in the project folder, which will be filled with some directories with intermediate data.

After a successful build you should see a pkgroot directory in the build folder, which contains your binary in the usr/local/bin sub-path.

/usr/local/bin is the default location for command line tools in the Command Line Tool project template. It suits me fine most of the time, but you can change it by modifying the ‘Installation Directory’ build setting in Xcode and re-building from the command line.

Build the pkg

Command Line Tools can be signed, but not directly notarized. You can however notarize a zip, dmg, or pkg file containing a Command Line Tool. Also, it is much easier for users and administrators to install your tool when it comes in a proper installation package.

We can use the pkgroot directory as our payload to build the installer package:

% pkgbuild --root build/pkgroot \
           --identifier "com.example.hello" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Armin Briegel (ABCD123456)" \
           build/hello-1.0.pkg

I have broken the command into multiple lines for clarity, you can enter the command in one line without the end-of-line backslashes \. You want to replace the values for the identifier, version and signing certificate with your data.

This will build an installer package which would install your binary on the target system. You should inspect the pkg file with Pacifist or Suspicious Package and do a test install on a test system to verify everything works.

If you want to learn more about installer packages and pkgbuild read my book “Packaging for Apple Administrators.”

Notarizing the Installer Package

Xcode has a command line tool altool which you can use to upload your tool for notarization:

xcrun altool --notarize-app \
             --primary-bundle-id "com.example.com" \
             --username "username@example.com" \
             --password "@keychain:Developer-altool" \
             --asc-provider "ABCD123456" \
             --file "build/hello-1.0.pkg"

The username is your developer account email.

The asc-provider is your ten digit Team ID. If you are only a member in a single team you do not need to provide this.

The password uses a special @keychain: keyword that tells altool to get the app-specific password out of a keychain item named Developer-altool. (Remember we created that earlier?)

This will take a while. When the command has successfully uploaded the pkg to Apple’s Notarization Servers, it will return a RequestUUID. Your notarization request will be queued and eventually processed. You can check the status of your request with:

xcrun altool --notarization-info "Your-Request-UUID" \
             --username "username@example.com" \                                    
             --password "@keychain:Developer-altool"

Apple will also send an email to your developer account when the process is complete. I my experience this rarely takes more than a minute or two. (Being in Central EU time zone might be an advantage there). When the process is complete, you can run the above notarization-info command to get some details. The info will include a link that contains even more information, which can be useful when your request is rejected.

Note that the info links expire after 24 hours or so. You should copy down any information you want to keep longer.

Completing the Process

You will not receive anything back from Apple other than the confirmation or rejection of your request. When a Mac downloads your installer package and verifies its notarization status it will reach out to Apple’s Notarization servers and they will confirm or reject the status.

If the Mac is offline at this time, or behind a proxy or firewall that blocks access to the Apple Servers, then it cannot verify whether your pkg file is notarized.

You can, however, ‘staple’ the notarization ticket to the pkg file, so the clients do not need to connect to the servers:

% xcrun stapler staple build/hello-1.0.pkg

You can also use stapler to verify the process went well:

% xcrun stapler validate build/hello-1.0.pkg

But since stapler depends on the developer tools to be installed, you should generally prefer spctl to check notarization:

% spctl --assess -vvv --type install build/hello-1.0.pkg

Automating the Process

Obviously, I built a script to automate all this. Put the following script in the root of the project folder, modify the variables at the start of the script (lines 20–38) with your information, and run it.

The script will build the tool, create a signed pkg, upload it for notarization, wait for the result, and then staple the pkg.

You can use this script as an external build tool target in Xcode. There are other ways to integrate scripts for automation in Xcode, but all of this is a new area for me and I am unsure which option is the best, and which I should recommend.

Links and Videos

These links and videos, especially Howard Oakley’s post and Tom Bridge’s PSU Presentation have been hugely helpful. Also thanks to co-worker Arnold for showing me this was even possible.

Going forward

Notarization is a key part of Apple’s security strategy going in macOS.

As MacAdmins we will usually deploy software through management systems, where the Gatekeeper mechanisms which evaluate notarization are bypassed. There are, however, already special cases (Kernel Extensions) where notarization is mandatory. It is likely that Apple will continue to tighten these requirements in the future. The macOS Mojave 10.14.5 update has shown that Apple may not even wait for major releases to increase the requirements.

If you are building your own tools and software for macOS and plan to distribute the software to other computers, you should start signing and notarizing.

On the other hand, I find the introduction of Notarization to macOS encouraging. If Apple wanted to turn macOS into a “App Store only system” like iOS, they would not have needed to build the notarization process and infrastructure. Instead, Apple seems to have embraced third-party-software from outside the App Store.

Notarization allows Apple to provide a security mechanism for software distributed through other means. It cannot be 100% effective, but when used correctly by Apple and the software developers it will provide a level of validation and trust for software downloaded from the internet.

Weekly News Summary for Admins — 2019-09-06

You can tell from the various articles and support posts that release day is definitely getting closer.

iOS 10.13.1 beta 2 was release and it is questionable if we will see another beta of either iOS, iPadOS or macOS before release. Apple has loosened notarization requirements for macOS Catalina temporarily (until Jan 2020) to help developer handle the transition.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Headlines

  • Rich Trouton’s and Charles Edge’s “Apple Device Management: A Unified Theory of Managing Macs, iPads, iPhones, and AppleTVs” book is available for pre-order on Amazon US, UK, DE (Affiliate Links)

On Scripting OS X

News and Opinion

macOS Catalina and iOS 13

MacAdmins on Twitter

  • Mike Boylan:
    “This is true. Age != skill (in both directions). Am fortunate that most people have treated me with respect.” (Thread)
  • Jason Fried: “When Google puts 4 paid ads ahead of the first organic result for your own brand name, you’re forced to pay up if you want to be found. It’s a shakedown. It’s ransom. But at least we can have fun with it. Search for Basecamp and you may see this attached ad.” (Image)
  • Benedict Evans: “Hell just froze over. beta.music.apple.com”
  • TimPerfitt: ‘Catalina “Restore from snapshot”’ (Image)

Bugs and Security

Support and HowTos

Apple Support

Updates and Releases

To Watch

To Listen

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Get Current User in Shell Scripts on macOS

…or, how to deal with deprecated bash and python…

There are many solutions to get the current logged in user in macOS to use in a shell script. However, the semi-official, “sanctioned” method has always involved a rather elaborate python one-liner, originally published by Ben Toms:

loggedInUser=$(/usr/bin/python -c 'from SystemConfiguration import SCDynamicStoreCopyConsoleUser; import sys; username = (SCDynamicStoreCopyConsoleUser(None, None, None) or [None])[0]; username = [username,""][username in [u"loginwindow", None, u""]]; sys.stdout.write(username + "\n");')

Update, because this a FAQ:

There are various other solutions to get the current user which use stat, who, or last commands. These will work in most situations. But there are edge cases, mostly with Fast User Switching, where these methods don’t return the correct user.

From python to scutil

While this has worked wonderfully and more reliably than other solutions, it always looked inelegant, and added a dependency on the python binary. This used to be fine, as the python binary shipped with macOS. But in the macOS Catalina release notes, we have learned that some, yet undefined, future version of macOS will not include the Python, Ruby, and Perl interpreters any more.

With prescient timing, Erik Berglund figured out a different method, which still accesses the same system framework, but through the scutil command rather than the PyObjC bridge.
Over time, different contributors in the MacAdmins Slack have optimized this command:

loggedInUser=$( scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

Sidenote: the scutil version runs 15–20x faster than the python version. Speed is not usually a major consideration for MacAdmin scripts, but it’s a nice improvement.

From bash to sh

This works great in bash and zsh. However, the built-in bash has been deprecated as well. (I have talked about this, a lot.)

While zsh is a great replacement for bash for interactive terminal use and scripting on the ‘full’ macOS system, there are environments (Recovery) where zsh is not present.

The only shell Mac Admins can rely on in all contexts is now /bin/sh.

Because of this I recommend using /bin/sh for installer scripts (pre- and postinstall scripts in pkgs).

The Posix sh standard does not include the ‘here string’ <<< used in Erik’s command. Nevertheless, when you use the above construct in sh on macOS it will still work fine. This is because sh on macOS is actually handled by bash in sh compatibility mode.

And while bash in sh compatibility mode will recreate the odd syntax and quirks of sh for compatibility, it will also happily run ‘bashims’ that don’t even exist in sh. E.g. double brackets [[...]] or here strings <<<.

As long as bash is taking care of sh scripts, you will be fine. Myself, I would not have noticed this if I had not ran a sh script through shellcheck. Shellcheck told me that Posix sh should not understand the here string:

loggedInUser=$( scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/ && ! /loginwindow/ { print $3 }' )
                       ^-^ SC2039: In POSIX sh, here-strings are undefined.

But we’re fine on macOS, because sh is really bash right? No need to worry…

Predicting the demise of bash

We don’t yet know when, but the /bin/bash binary will eventually be removed from macOS. The version included with macOS has not been updated since 2014. It is unlikely that the twelve year old bash v3.2 will receive patches for security vulnerabilities in the future.

With the switch to zsh as the default shell in Catalina Apple is laying the groundwork to remove bash v3.2 from the system in a future release. cron and login hooks were infamously deprecated in macOS years ago and are still around, so we could be looking at years. But my suspicion is that bash is just one security vulnerability away from being removed.

With the High Sierra and Mojave updates, Apple has shown that they do not need to wait for major releases to remove or add functionality or restrictions to the system.

I believe /bin/bash has more than a few months still. However, Apple’s messaging on this switch is uncharacteristically strong. It would not be terribly surprising if the Catalina Spring Update in March 2020 removed bash.

The amount of old installer packages which would break on such a change is terrifying. Apple might be willing to pay this price, to avoid a publicized security vulnerability. A severe bash vulnerability would definitely get a lot of publicity. Remember Shellshock? That was the last time bash v3.2 was patched.

Instead of removing /bin/bash Apple might be able to replace it with zsh in bash emulation mode. So far, I have found no signs for this. While it wouldn’t be perfect, it would be a safer move.

Python 2.7 will still get updates until the end of 2019. There are also some commands and tools in the system written in python 2.7. It will be harder for Apple to migrate these to other languages, so I think Python has a longer countdown clock. But it cannot hurt to start preparing now.

Whither sh?

Whenever /bin/bash is removed, what will happen with /bin/sh at that point? The answer can be deduced from the support article on the shell switch.

How to test your shell scripts

To test script compatibility with Bourne-compatible shells in macOS Catalina, you can change /var/select/sh to /bin/bash, /bin/dash, or /bin/zsh. If you change /var/select/sh to a shell other than bash, be aware that scripts that make use of bashisms may not work properly.

zsh can be made to emulate sh by executing the command zsh --emulate sh.

We can actually see that Apple has two options here. sh processes could be run by zsh in emulation mode or by dash.

dash is the Debian Almquist shell, which is a minimal implementation of the Posix sh standard. dash is a newcomer to macOS on Catalina. New arrivals on a release where Apple seems intent on cleaning out ballast (32-bit, Kernel Extensions, bash v3, Python, Perl, and Ruby) are immediately of interest.

When you check on Catalina Recovery system, you can see that /bin/dash has been added, but /bin/zsh is absent. (/bin/bash is still present on Catalina Recovery and the sh binary is also still bash.)

Putting all of this together, I would hazard that Apple is planning to use dash to take over from ‘bash as sh.’ This will also bring macOS in line with most other Unix and Linux distributions where dash commonly handles sh processes.

With the symbolic link mechanism described in the support article, Apple could switch sh to dash in an update while bash is still present. This could allow Developers or Mac Admins to switch their managed systems back to bash when they need to mitigate problems.

When Apple switches to dash, sh scripts that use bashisms will break.

You can start testing this in Catalina by changing the symbolic link at /var/select/sh:

% sudo ln -sf /bin/dash /var/select/sh

(Change /bin/dash to /bin/bash to revert to default.)

When I test the above command to get the current user with dash, it returns:

Syntax error: redirection unexpected

(You can also change the shebang of a script you want to test to #!/bin/dash then you do not have to change the system sh.)

Getting the Current User in sh, Future Proof (I hope)

The solution is easy enough. Replace the here string with a pipe. For sh scripts, change the syntax to:

loggedInUser=$( echo "show State:/Users/ConsoleUser" | scutil | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

This will make Posix sh, dash, and shellcheck happy and will still work in bash and zsh.

This syntax (echoing into a pipe) is considered inelegant and inefficient, because it requires an additional process (echo) and an additional pipe. Here docs and here strings were introduced exactly to avoid this. Obviously, if you prefer, you can continue to use the here string form of the command for bash and zsh.

Conclusion

There is no need panic and search and replace the python command in all your scripts immediately. I would recommend replacing this line when you update a script. Keep all of this in mind when working on your administration, installation and workflow scripts going forward.

You may also find scripts in third party documentation and installers that need updating, and you should let the author or vendor know.

  • /bin/bash (v3.2) is deprecated and has not received updates since 2014
  • /usr/bin/python (v2.7) is deprecated and will not receive new updates starting in Jan 2020
  • both will eventually be removed from macOS
  • we don’t know when exactly, could be years, could be months
  • even when you do not use Python for scripting, you may be using Python ‘one-liners’ in shell scripts
  • bash scripts should be moved to zsh (general use) or sh (general and installation scripts)
  • sh processes in macOS will likely be handled by dash in the future
  • use shellcheck to find and correct bashisms in sh scripts
  • shellcheck does not work for zsh scripts

Weekly News Summary for Admins — 2019-08 -30

Things are heating up for September release season. We got new betas for iOS and iPadOS 13.0 and macOS Catalina beta7. Surprisingly, there also is a beta for iOS 13.1, which includes some features shown at WWDC, which were removed from the 13.0 beta. While I generally approve of this flexibility, it makes testing confusing and more complex.

We also got a second supplemental update for 10.14.6, fixing some more bugs and security holes. The newest 10.14.6 has the build number 18G95. The silver lining in all this version chaos is, that we will likely not end Mojave on a forked hardware build.

Apple sent out the invitations to the iPhone event on September 10, 10am PDT (19:00 CEST). If previous years can be used as a guide, this could mean the iOS 13.0 release will be on September 13 or 20 and the macOS 10.15.0 Catalina release on September 20 or 27.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

The “Moving to zsh” class on September 6 is now full. Looking forward to seeing you all there! If you missed to sign up for this class, the next “Scripting macOS” beginners’ class on October 30–31 will be updated for Catalina and contain plenty of zsh, as well.

News and Opinion

macOS 10.13 Catalina and iOS 13

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Listen

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Check Installer Pkgs for deprecated scripts

macOS 10.15 Catalina will deprecate the built-in /bin/bash. I have talked about this at length.

The release notes for Catalina also tell us that other built-in scripting runtimes, namely Python, Perl, and Ruby. Will not be included in future macOS releases (post-Catalina) any more.

This means, that if you want to use bash, Python, Perl, or Ruby on macOS, you will have to install, and maintain your own version in the future.

However, scripts in installation packages, cannot rely on any of these interpreters being available in future, post-Catalina versions of macOS. Installer pkgs can be run in all kinds of environments and at all times, and you would not want them to fail, because a dependency is missing.

The good news is that we still have time. All the runtimes mentioned above are still present in Catalina, so the packages will continue to work for now. But if you are building installation scripts, you need to check if any of the installation scripts use one of these interpreters and fix them.

I recommend to use /bin/sh for installation scripts, since that will run in any macOS context, even the Recovery system.

If you are using third-party installer packages, you may also want to check them for these interpreters, and notify the developer that these packages will break in future versions of macOS.

To check a flat installer package, you would expand it with pkgutil --expand and then look at script files in the Scripts folder. This will work fine for a package or two, but gets tedious really quickly, especially with large distribution pkgs with many components (e.g. Office).

So… I wrote a script to do it. The script should handle normal component pkgs, distribution pkgs and the legacy bundle pkgs and mpkgs.

You can get the pkgcheck script from my Github repo.

What the script does

Once I had written the code to inspect all these types of pkgs, I realized I could grab all other kinds of information, as well. The pkgcheck.sh script will check for:

  • Signature and Notarization
  • Type of Package: Component, Distribution, legacy bundle or mpkg
  • Identifier and version (when present)
  • Install-location
  • for Distribution and mpkg types, shows the information for all components as well
  • for every script in a pkg or component, checks the first line of the script for shebangs of the deprecated interpreters (/bin/bash, /usr/bin/python, /usr/bin/perl, and /usr/bin/ruby) and print a warning when found

How to run pkgcheck.sh

Run the script with the target pkg file as an argument:

% ./pkgcheck.sh sample.pkg

You can give more than one file:

% ./pkgcheck.sh file1.pkg file2.pkg ...

When you pass a directory, pkgcheck.sh will recursively search for all files or bundle directories with the pkg or mpkg extension in that directory:

% ./pkgcheck.sh SamplePkgs

Features and Errors

There are a few more things that I think might be useful to check in this script. Most of all, I want to add an indicator whether a component is enabled by default or not. If you can think of any other valuable data to display, let me know. (Issue or Pull Request or just ping me on MacAdmins Slack)

I have tested the script against many pkgs that I came across. However, there are likely edge cases that I haven’t anticipated, which might break the script. If you run into any of those, let me know. (File an Issue or Pull Request.) Having the troublesome pkg would of course be a great help.

Note: the script will create a scratch directory for temporary file extractions. The script doesn’t actually expand the entire pkg file, only the Scripts sub-archive. The scratch folder will be cleaned out at the beginning of the next run, but not when the script ends, as you might want to do some further inspections.

Sample outputs

This is a sample pkg I build in my book, it has pre- and postinstall scripts using a /bin/bash shebang:

% ./pkgcheck.sh SourceCodePro-2.030d.pkg
SourceCodePro-2.030d
SamplePkgs/SourceCodePro-2.030d.pkg
Signature:      None
Notarized:      No
Type:           Flat Component PKG
Identifier:     com.example.SourceCodePro
Version:        2.030d
Location:       /
Contains 2 resource files
postinstall has shebang #!/bin/bash
preinstall has shebang #!/bin/bash

This is the experimental notarized pkg installer for desktoppr:

% ./pkgcheck.sh desktoppr-0.2.pkg
desktoppr-0.2
SamplePkgs/desktoppr-0.2.pkg
Signature:      Developer ID Installer: Armin Briegel (JME5BW3F3R)
Notarized:      Yes
Type:           Flat Component PKG
Identifier:     com.scriptingosx.desktoppr
Version:        0.2
Contains 0 resource files

And finally, this is a big one, the Microsoft Office installer: (they have some work to do to clean up those scripts)

% ./pkgcheck.sh Microsoft\ Office\ 16.27.19071500_Installer.pkg
Microsoft Office 16.27.19071500_Installer
SamplePkgs/Microsoft Office 16.27.19071500_Installer.pkg
Signature:      Developer ID Installer: Microsoft Corporation (UBF8T346G9)
Notarized:      No
Type:           Flat Distribution PKG
Contains 11 component pkgs

    Microsoft_Word_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Word.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 3 resource files

    Microsoft_Excel_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Excel.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_PowerPoint_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_PowerPoint.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_OneNote_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_OneNote.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    Microsoft_Outlook_Internal
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_Outlook.app
    Version:        16.27.19071500
    Location:       /Applications
    Contains 2 resource files

    OneDrive
    Type:           Flat Component PKG
    Identifier:     com.microsoft.OneDrive
    Version:        19.70.410
    Location:       /Applications
    Contains 30 resource files
    postinstall has shebang #!/bin/bash
    od_logging has shebang #!/bin/bash
    od_service has shebang #!/bin/bash
    od_migration has shebang #!/bin/bash
    preinstall has shebang #!/bin/bash

    Office16_all_autoupdate
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Microsoft_AutoUpdate.app
    Version:        4.13.19071500
    Location:       /Library/Application Support/Microsoft/MAU2.0
    Contains 2 resource files
    postinstall has shebang #!/bin/bash
    preinstall has shebang #!/bin/bash

    Office16_all_licensing
    Type:           Flat Component PKG
    Identifier:     com.microsoft.pkg.licensing
    Version:        16.27.19071500
    Location:       /
    Contains 2 resource files
    dockutil has shebang #!/usr/bin/python

    Office_fonts
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.DFonts
    Version:        0
    Location:       /private/tmp/com.microsoft.package.DFonts
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

    Office_frameworks
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Frameworks
    Version:        0
    Location:       /private/tmp/com.microsoft.package.Frameworks
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

    Office_proofing
    Type:           Flat Component PKG
    Identifier:     com.microsoft.package.Proofing_Tools
    Version:        0
    Location:       /private/tmp/com.microsoft.package.Proofing_Tools
    Contains 1 resource files
    postinstall has shebang #!/bin/bash

Weekly News Summary for Admins — 2019-08-23

Summer vacation time is coming to an end and the beta season for macOS Catalina and iOS 13 is heating up! Another Catalina beta dropped this week, and third parties are now starting to announce their own betas and/or preliminary support.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

The “Moving to zsh” class on September 6 is now full. Looking forward to seeing you all there! If you missed to sign up for this class, the next “Scripting macOS” beginners’ class on October 30–31 will be updated for Catalina and contain plenty of zsh, as well.

News and Opinion

macOS 10.15 Catalina

MacAdmins on Twitter

  • William Smith: “Excited for this release from Microsoft because it’ll be the first app they’ve ever released for Mac that’s fully manageable from day 1. There’s been a lot of outreach the past few years from them to MacAdmins to make something like this possible.”
  • Mike Stern: “SF Mono is now available! Also, SF Pro and SF Compact have been updated to reflect some additions and changes to SF Symbols. developer.apple.com/fonts”

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Listen

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Shellcheck and zsh

The sad news is that the shellcheck binary does not really know how to deal with zsh scripts:

% shellcheck script.zsh

In script.zsh line 1:
#!/bin/zsh
^-- SC1071: ShellCheck only supports sh/bash/dash/ksh scripts. Sorry!

You can force shellcheck to interpret a zsh script with the --shell option:

% shellcheck --shell=bash dirlist.sh

This might show errors, such as Quote to prevent word splitting/globbing which aren’t really errors in zsh so its usefulness will be limited.

If you want a specific script to be always interpreted as a bash script by shellcheck you can also add this line:

# shellcheck shell=bash

in the script, right after the shebang line.

This can be useful if you use zsh in bash emulation mode with

emulate -LR bash

Moving to zsh, part 8 – Scripting zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

This is the final article in this series. (If I ever announce an eight part series again, please somebody intervene!) However, I am quite sure it will not be the last post on zsh

All the previous posts described how zsh works as an interactive shell. The interactive shell is of course the most direct way we use a shell and configuring the shell to your taste can bring a huge boost in usefulness and productivity.

The other, equally, important aspect of a shell is running script files. In the simplest perspective, script files are just series of interactive commands, but of course they will get complex very quickly.

sh, bash, or zsh?

Should you even script in zsh? The argument for bash has been that it has been pre-installed on every Mac OS X since 10.2. The same is true for zsh, with one exception: the zsh binary is not present on the Recovery system. It is also not present on a NetInstall or External Installation System, but these are less relevant in a modern deployment workflow, which has to work with Secure Boot Macs.

If you plan to run a script from Recovery, such as an installr or bootstrappr script or as part of an MDS workflow, your only choices are /bin/sh and /bin/bash. The current /bin/bash binary is 12 years old, and Apple is messaging its demise. I would not consider that a future proof choice. So, if your script may run in a Recovery context, i would recommend /bin/sh over either /bin/bash or /bin/zsh

Since installation packages can be run from the Recovery context as well, and you cannot really always predict in which context your package will be used, I would extend the recommendation to use /bin/sh for all installation scripts as well.

While sh is surely ubiquitous, it is also a ‘lowest common denominator’, so it is not a very comfortable scripting language to work in. I recommend using shellcheck to verify all your sh scripts for bashisms that might have crept in out of habit.

When you can ensure your script will only run on a full macOS installation, zsh is good choice over sh. It is pre-installed on macOS, and it offers better and safer language options than sh and some advantages over bash, too. Deployment scripts, scripts pushed from management systems, launch daemons and launch agents, and script you write to automate your admin workflows (such as building packages) would fall in this category.

You can also choose to stick with bash, but then you should start installing and using your own bash 5 binary instead of the built in /bin/bash. This will give you newer security updates and features and good feeling that when Apple does eventually yank the /bin/bash binary, your scripts will keep working.

Admins who want to keep using Python for their scripts are facing a similar problem. Once you choose to use a non-system version of bash (or python), it is your responsibility to install and update it on all your clients. But that is what system management tools are for. We will have to get used to managing our own tools as well as the users’ tools, instead of relying on Apple.

Shebang

To switch your script from using bash to zsh, you have to change the shebang in the first line from #!/bin/bash to #!/bin/zsh.

If you want to distinguish your zsh script files, the you can also change the script’s file extension from .sh to .zsh. This will be especially helpful while you transfer scripts from bash to zsh (or sh). The file extension will have no effect on which interpreter will be used to run the script. That is determined by the shebang, but the extension provides a visible clue in the Finder and Terminal.

zsh vs bash

Since zsh derives from the same Bourne shell family as bash does, most commands, syntax, and control structures will work just the same. zsh provides alternative syntax for some of the structures.

zsh has several options to control compatibility, not only for bash, but for other shells as well. We have already seen that options can be used to enable features specific for zsh. These options can significantly change how zsh interprets your scripts.

Because you can never quite anticipate in which environment your particular zsh will be launched in, it is good practice to reset the options at the beginning of your script with the emulate command:

emulate -LR zsh

After the emulate command, you can explicitly set the shell options your script requires.

The emulate command also provides a bash emulation:

emulate -LR bash

This will change the zsh options to closely emulate bash behavior. Rather than relying on this emulation mode, I would recommend actually using bash, even if you have to install and manage a newer version yourself.

Word Splitting in Variable Substitutions

Nearly all syntax from bash scripts will ‘just work’ in zsh as well. There are just a few important differences you have to be aware of.

The most significant difference, which will affect most scripts is how zsh treats word splitting in variable substitutions.

Recap: bash behavior

In bash substituted variables are split on whitespace when the substitution is not quoted. To demonstrate, we will use a function that counts the number of arguments passed into it. This way we can see whether a variable was split or not:

#!/bin/bash
export PATH=/usr/bin:/bin:/usr/sbin:/sbin

function countArguments() {
    echo "${#@}"
}

wordlist="one two three four five"

echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 5

echo "substitution with quotes"
countArguments "$wordlist"
# -> 1

In bash and sh the contents of the variable split into separate arguments when substituted without the quotes. Usually you do not want the splitting to occur. Hence the rule: “always quote variable substitutions!”

zsh behavior: no splitting

zsh will not split a variable when substituted. With zsh the contents of a variable will be kept in one piece:

#!/bin/zsh
emulate -LR zsh # reset zsh options
export PATH=/usr/bin:/bin:/usr/sbin:/sbin

function countArguments() {
    echo "${#@}"
}

wordlist="one two three four five"

echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 1

echo "substitution with quotes"
countArguments "$wordlist"
# -> 1

The positive effect of this is that you do not have to worry about quoting variables all the time, making zsh less error prone, and much more like other scripting and programming languages.

Splitting Arrays

The wordlist variable in our example above is a string. Because of this it returns a count of 1, since there is only one element, the string itself.

If you want to loop through multiple elements of a list

In bash this happens, whether you want to or not, unless you explicitly tell bash not to split by quoting the variable.

In zsh, you have to explicitly tell the shell to split a string into its components. If you do this naïvely, by wrapping the string variable in the parenthesis to declare and array, it will not work:

wordlist="one two three"
wordarray=( $wordlist )

for word in $wordarray; do
    echo "->$word<-"
done

#output
->one two three<-

Note: the for loop echoes every item in the array. I have added the -> characters to make the individual items more visible. In the subsequent examples, I will not repeat the for loop, but only show its output. So the above example will be shortened to:

wordarray=( $wordlist )
->one two three<-

There are few options to do this right.

Revert to sh behavior

First, you can tell zsh to revert to the bash or sh behavior and split on any whitespace. You can do this by pre-fixing the variable substitution with an =:

wordarray=( ${=wordlist} )
->one<-
->two<-
->three<-

Note: if you find yourself using the = frequently, you can also re-enable sh style word splitting with the shwordsplit option. This will of course affect all substitutions in the script until you disable the option again.

setopt shwordsplit
wordarray=( $wordlist )
->one<-
->two<-
->three<-

This option can be very useful when you quickly need to convert a bash script to a zsh script. But you will also re-enable all the problems you had with unintentional word splitting.

Splitting Lines

If you want to be more specific and split on particular characters, zsh has a special substitution syntax for that:

macOSversion=$(sw_vers -productBuild) # 10.14.6
versionParts=${(s/./)macOSVersion}
->10<-
->14<-
->6<-

If you want to split on a newline character \n the syntax is slightly different:

citytext="New York
Rio
Tokyo"

cityarray=( ${(ps/\n/)citytext} )
->New York<-
->Rio<-
->Tokyo<-

Since newline is a common character to split text on, there is a short cut:

cityarray=( ${(f)citytext} )

Since the newline character is a legal character in file names, you should use zero-terminated strings where possible:

foundDirs=$(find /Library -type d -maxdepth 1 -print0)
dirlist=${(ps/\0/)foundDirs}

Again, there is a shortcut for this:

dirlist=${(0)foundDirs}

Array index starts at 1

Once you have split text into an array, remember, that in zsh array indices start at 1:

% versionList=( ${(s/./)$(sw_vers -productVersion)} )
% echo ${versionList[1]}
10
% echo ${versionList[2]}
14
% echo ${versionList[3]}
6

If you think this is wrong and absolutely require a zero-based index, you can set the KSH_ARRAYS shell option:

% setopt KSH_ARRAYS
% echo ${versionList[0]}
10
% echo ${versionList[1]}
14
% echo ${versionList[2]}
6
% echo ${versionList[3]}

Conclusion

Switching your scripts from bash to zsh requires a bit more work than merely switching out the shebang. However, since /bin/bash will still be present in Catalina, you do not have to move all scripts immediately.

Moving to sh instead of zsh can be safer choice, especially for package installation scripts.

In zsh, there always seems to be some option to disable or enable a particular behavior.

This concludes my series on switching to zsh on macOS. I hope you found it helpful.

After having worked with zsh for a few weeks, I already find some of its features indispensable. I am looking forward to discovering and using more features over time. When I do, I will certainly share them here.

Weekly News Summary for Admins — 2019-08-16

Longer summary this week to catch up on the last three weeks of traveling!

The last post celebrating the 100th email resulted in quite a few new subscribers. Welcome!

I posted my speaking and training class schedule for the next few months. It’d be great to meet you somewhere!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • Denise Yu: “I illustrated my three favourite binaries”
  • Pepijn Bruienne: “To my surprise we recently updated the Authorization Plugin documentation. Surprise because it’s not seen a lot of change in recent times, but you get TWO new callbacks, RemoveHintValue and RemoveContextValue!”
  • Mr. Macintosh: “2019–004 for High Sierra 10.13, Sierra 10.12 and BridgeOS have been re released! I will update the main article and reply to this post when I have the new build numbers.”
  • Rosyna Keller: “The updated Hardened Runtime docs are out! The overview includes more information on how to enable it in Xcode and explains that the hardened runtime is designed to stop certain classes of exploits.” (Thread)
  • Phil Stokes: “Thanks to @SentinelOne @patrickwardle, my free ebook of How To Reverse macOS Malware is now available. Learn to set up a safe Mac test lab, find malware samples, static + dynamic analysis. eBook is free; tools you need are free.”
  • Ronald Oussoren: “The version 6 branch of PyObjC has finally caught up with the current beta: tests pass in macOS 10.15 beta 5 with Python 3.8b3, including bindings for all new frameworks.”
  • Edward Marczak: “Support local meetups! Particularly in tech: we need more discussion between disparate teams. There’s undoubtedly a meetup near you that matches your style. Support can mean presenting, but also just showing up. Without you, these gatherings go away. With you, they get stronger.”

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Watch

To Listen

Just for Fun

  • Timo Elliott: “How to evaluate your existing enterprise infrastructure…” (click for cartoon)
  • Ed Marczak: “We have a long way to go in exploring this medium. ” (read linked thread)

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Autumn 2019 Schedule

As I am emerging from vacation, I made an overview of my agenda for the next few months and realized there are quite a few public events. It might be useful to share my public agenda. If you are coming to any of these events, I’d be really happy to meet you!

Sep 6: Moving to zsh Class

Half-day class covering zsh on macOS. We will cover:

  • why Apple is switching the default shell
  • how this affects you and your users
  • how to configure zsh to increase your Terminal productivity
  • how to transfer your bash configuration
  • scripting zsh

The class will be held in the Pro Warehouse training room in Amsterdam on Sep 6, 10:00–14:00. You can sign up at the Pro warehouse page.

You can combine this training with the next event.

Sep 6: Dutch MacAdmins Meeting/Pro Academy Opening

We are celebrating the opening of the Pro Academy by hosting the Dutch MacAdmins meeting. Participation is free. Among many other presentations I will show how (and why) to build a command line tool in Swift.

Presentations will be (mostly) in English, so the event is suitable for international guests. Participation is free and you can sign up on Eventbrite.

Oct 1–4: MacSysAdmin Gothenburg

I am really excited about presenting at MacSysAdmin again. My topic is “Moving to zsh,” where I will discuss the changes (not just the changed shell) in Catalina and what they mean for MacAdmins.

There are still a few tickets available, but this conference usually fills up. So don’t delay and go register if you haven’t done so yet!

Oct 30–31: Scripting macOS Class

This is our ‘entry to scripting’ class, where we teach the basics of shell scripting. This will be the second time we are teaching this class and it will be updated for Catalina. Most of the examples used to teach scripting come from real administrator workflows.

This is a two day class held in the Pro Warehouse training room in Amsterdam on October 30 and 31. You can sign up at the Pro warehouse page.

Nov 12–14: JamfNation User Conference

I am not going to JNUC myself, but my co-worker and director Mischa van der Bent is. He will be presenting on “Offboarding in a Modern Deployment Workflow”, a topic which will include, among many other things, our Erase&Install application.

Book Updates

I am working on updates with Catalina content for all three books. They should be published around the Catalina release date, maybe a little later.

As always, if you have already purchased one of my book, the update will be free in the Apple Books application on your iOS device or Mac. There is no need to wait for the update if you are interested now. Purchase now and get the update when it is ready!

I am also working on new books. I had been working on some interesting content, but the experiences from the scripting classes and the Catalina announcements have made me take a step back and re-evaluate the plans and progress so far. Re-writing will be necessary before I am happy enough to publish.

From the topics in all the events above, you should be able to deduce at least one of the topics, though.