The unexpected return of JavaScript for Automation

Monterey has deprecated the pre-installed python on macOS. To be precise, built-in python has been deprecated since macOS Catalina, but Monterey will now throw up dialogs warning the user that an app or process using built-in python needs to be updated.

I and others have written about this before:

So far, I have recommended to build native Swift command line tools to replace python calls. However, from discussions in MacAdmins Slack, a new option has emerged. Most of the credit for popularizing and explaining this goes to @Pico (@RandomApps on Twitter) in the #bash and #scripting channels.

(Re-)Introducing JavaScript for Automation

AppleScript has been part of macOS since System 7.1. In the late nineties, there was concern that it wouldn’t make the transition to Mac OS X, but AppleScript made the jump and has happily co-existed with the Terminal and shell scripting as an automation tool on macOS. AppleScript has a very distinct set of strengths (interapplication communication) and weaknesses (awkward syntax and inconsitent application functionality and dictionaries) but it has been serving its purpose well for many users.

With Mac OS X 10.4 Tiger, Apple introduced Automator, which provided a neat UI to put together workflows. Much of Automator was based on AppleScript and users expected a more and improved AppleScript support because of that going forward. Instead, we saw AppleScript’s support from Apple and third parties slowly wane over the years.

AppleScript is stil very much present and functional in recent versions of macOS. It just seems like it hasn’t gotten much love over the last decade or so. Now that Shortcuts has made the jump from iOS, there may be hope for another revival?

The last major changes to AppleScript came with Mavericks and Yosemite. Mavericks (10.9) included a JavaScript syntax for the Open Scripting Architecture (OSA), which is the underlying framework for all AppleScript functionality. Apple called this “JavaScript for Automation.” Because this is a mouthful, it often abbreviated as JXA.

The JavaScript syntax and structure is more like a “real” programming language, than the “english language like” AppleScript. Once again this raised hopes that this could attract more scripters to AppleScript and thus encourage Apple and third party developers to support more AppleScript. But unfortunately, this positive re-inforcement did not take off.

Then Yosemite (10.10) made the AppleScript-Objective-C bridge available everywhere in AppleScript. Previously, the Objective-C bridge was only available when you built AppleScript GUI applications using AppleScript Studio in Xcode. The Objective-C bridge allows scripters to access most of the functionality of the system frameworks using AppleScript or JXA.

The coincidence of these two new features might be the reason that the ObjC bridge works much better using JXA than it does with the native AppleScript syntax.

JXA and Python

What does JXA and the AppleScriptObjC bridge have to do with the Python deprecation in modern macOS?

One reason python became so popular with MacAdmins, was that the pre-installed python on Mac OS X, also came with PyObjC, the Objective-C bridge for python. This allowed python to build applications with a native Cocoa UI, such as AutoDMG and Munki’s Managed Software Center. It also allowed for short python scripts or even one-liners to access system functionality that was otherwise unavailable to shell scripts.

For example, to determine if a preference setting in macOS is enforced with a configuration profile, you can use CFPreferences or NSUserDefaults.

Objective-C/C:

BOOL isManaged =CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

Swift:

let isManaged = CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

The Objective-C bridge allows to use this call from python, as well:

from Foundation import CFPreferencesAppValueIsForced
isManaged=CFPreferencesAppValueIsForced("idleTime", "com.apple.screensaver")

With JXA and the AppleScriptObjC bridge, this will look like this:

ObjC.import('Foundation');
$.CFPreferencesAppValueIsForced(ObjC.wrap('idleTime'), ObjC.wrap('com.apple.screensaver'))

Now, this looks really simple, but working with any Objective-C bridge is always fraught with strange behaviors, inconsistencies and errors and the JXA ObjC implementation is no different.

For example, I wanted to change the code above to return the value of the setting instead of whether it is managed. The CFPreferences function for that is called CFPreferencesCopyAppValue and it works fine in Swift and Python, but using JXA it only ever returned [object Ref]. The easiest solution was to switch from the CFPreferences functions to using the NSUserDefaults object:

ObjC.import('Foundation');
ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('$1').objectForKey('$2'))

(Once again many thanks to @Pico on the MacAdmins Slack for helping me and everyone else with this and also pointing out, that there is a different, somewhat complicated, solution to the object Ref problem. I will keep that one bookmarked for situations where there is no alternative Cocoa object.)

We used this to remove the python dependency from Mischa van der Bent’s CIS-Scripts.

JXA in shell scripts

To call JXA from a shell script, you use the same osascript command as for normal AppleScript, but add the -l option option to switch the language to JavaScript:

osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation');
    ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('idleTime').objectForKey('com.apple.screensaver'))
EndOfScript

For convenience, you can wrap calls like this in a shell function:

function getPrefValue() { # $1: domain, $2: key
      osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation');
    ObjC.unwrap($.NSUserDefaults.alloc.initWithSuiteName('$1').objectForKey('$2'))
EndOfScript
}

function getPrefIsManaged() { # $1: domain, $2: key
     osascript -l JavaScript << EndOfScript
     ObjC.import('Foundation')
     $.CFPreferencesAppValueIsForced(ObjC.wrap('$1'), ObjC.wrap('$2'))
EndOfScript
}

echo $(getPrefValue "com.apple.screensaver" "idleTime")
# -> actual value
echo $(getPrefIsManaged "com.apple.screensaver" "idleTime")
# -> true/false

Note that the $ character does a lot of work here. It does the shell variable substitution for the function arguments in the case of $1 and $2. These are substituted before the here doc is piped into the osascript command. The $. at the beginning of the command is a shortcut where $ stands in for the current application and serves as a root for all ObjC objects.

There is also a $(…) function in JXA which is short for ObjC.unwrap(…) but I would recommend against using that in combination with shell scripts as shell’s command substitution has the same syntax and would happen before the JavaScript is piped into osascript.

There is a GitHub wiki with more detailed documentation on using JXA, and the JXA Objective-C bridge in particular.

JXA for management tasks

I’ll be honest here and admit that working with JXA seems strange, inconsistent, and — in weird way — like a step backwards. Putting together a Command Line Tool written in Swift feels like a much more solid (for lack of a better word) way of solving a problem.

However, the Swift binary command line tool has one huge downside: you have to install the binary on the client before you can use it in scripts and your management system. Now, as MacAdmins, we usually have all the tools and workflows available to install and manage software on the client. That’s what we do.

On the other hand, I have encountered three situations (set default browser, get free disk space, determine if a preference is managed) where I needed to replace some python code in the last few months and I would have no trouble finding a few more if I thought about it. Building, maintaining, and deploying a Swift CLI tool for each of these small tasks would add up to a lot of extra effort, both for me as the developer and any MacAdmin who wants to use the tools.

Alternatively, you can deploy and use a Python 3 runtime with PyObjC, like the MacAdmins Python and continue to use python scripts. That is a valid solution, especially when you use other tools built in python, like Outset or docklib. But it still adds a dependency that you have to install and maintain.

In addition to being extra work, it adds some burden to sharing your solutions with other MacAdmins. You can’t just simply say “here’s a script I use,” but you have to add “it depends on this runtime or tool, which you also have to install.

Dependencies add friction.

This is where JXA has an advantage. Since AppleScript and its Objective-C bridge are present on every Mac (and have been since 2014 when 10.10 was released) there is no extra tool to install and manage. You can “just share” scripts you build this way, and they will work on any Mac.

For example, I recently built a Swift command line tool to determine the free disk space. You can download the pkg, upload it to your management system, deploy it on your clients and then use a script or extension attribute or fact or something like to report this value to your management system. Since there is a possibility that the command line tool is not yet installed when the script runs, you need to add some code to check for that. All-in-all, nothing here is terribly difficult or even a lot of work, but it adds up.

Instead you can use this script (sample code for a Jamf extension attribute):

#!/bin/sh

freespace=$(/usr/bin/osascript -l JavaScript << EndOfScript
    ObjC.import('Foundation')
    var freeSpaceBytesRef=Ref()
    $.NSURL.fileURLWithPath('/').getResourceValueForKeyError(freeSpaceBytesRef, 'NSURLVolumeAvailableCapacityForImportantUsageKey', null)
    ObjC.unwrap(freeSpaceBytesRef[0])
EndOfScript
)

echo "<result>${freespace}</result>"

Just take this and copy/paste it in the field for a Jamf Extension Attribute script and you will get the same same free disk space value as the Finder does. If you are running a different management solution, it shouldn’t be too difficult to adapt this script to work there.

The Swift tool is nice. Once it is deployed, there are some use cases where it could be useful to have a CLI tool available. But most of the time, the JXA code snippet will “do the job” with much less effort.

Note on Swift scripts

Some people will interject with “but you can write scripts with a swift shebang!” And they are correct. However, scripts with a swift shebang will not run on any Mac. They will only run with Xcode, or at least the Developer Command Line Tools, installed. And yes, I understand this is hard for developers to wrap their brains around, but most people don’t have or need Xcode installed.

When neither of these are installed yet, and your management system attempts to run a script with a swift shebang, it will prompt the user to install the Developer command line tools. This is obviously not a good user experience for a managed deployment.

As dependencies go, Xcode is a fairly gigantic installation. The Developer Command Line Tools much less so, but we are back in the realm of “install and manage a dependency.”

Parsing JSON

Another area where JXA is (not surprisingly) extremely useful is JSON parsing. There are no built-in tools in macOS for this so MacAdmins either have to install jq or scout or fall back to parsing the text with sed or awk. Since JSON is native JavaScript, JXA “just works” with it.

For example the new networkQuality command line tool in Monterey has a -c option which returns JSON data instead of printing a table to the screen. In a shell script, we can capture the JSON in a variable and substitute it into a JXA script:

#!/bin/sh

json=$(networkQuality -c)

osascript -l JavaScript << EndOfScript
    var result=$json
    console.log("Download:  " + result.dl_throughput)
    console.log("Upload:    " + result.ul_throughput)
EndOfScript

Update: (2021-11-24) Paul Galow points out that this syntax might allow someone to inject code into my JavaScript. This would be especially problematic with MacAdmin scripts as those often run with root privileges. The way to avoid this injection is too parse the JSON data with JSON.parse :

#!/bin/sh 

json=$(networkQuality -c) 

osascript -l JavaScript << EndOfScript     
  var result=JSON.parse(\`$json\`)     
  console.log("Download:  " + result.dl_throughput)     
  console.log("Upload:    " + result.ul_throughput) 
EndOfScript

(I am leaving the original code up there for comparison.)

Conclusion

After being overlooked for years, JXA now became noticeable again as a useful tool to replace python in MacAdmin scripts, without adding new dependencies. The syntax and implementation is inconsistent, buggy, and frustrating, but the same can be said about the PyObjC bridge, we are just used it. The community knowledge around the PyObjC bridge and solutions goes deeper.

However, as flawed as it is, JXA can be a simple replacement for the classic python “one-liners” to get data out of a macOS system framework. Other interesting use cases are being discovered, such as JSON parsing. As such, JavaScript for Automation or JXA should be part of a MacAdmins tool chest.

Monterey, python, and free disk space

With Montery, many MacAdmins have been seeing dialogs that state:

“ProcessName” needs to be updated

and often the “ProcessName” is your management system. As others have already pointed out, the process, or scripts this process is calling, is using the pre-installed Python 2.7 at /usr/bin/python.

This is Apple’s next level of warning us that that the pre-installed Python (and Perl and Ruby) is deprecated and going away in “future version of macOS.” I have written about this before.

Even though the management system will be identified as the process that “needs to be updated,” the culprits are scripts and scriptlets that the management system calls for for management tasks (e.g. policies, tasks, scripts) and information gathering (e.g. extension attributes, facts, etc.). Ben Tom’s post above has information on how to identify scripts which may use python in a Jamf Pro server.

You can suppress the warning using a configuration profile. While this a useful measure to avoid confusing users with scary dialogs, you will have to start identifying and fixing scripts that are written entirely in python or just use simple python calls, and replacing them with non-python solutions.

Python 2.7 is not getting any more security patches and I assume Apple is eager to remove it from macOS. The clock is really ticking on this one.

Current User

The most common python call is probably the one which determines the currently logged in user. The python call for this was developed by Mike Lynn and popularized by Ben Toms in this post and has been a reliable MacAdmin tool for years. I have written about this and introduced a shell-based solution discovered by Erik Berglund.

But there are other use cases, where it is not so straight forward to replace the python code. The built-in python is so popular for MacAdmin tasks because it comes with PyObjC which allows access to the macOS system frameworks. With a few python calls you can avoid having to build an Objective-C or Swift command line tool.

Desktop Picture

I built desktoppr for this reason. The standard way to set a desktop picture with locking it down was a line of AppleScript. But, starting in macOS Mojave, sending AppleEvents to another process (in this case Finder) required a PPPC profile. You can also set the desktop picture using a framework call. There were python scripts out there, but the Swift solution will survive them…

Available Disk Space

Yesterday, I came across another such problem. With the recent versions of macOS, getting a value of the available disk space is not as strightforward as it used to be. There are a lot of files and data on the system, which will be cleared out when some process requires more disk space. Most of this is cache data or data that can be restored from cloud storage. But this ‘flexible’ available disk space will not be reported by the traditional tools, such as df or diskutil. The available disk space these tools report will be woefully low.

The available disk space which Finder reports will usually be much higher. There is functionality in the macOS system frameworks where apps can get the values for available that takes the ‘flexible’ files into account. There is even useful sample code!

Starting with this sample code, I built a command line tool that reports the different levels of ‘available’ disk space. When you run diskspace it will list them all. There are raw and ‘human-readable’ formats.

> diskspace                  
Available:      70621810688
Important:      231802051028
Opportunistic:  214051607271
Total:          494384795648
> diskspace -H              
Available:      70.62 GB
Important:      231.8 GB
Opportunistic:  214.05 GB
Total:          494.38 GB

The ‘Available’ value matches the actually unused disk space that df and diskutil will report. The ‘Important’ value matches what Finder will report as available. The ‘Opportunistic’ value is somewhat lower, and from Apple’s documentation on the developer page, that seems to be what we should use for automated background tasks.

For use in scripts, you can get each raw number with some extra flags:

> diskspace --available               
70628638720
> diskspace --important
231808547284
> diskspace --opportunistic
214057661159
> diskspace --total
494384795648

You can get more detail by running diskspace --help.

In Scripts

If you wanted to check if there is enough space to run the macOS Monterey upgrade (26 GB) you could do something like this:

if [[ $(/usr/local/bin/diskspace --opportunistic ) -gt 26000000000 ]]; then
     echo "go ahead"
else
    echo "not enough free disk space"
fi

Jamf Extension Attributes

Or, you can use diskspace in a Jamf Extension Attribute:

#!/bin/sh

diskspace="/usr/local/bin/diskspace"

# test if diskspace is installed
if [ ! -x "$diskspace" ]; then
    # return a negative value as error
    echo "<result>-1</result>"
fi

echo "<result>$($diskspace --opportunistic)</result>"

Since, this extension attribute relies on the diskspace tool being installed, you should have a ‘sanity check’ to see that the tool is there.

Get and install the tool

You can get the tool from the GitHub repo and I have created a (signed and notarized) installer pkg that will drop the tool in /usr/local/bin/diskspace.

Scripting macOS, part 7: Download and Install Firefox

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Follow this blog or the Twitter account for updates on the book’s progress!

Download and Install Firefox

To further illustrate the progress from the idea of a workflow to a working script, let us look at another, more involved example.

To download and install the latest version of Firefox a user has to go to the Firefox website and download the latest version, which will come as a disk image (dmg) file. Then the user needs locate the dmg in the ~/Downloads folder and open it to mount the virtual disk image. Finally, they need to copy the Firefox application from the virtual disk to the Applications folder.

When we want to automate the task ‘Download and Install Firefox,’ we have the following steps:

  • download latest Firefox disk image
  • mount downloaded disk image
  • copy Firefox application to /Applications
  • unmount disk image

From this list of steps, we can build the first ‘frame’ of our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

This breaks the workflow into smaller pieces, that we will now tackle individually.

Download from the Command Line

You can use the curl command to download data in the command line. The curl command is very complex and has many options. We will only discuss the few options that we require for our task here. As always, you can find a detailed description of the curl command and its options in the curl man page.

The URI to download the latest Firefox is
https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US

However, when you try to curl this URI, you only get the following:

> curl "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US"
<a href="https://download-installer.cdn.mozilla.net/pub/firefox/releases/86.0.1/mac/en-US/Firefox%2086.0.1.dmg">Found</a>.

This is a re-direction, that is commonly used to have a single URI, that is redirected to different final URIs, so that when the software updates, the same URI always returns the latest version.

We can tell curl to follow these redirections with the --location option.

By default, the curl command will output the download to standard out. To save the download to a file, we can use the --output option with a file name.

> curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" --output Firefox.dmg

This command will download the latest Firefox disk image to a file named Firefox.dmg in your current working directory. We can use this as our first step:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image

# copy Firefox application to /Applications

# unmount disk image

Note: Like many other command line tools, curl has short and long options. The short options for –location and –output are -L and -o.
Short options are convenient in the interactive shell, as they save typing and reduce the potential for typos. But they are much less readable, and you usually have to look up their function in the documentation. For that reason, I recommend using the long, descriptive options in scripts.

Working with Disk Images

The command line tool to work with disk image (dmg) files on macOS is hdiutil. This is also a very powerful command with many verbs and options. You can find all the detail in the hdiutil man page.

To mount a disk image, use the attach verb:

> hdituil attach Firefox.dmg

This will output some information and mount the virtual disk. The last line ends with the path to the mounted virtual disk /Volumes/Firefox.

By default, you can see the mounted volume in Finder. We do not really need the disk image to appear in Finder while the script is running. We can suppress this behavior with the -nobrowse option.

Since we are only going to read from the disk image, we can tell hdiutil to mount the dmg in readonly mode with the -readonly option. This speeds things up a bit.

> hdiutil attach Firefox.dmg -nobrowse -readonly

You can unmount or eject the virtual disk with

> hdiutil detach -force /Volumes/Firefox

The -force option will unmount the disk image, even when another process is still using it.

Thehdiutil command covers two of our steps, so we can fill them in:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications

# unmount disk image
hdiutil detach /Volumes/Firefox -force

Copying the Application

When you manually install Firefox the disk image shows you a nice graphic that reminds you to drag the app to the Applications folder. Once the disk image is mounted, the cp command can be used to do this in the shell:

> cp -R /Volumes/Firefox/Firefox.app /Applications/

This provides the last missing step in our script:

#!/bin/zsh

# Download Firefox
#
# downloads and installs the latest version of Firefox


# download latest Firefox disk image
curl --location "https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US" \
     --output Firefox.dmg

# mount downloaded disk image
hdiutil attach Firefox.dmg -nobrowse -readonly

# copy Firefox application to /Applications
echo "copying Firefox to /Applications"
cp -R /Volumes/Firefox/Firefox.app /Applications/

# unmount disk image
hdiutil detach /Volumes/Firefox/ -force

You can now test the script. If Firefox is running, you want to quit it before you run the script. You may also want to delete the existing copy of Firefox from the Applications folder, to be sure that your script is doing the work.

Lists of Commands—Conclusion

We have been able to automate a fairly complex workflow with a script of four commands.

To be perfectly honest, this script (as well as all the others we have built so far) is not complete yet.

A ‘proper’ script needs to be able to react to errors that occur. In our example, imagine the download fails. The script should be able to detect the failure before it overwrites the installed, functional Firefox application.

We will get to this kind of error handling later.

Nevertheless, this script is already useful in its current form. You can try to adapt this script to work with some other software you can download as a disk image.

You can also add extra commands that

  • delete the downloaded disk image at the end
  • open the newly installed Firefox app after installation
  • quit or kill the Firefox process before copying the new version

In the book “Scripting macOS”, you will learn more scripting techniques, and we will re-visit some of these sample scripts and keep improving them.

Follow this blog or the Twitter account for updates on the book’s progress!

Note: After using different variations of these kinds of workflows, I did put together a more generic script to download and install various kinds of software, called ‘Installomator.’ You can see the script at its open source repository on GitHub.

Scripting macOS, part 6: Turn it off and on again

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Turn it off an on again

A common strategy to solve problems with computers is ‘turning it off and on again.’ For example, when you have Wi-Fi connectivity problems, one of the first things you should try it to turn the Wi-Fi off, wait a few seconds, and then turn it on again.

Let’s write a script to do that.

Create the ‘frame’

When you start out writing a script, you will usually have only a vague idea of what needs to be done, and even less an idea of how to actually do it. In this case is helps to write out the steps that your script should perform:

  • turn off Wi-Fi
  • wait a few seconds
  • turn on Wi-Fi

With this list we have broken down the problem into smaller, more manageable steps, which we can each solve on their own.

I usually take this list of steps and build a script ‘frame’ from them:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi

# wait for a few seconds

# turn on Wi-Fi

We just copied our list of steps into the text file for our script and made them comments by adding the # character at the beginning of the line. We also added the shebang in the first line and two more comment lines at the beginning, which name the script and have short description on what it is supposed to do.

Since the script in this form only consists of the shebang and comments, it does nothing. But it provides a frame to fill in. Now we can tackle these steps one at a time.

Control Wi-Fi

When you want to control network related settings on macOS, the networksetup command is the first place you should look. The networksetup command allows you to configure the settings in the ‘Network’ pane in ‘System Preferences’— and a few more. The networksetup command has dozens of options. You can get a list and descriptions by running networksetup -help or in the networksetup man page.

To turn off Wi-Fi, you can use this option:

> networksetup -setairportpower <hwport> off

The value you need to use for <hwport> depends on what kind of Mac you are working on. For MacBooks, it will be en0 (unless you have a very old MacBook with a built-in ethernet port). For Macs with a single built-in ethernet port, it will be en1. For Macs with two built-in ethernet ports it will be en2.

You can also use the networksetup command to list all available hardware ports and their names:

> networksetup -listallhardwareports

Hardware Port: Wi-Fi
Device: en0
Ethernet Address: 12:34:56:78:9A:BC

Look for the Hardware Port named Wi-Fi. The value shown next to Device is the one you have to use. So, for me, on a MacBook, I will use:

> networksetup -setairportpower en0 off

We will use en0 in our sample script going forward. If your Wi-Fi port is different, remember to change it in your script going forward

Note: Apple used to brand Wi-Fi as ‘Airport’ and this naming still lingers in some parts of macOS. Changing networksetup’s options to match the new branding would break all existing scripts.

When you replace the off with on it will turn the Wi-Fi back on:

> networksetup -setairportpower en0 on

We have solved the first and third step for our script and tested them in the interactive shell. Now, we can fill them into our script frame:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi
networksetup -setairportpower en0 off 

# wait for a few seconds

# turn on Wi-Fi
networksetup -setairportpower en0 on

You can now save the script as reset_wifi.sh, set its executable bit and run it:

> chmod +x reset_wifi.sh
> ./reset_wifi.sh

This should already work. You should see the Wi-Fi icon in the dock switch to the animation indicating that it is re-connecting with your network.

Taking a break

We still want the script to ‘take a break’ and wait for a few seconds between the turning off and turning back on commands. This will allow other parts of system to ‘settle’ and react to the change.

In the shell you can use the sleep command to achieve this. It takes a single argument, the time in seconds it should pause before continuing:

> sleep 5

When you enter this command in the interactive shell, you should notice that it takes 5 seconds for the next prompt to appear. The sleep command is delaying progress for the given time.

When you watch the CPU load in Activity Monitor while you run the sleep command, you will not see a spike in load. The sleep command merely waits, letting other processes and the system do their thing.

Let us insert a ten second break into our script between the commands to turn Wi-Fi off and on:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again

# turn off Wi-Fi
networksetup -setairportpower en0 off 

# wait for ten seconds
sleep 10

# turn on Wi-Fi
networksetup -setairportpower en0 on

Now, when you run the script. You can see that Wi-Fi icon in the menu bar should switch to ‘disabled’ and then return to the ‘searching’ animation and eventually reconnect after ten seconds.

Feedback

While you are running this script, there is no feedback in the Terminal though. It just takes a disconcerting time for the prompt to return. We can add some output to let the user of the script know what is happening:

#!/bin/zsh

# Reset Wi-Fi
# by turning it off and on again


# turn off Wi-Fi
echo "Disabling Wi-Fi"
networksetup -setairportpower en0 off 

# wait for a ten seconds
echo "Waiting..."
sleep 10

# turn on Wi-Fi
networksetup -setairportpower en0 on
echo "Re-enabled Wi-Fi"

Now, the user will get some feedback in Terminal that lets them know what is going on.

>  ./reset_wifi.sh
Disabling Wi-Fi
Waiting...
Re-enabled Wi-Fi

Script Building Process

We formed an idea or a goal: ‘Turn Wi-Fi off and back on’

Then we described the steps necessary to achieve that goal:

  • turn off Wi-Fi
  • wait a few seconds
  • turn on Wi-Fi

We used these descriptive steps to create a frame or scaffolding script to ‘fill in’ with the proper commands.

Then we explored commands that would achieve these steps in the interactive terminal.

Once we determined the correct commands and options we placed them into our script ‘frame’ in the correct order.

As you add the commands, test whether the script shows the expected behavior.

Then we also added some output, to provide user feedback.

This is a really simple example script to illustrate this process. Nevertheless, we will re-visit these process steps with every script we build and you should follow this process (or a similar flow) when building your own scripts.

The process will not always be quite so clearly defined and you may have to iterate or repeat some of these tasks multiple times, before you find a combination of commands that do what you want. Less experienced scripters will naturally spend more time on ‘exploring commands.’ This is part of the normal learning process.

With larger, more complex scripts, each descriptive step may need to broken into even smaller sub-steps to manage the complexity. Again, experienced scripters will be able tackle more complex steps faster. Do not let that intimidate you when you are just beginning! Experience has to be built ‘the hard way’ by learning and doing.

In many situations you can learn from other people’s scripts that solve a similar problem. This is a well-proven learning strategy. I would still recommend to ‘explore’ the commands used in other people’s scripts before adding them to your scripts, as it will deepen your knowledge and experience with the commands. Sometimes, you may even be able to anticipate and avoid problems.

Next Post: Download and Install Firefox

Installomator v0.7b1 – Prerelease

We have posted a new version of Installomator. This one brings with it major changes in how we assemble the actual script. Since this is such a big change, we decided to do a beta release first.

The changes in detail:

  • script is now assembled from fragments. This helps avoid merging conflicts on git and allows the core team to work on the script logic while also accepting new labels. See the “Assemble Script ReadMe” for details.
  • Change in finding installed apps. We now look in /Applications and /Applications/Utilities first. If not found there, we use spotligt to find it. (We discovered a problem when a user has Parallels Windows installed with Microsoft Edge in it. Then Installomator wanted to update the app all the time, becaus spotligt found that Windows version of the app that Parallels created.)
  • Added bunch of new labels
  • Improved buildCaseStatement.sh a lot. It is a great start when figuring out how to create a new label for an app, or a piece of software.
  • Mosyle changed their app name from Business to Self-Service

Why the changes?

Since the Installomator.sh script has grown to over 3000 lines, its management on git has become very unwieldy. The single file with all the logic and the data required to download and install the applications creates constant merge conflicts which add to the workload of the repo admins, especially when part of the team is working on the logic of the script while we still get PRs to add labels.

Because of that we have split the main script into multiple files which are easier to manage. Having multiple files results in less merge conflicts.

What changes when I use the script?

Nothing. When you just use the Installomator.sh, you still copy its contents from the Installomator.sh script at the root of the repository into your management service (don’t forget to change the DEBUG value). Or you install the script to the clients using the installer pkg from the Releases.

The changes will only affect you when you want to build your own application labels, modify existing labels or other wise modify the script.

How do I build my own labels now?

This is where you need to learn about the new system. To reduce merge conflicts, we have broken the big Installomator.sh script into smaller pieces. There is a utility script that can assemble the script from the pieces and even run it right away fro testing. You can get the details in the “Assemble script ReadMe”

We hope that these changes will make it easier for the Installomator team and other contributors to keep growing and improving the script.

Scripting macOS, part 5: Lists of Commands

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week. Enjoy!

Beyond Hello, World

Now that you have a minimal working script, let’s extend it a bit.

Create a copy of the hello.sh script file, set its executable bit, and open it in your favored text editor:

> cp hello.sh hello_date.sh
> chmod +x hello_date.sh
> bbedit hello_date.sh

A script is a list of commands that the interpreter will process sequentially. Up to now, we only have a single command, the line with echo, in our script. We will add another one:

Change the text in the script file:

#!/bin/zsh

# Greetings
echo "Hello, World!"
date

We have added a line to the script with the date command.

If you are unfamiliar with this command, you can try it out in the interactive command line:

> date    
Tue Feb 23 10:23:05 CET 2021

When you invoke date with out any arguments it will print out the current date and time. The date command has many other functions for doing date and time calculations which we will not use right now, but you can read about in the date man page.

Save the modified script and execute it:

> ./hello_date.sh
Hello, World!
Tue Feb 23 10:27:06 CET 2021

As we can tell from the output, each command in the script was executed one after the other and the output of each command was shown in Terminal.

You can insert more echo commands before the date to make the output prettier:

#!/bin/zsh

# Greetings
echo "Hello, World!"

# print an empty line
echo

# print without line break
echo -n "Today is: "

date

As we have learned earlier, empty lines and lines starting with a # character will be ignored, but can serve to explain and clarify your code. I will be using comments in the example scripts for quick explanations of new commands or options.

Here I added the -n option to the third echo command. This option suppresses the new line character or line break that echo adds automatically at the end of the text. This will then result in the output of the date command to print right after the ‘Today is:’ label, rather than in a line of its own.

Take a look:

> ./hello_date.sh
Hello, World!

Today is: Tue Feb 23 10:30:44 CET 2021

The date command allows for changing the format of the date and time output. It uses the same formatting tokens as the C strftime (string format time) function. You can read the strftime man page for details. The %F format will print the output as ‘year-month-date:’

> date +%F
2021-02-23

You can combine place holders:

> date +"%A, %F"
Tuesday, 2021-02-23

You can experiment with the date formatting placeholders from the strftime man page in the interactive shell. Once you have built a formatter that you like, you can add it to the date command in your script.

#!/bin/zsh

# Greetings
echo "Hello, World!"

# print an empty line
echo

# print without line break
echo -n "Today is: "

date +"%A, %B %d"

It is very common that you will test and iterate a command and arguments in the interactive shell before you add or insert it into your script. This can be a much easier and safer means of testing variations of commands and options than changing and saving the entire script and running it repeatedly.

Scripts are basically ‘lists of commands.’ Once you know the steps to perform a workflow in the interactive terminal, you can start building a script. Let’s look at another example.

Desktop Picture Installer

When you copy an image file to /Library/Desktop Pictures, it will appear in the list of pictures in the ‘Desktop & Screen Saver’ pane in System Preferences. On macOS 10.15 Catalina and higher you may have to create that directory.

You can easily build an installer package (pkg file) that installs an image file into that location with the pkgbuild command.

First, create a directory to hold all the sub-directories files we will need. There needs to be a payload directory in the project directory. Copy the image file into the payload directory:

> mkdir BoringDesktop
> cd BoringDesktop
> mkdir payload
> cp /path/to/BoringBlueDesktop.png payload

You can then build an installer package with the following command:

> pkgbuild --root payload --install-location "/Library/Desktop Pictures/" --identifier blog.scripting.BoringBlueDesktop --version 1.0 BoringDesktop-1.0.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to BoringDesktop-1.0.pkg

This will create a file BoringDesktop-1.0.pkg. When you double-click this file, it will open the Installer application and, when you proceed with the installation, put the image file in /Library/Desktop Pictures.

Note: To learn more about using and building installer package files for macOS, read my book “Packaging for Apple Administrators.”

The pkgbuild command has a lot of arguments and when you get any of them just slightly wrong, it may affect how the installer package works. This is a very simple example, but when you build more complex installer packages, you may be building and re-building many times and you want to avoid errors due to typos.

We can copy this big command and place it in a script file:

#!/bin/zsh
pkgbuild --root payload --install-location "/Library/Desktop Pictures/" --identifier blog.scripting.BoringBlueDesktop --version 1.0 BoringDesktop-1.0.pkg

Then make the script file executable:

> chmod +x buildBoringDesktopPkg.sh

Now you can just run the script and not worry about getting the arguments ‘just right.’

But we can go one step further and make the script more readable. We can add a comment describing what the script does.

The shell usually assumes the line break to be the end of the command. But when you place a single backslash \ as the last character in a line, the shell will continue reading the command in the next line. That way, you can break this long, difficult to read command into multiple lines:

#!/bin/zsh

# builds the pkg in the current directory

pkgbuild --root payload \
         --install-location "/Library/Desktop Pictures/" \
         --identifier blog.scripting.BoringBlueDesktop \
         --version 1.0 \
         BoringDesktop-1.0.pkg

The arguments are now more readable. When you want to change one of the values, e.g. the version, it is easier to locate.

Even though this script only contains a single command, it improves the workflow significantly, as you do not have to remember a long complex command with many arguments.

Of course, you can copy and modify this script for other package building projects.

Next Post: Turning it off an on again

Scripting macOS, part 4: Running the Script

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week over the summer. Enjoy!

Running the Script

Now that we have the code for a functional (even though minimal) script, you can run it from the command line with:

> ./hello.sh

(Verify that the current directory is your script project folder.) You probably have wondered why you need the ./ before the script name. When you type the script name without the ./ it will fail:

> hello.sh
zsh: command not found: hello.sh

What does the ./ mean and why do we need it?

Current Working Directory

In the shell, the . character represents the current working directory or the directory that you ‘cd-ed’ to. Usually, the working directory is shown in the window title bar of Terminal and in the shell prompt (the status text shown before your cursor in the Terminal window). You can also have the system show your current working directory with the pwd command:

> pwd
/Users/armin/Projects/ScriptingMacOS

When you type ./hello.sh the shell will substitute the current working directory for the . character to get

/Users/armin/Projects/ScriptingMacOS/hello.sh

Your path will of course look different.

This means that when I use the ./hello.sh form, the shell knows exactly which file I want to execute and where to look for it. There is no ambiguity.

Now we know what the ./ means, but why is it necessary?

Finding Commands

A command you enter in the shell is just a string of text to begin with. The shell has to parse this text into pieces to determine what needs to be done.
We will use the chmod command from earlier as an example:

> chmod +x hello.sh

The text you entered is ‘chmod +x hello.sh’. The shell will split this text into pieces on the spaces (or tab characters). This will yield three elements:
chmod,’ ‘+x,’ and ‘hello.sh

The shell is only really interested in the first element. The shell will try to interpret the first element of your entry as a command.

Note: Many programming languages start counting at zero. This first element is also called ‘argument zero’ or $0.

The remaining elements are arguments, which the shell will pass on to the command. Arguments are optional. Not all commands require or even have arguments.

Note: This is a simplified description of the parsing process in shells. The reality is quite a bit more complex. But this description is ‘close enough’ for most situations. We will explore some of the nuances later.

If you are interested, you can get all the details in the shell’s documentation:

  • Zsh Documentation: Shell Grammar
  • bash man page, search for ‘Command Execution’

When there is a ‘/‘ character anywhere in the first element, the shell will interpret the first element as a full or relative path to an executable file and attempt to run that.

This is the behavior we are using when we type ./hello.sh. Since that contains a ‘/‘ the shell will resolve the path and execute our script.

We are intentionally using the seemingly redundant ./ prefix to tell the shell to run the script from the current working directory.

When the first element does not contain a ‘/,’ the shell will check if it is one of the following:

  • a shell function
  • a shell built-in command or reserved word
  • an external command

Shell Functions

Shell functions are (mainly) customized shell behavior declared by the user. They look and work like commands.

Note: If you are interested in customizing your shell environment, you can find details in my books “Moving to zsh” and “macOS Terminal and Shell.”

Shell Built-ins

Shell built-in commands cover tasks that are either inherent to the shell or can be performed much faster within the shell than as an external command. These are commands that affect the internal state of the current shell process like cd, alias, or history, or commands that are simpler and faster to implement as built-ins like echo or read.

You rarely need to worry about whether a command is a built-in.

Many built-in commands will have an executable external command file as well. This serves as a ‘fallback’ for the rare situation that the shell built-in is not available.

You can use the (aptly, but confusingly, named) command built-in to determine if a command is built-in or external:

> command -V cd
cd is a shell builtin
> command -V sw_vers
sw_vers is /usr/bin/sw_vers

External Commands

When the command entered is neither a function, nor a built-in, and does not contain a ‘/,’ then the shell will go search for an executable file with that name.

The PATH environment variable determines the locations in the file system and the order in which to search them.

The default PATH on macOS in an interactive shell is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The PATH variable contains a colon-separated list of directories. When you have third-party software installed, or customized your shell configuration, your shell may have additional directories. The default PATH splits into the following five directories:

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

Note: Variable names in the shell are case-sensitive. The variable names MY_VAR and my_var represent different variable and values.
Zsh, however, has the concept of ‘connected variables.’ In zsh, PATH and path are connected variables. The upper-case PATH contains the colon-separated list of directory paths, while the lower-case path is an array of paths. The actual list of directories will be the same and changing one variable will change the other. Zsh has this concept to maintain compatibility with other shells with the PATH, while also allowing you to use array operators on the path.
I will use the colon-separated PATH, because it is more compatible with other shells.

When a user enters a command like this:

> system_profiler

This is neither a function or alias, nor a built-in command. It also does not contain a ‘/.’

The shell will check for the presence of an executable file with the name system_profiler in all the directories given in the PATH. It will start with /usr/local/bin, then /usr/bin, then /bin, until it finally finds a matching file in /usr/sbin.

Then the shell will attempt to execute /usr/sbin/system_profiler.

When no matching files can be found in any of the directories listed in the PATH, the shell will present a ‘command not found’ error.

> cantFindMe
zsh: command not found: cantFindMe

If you are curious which file the shell will use for a given command you can use the command or the which tool:

> command -V system_profiler
system_profiler is /usr/sbin/system_profiler
> which system_profiler
/usr/sbin/system_profiler

PATH precedence

Once the shell finds a matching file, it will stop searching the remaining paths. If there were a second matching executable in /sbin, the shell would never find and execute it. The order of the directory paths given in the PATH variable determines the precedence.

We can test this by placing an executable with a file name matching an existing command in /usr/local/bin. Since /usr/local/bin comes first in the default interactive PATH, the shell should prefer our executable over the default command.

> sudo ditto hello.sh /usr/local/bin/system_profiler

You need administrator privileges to modify the contents of /usr/local/bin. The ditto command preserves all the file’s metadata (like privileges and extended attributes), which makes it preferable to cp for this task.

Then open a new Terminal window. You need to do this because every shell instance will cache or ‘remember’ the lookup for a command to speed up the process later. The shell instance in your current terminal window will remember the last lookup for the system_profiler command. A new Terminal window will start a new ‘fresh’ shell instance, forcing a new lookup:

> which system_profiler
/usr/local/bin/system_profiler
> system_profiler
Hello, World!

Obviously, overriding a system provided command this way can break your workflows and scripts. To be safe, remove our script from /usr/local/bin right away:

> sudo rm /usr/local/bin/system_profiler

There are some situations where overriding a system-provided command is desirable, though. For example, you could install the latest version of bash 5 as /usr/local/bin/bash. And then, when you invoke bash from your interactive terminal, you will launch that version, instead of the outdated version that comes with macOS.

Note: You cannot simply overwrite /bin/bash with the newer version on macOS. The /usr/bin, /bin, /usr/sbin, and /sbin folders are protected by System Integrity Protection and the read-only system volume.

When bash is invoked with the absolute path /bin/bash, the path will need to be updated to use the newer version, though. This includes the shebangs in scripts, and the UserShell attribute in a user’s account record for their default shell.

Extending your tool set

As you get more confident and experienced with scripting, you will assemble a set of scripts that you will use regularly. When you use a script often, it would be nice if the shell recognized them as commands, without having to type the path to them.

To achieve that you can add the directory containing the scripts to the PATH variable. I put my frequently used tools in ~/bin. The name is chosen to be somewhat consistent with the four standard locations for tools, but really does not matter. You can append your tool directory to the PATH with:

> export PATH=$PATH:~/bin

This reads as: replace the value of the environment variable PATH with its current value and append :~/bin. You can verify the new value with

> echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/armin/bin

We added our custom directory to the end of the PATH variable, so you cannot accidentally override system tools.

To prevent other local users from changing your command files, you should set the directory’s privileges, so that only you can access:

> chmod 700 ~/bin

Now you can copy our hello.sh script to that folder. We will rename it to just hello, so it looks more like a command:

> ditto hello.sh ~/bin/hello

Then you can run your script just by typing

> hello
Hello, World!

Most of the scripts you build will not need to be as accessible as this.

Since you do not want to manually change the PATH every time you create a new Terminal window, you should add this line to your shell configuration file:

export PATH=$PATH:~/bin

Context Matters

Leveraging PATH to find and possibly override commands can be a useful and powerful tool. However, when scripting, you have to always remember, that the interactive Terminal environment may not be the environment that your script will run in ‘production.’

This is especially important for scripts run by

  • LaunchDaemons or LaunchAgents
  • AppleScripts
  • applications other than Terminal (such as Xcode)
  • installer packages
  • management systems

When run from any of these environments, the environment the script runs in will be different from your interactive shell environment. Most likely the PATH in any of these environments will be set to the minimal four system folders:

/usr/bin:/bin:/usr/sbin:/sbin

This should work well for most commands and tools. But when you are working with third party software, especially newer versions, then you need to pay very close attention.

When you build scripts for these environments, I recommend setting the PATH at the beginning of your script explicitly to the value you need:

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

When your scripts require tools, include their locations in the PATH. This way, the PATH environment is declared explicitly at the beginning of the script and there can be no confusion.

For the scripts in this series, the default PATH will be sufficient, so we will not need to worry about this.

Next: Lists of Commands

Scripting macOS, part 3: The Code

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week over the summer. Enjoy!

The Code

A script file is a text file with the executable bit set.

You can set the executable bit on any file. That alone does not turn it into a working script file. To get a working script file, the contents need to have the right structure or ‘syntax.’

Our example above is close to the minimum you need to have a working script file. We will look at the contents one line at a time.

#!/bin/zsh

# Greetings
echo "Hello, World!"

Shebang

The first line in our script is:

#!/bin/zsh

The first two characters #! form a special file signature or ‘magic number.’ They tell the shell and the system that this is not just any file, but a script file that should be interpreted.

These two characters have special names. The number sign is called the ‘hash’ and the exclamation mark is called the ‘bang.’ Together they are the ‘hashbang’ or the ‘shebang.’

The shebang characters have to be followed by the full path to the interpreter binary. There are no spaces in between the shebang characters and the interpreter path.

In this file, we have designated the zsh binary /bin/zsh.

When you type ./hello.sh to execute your script in Terminal, the shell will check if the file is executable. If it is executable, the shell will look at the first characters of the file. It ‘sees’ the shebang #! characters, and determines from them that this is a script file. Then it will read the rest of the first line to determine the interpreter. It will then hand over the entire script file to that interpreter binary.

This will have the same effect as running your script like this:

> /bin/zsh hello.sh

A new zsh process starts which reads and interprets the script file.

It does not matter which shell you use as your interactive shell. The shebang determines the interpreter for the script. That means you can run bash and sh scripts from zsh and vice versa.

When you are writing your scripts against sh or bash, you will have a shebang line of #!/bin/sh or #!/bin/bash, respectively.

The three shells are pre-installed as part of macOS and protected by System Integrity Protection and the read-only system volume. When you install newer versions of the interpreters, such as bash v5, then that binary will be in a different location. Most commonly /usr/local/bin.

If, for example, you have bash v5 installed as /usr/local/bin/bash, the shebang lines in the scripts that should use bash v5 will be:

#!/usr/local/bin/bash

When you then try to run these scripts on a mac that does not have bash v5 installed (or installed in a different location) then the scripts will fail.

> ./hello.sh
zsh: ./hello.sh: bad interpreter: /usr/local/bin/bash: no such file or directory

This can also happen when you run your script on a different platform. Even though the other platform may have the bash or zsh binary installed, it could be at a different path than on macOS. And the interpreter binary may be a different version than on macOS (especially with bash) which can lead to the script being interpreted differently.

For scripts that need to work in many different environments you will often see a shebang that uses the env command:

#!/usr/bin/env bash

The env command will use the current user’s environment, especially the PATH, to determine where the bash binary is and passes the script on to that. This will make your shebang more flexible, but you also give up some control as the environment will differ from system to system and from user to user.

For situations where you need control—such as management scripts—a fixed path for the shebang should be preferred, but then you also have ensure that the interpreter binary will be available at that path.

Whitespace

The line right after the shebang is empty. In shell scripts (and most other programming languages) empty lines are just ignored. Adding empty lines will make your code more readable.

Comments

The third line in our script starts with a number character or ‘hash.’

# Greetings

In shell scripts, lines that start with the hash character are ignored by the interpreter. You can and should use this to leave explanations and comments in your code.

Use comments in your code to leave explanations and notes.

echo Command

Now, finally, we will get to the actual code. The part of the script that does something.

echo "Hello, World!"

The echo command is one command that you rarely use in the interactive shell, but very frequently within scripts. It tells the shell to print (repeat or ‘echo’) text to the terminal’s output.

As usual in shells, the text we pass in to echo as an argument is ‘held together’ by quotes. This kind of text used in code is also called a ‘string’ because it is a ‘string of characters.’

Next: Running the Script

Scripting macOS, part 2: The Script File

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week over the summer. Enjoy!

The Script File

In the previous part, we created a text file and named it hello.sh. As we have mentioned before, all shell script files are text files.

File Extension

The file extension .sh is entirely optional. When you rename the file to merely hello it will remain a functional script:

> mv hello.sh hello
> ./hello
Hello, World!

When you use the script frequently as a command line tool, typing the additional .sh is cumbersome and does not ‘fit in’ very well with all the other command line tools. It is very common to remove the file extension in these cases.

However, the .sh file extension tells Finder and other applications what kind of file this is. You can assign the .sh file extension to be opened in your favored text editor when you double-click the file in Finder.

To do this, rename the file so it has the extension again:

> mv hello hello.sh

Then select the script file in Finder and open its Info panel (⌘I).

In the ‘General’ section of the info panel, you can see that Finder recognizes this file extension as a shell script.

In the ‘Open with’ section you can tell Finder which application to use to open this file. The popup list will show all applications you have installed that can open .sh files. Select your favorite text editor.

To tell the system to open all files with a .sh extension with your favorite text editor, click on the ‘Change All…’ button here. This will set your favorite text editor as the default application for the .sh file extension.

Some people like to use .bash and .zsh file extensions to further distinguish scripts written specifically in bash or zsh. While this can be useful in some workflows—especially when translating scripts from bash to zsh—it is generally not necessary. The .sh file extension works well for all these scripts. The .bash and .zsh file extensions are also not automatically recognized by the Finder or text editors as shell scripts.

Executable Bit

Aside from the actual script code, a shell script requires one more thing that distinguishes it from any other text file. You have to the tell the system that this file can be executed as a command.

This information is stored in the executable bit of the file privileges or file mode. You can see this information with the long form of the ls command:

> ls -l hello.sh 
-rwxr-xr-x@ 1 armin  staff  hello.sh

The file mode is displayed in the first ten characters in this output: -rwxr-xr-x.

The leading dash designates this item as a file. (A directory will show d, a symbolic link will show l, etc.)

The file access privileges are shown next. There are three sets of three flags or bits. The first three (in this case rwx) are the privileges for the file’s owner. The second set (r-x) are the privileges for group access and the last set (also r-x) shows the privileges for all other user on this system.

The three flags in each set are r, w, and x, in that order. They stand for read (r), write (w) and execute (x) and show the actions the users associated with each set can perform.

The file’s owner can read, write (or change), and execute (or run) this file. Users who are a member of the staff group, can read and execute this file, as well as every other user on this system.

Most documents and files merely contain text and data. The execute bit is not required. When you create a new file, the executable flag is disabled by default:

> touch newfile.txt
> ls -l newfile.txt
-rw-r--r--  1 armin  staff  newfile.txt

Having the execute bit disabled by default is a sensible security measure. You will have to explicitly enable it when building scripts or other executables.

When you are writing shell scripts, you eventually want to execute them. To do this you have to enable the execute bit. When then execute bit is unset, you cannot run the script from the command line.

To demonstrate this—and to gain some familiarity with the command line tools required to manipulate the file mode—disable the executable bit of our hello.sh script:

> chmod -x hello.sh

You can verify that the executable bit has been removed with the ls -l command:

> ls -l hello.sh
-rw-r--r--@ 1 armin  staff  45 Feb 17 14:36 hello.sh

When you now attempt to run the command, it will fail and the shell will tell you the reason: you do not have the right permissions. Without the executable script, your shell script is merely a text file.

> ./hello.sh
zsh: permission denied: ./hello.sh

You can re-set the executable bit with

> chmod +x hello.sh

You do not need write (w) permissions to execute a script, but you require read (r) permissions.

You can remove the executable privilege for group and/or other users for security. Since most Macs are used as single user systems this is not really relevant here. When you create, install, and run shell scripts on shared servers with other Unix or unix-like servers with many users, the proper access privileges may be extremely important. Please consult with the system administrator.

You will usually have to remember to set the executable bit when you create a new script. Some text editors may set it for you when they recognize you are working on a shell script.

Note: The executable bit has a different meaning for directories than it does for files. You can find the details in the chmod man page or in my book ‘macOS Terminal and Shell

Next: The Code

Scripting macOS, part 1: First Script

This series is an excerpt from the first chapter of my upcoming book “Scripting macOS” which will teach you to use and create shell scripts on macOS.

I will publish one part every week over the summer. Enjoy!

First Script

When learning programming languages it is tradition that the first program you write displays ‘Hello, World!’

Printing these words to the screen is usually a really simple task. In most programming languages it requires just a single line. Nevertheless, creating a simple program or script like this will teach you a lot about all the other tasks you need to create and run a script.

Before you start, you should create a folder where you store all the scripts you will create. A Documents subfolder is a good location. I have a Projects folder in my home directory where I create subfolders for all my scripting and programming projects. You may already have some structure like this that makes sense for you.

You can create this in Finder and the navigate there in Terminal. But, just to practice working in Terminal, you can create your script project folder in the command line:

> cd ~/Documents
> mkdir ScriptingMacOS
> cd ScriptingMacOS

Where- and however you create this folder, we will refer to it as your script project folder from now on. When you interact with the scripts from the command line, always remember to change the working directory there.

Note: When you know how to use a version control system to manage code, such as git, you can set up or initialize this project folder as repository now. Shell scripts are well suited to be managed with version control systems and I would recommend to use it when building scripts. But explaining the details of git as well as teaching shell scripting would be overwhelming and beyond the scope of this series.

Create your first script

Create a new text file in your favored text editor.

Note: If you do not have a favorite text editor yet, I recommend BBEdit. You can use it for free or pay the license fee to unlock the full feature set. After installing BBEdit, be sure to run ‘Install Command Line Tools…’ from the BBEdit menu.

I will be using the bbedit command line tools in my examples, but other text editors have similar commands.

When you are using a text editor with a command line tool, you can create a new empty file from the command line and open it in the text editor like this:

> bbedit hello.sh

Enter the following text into the text document:

#!/bin/zsh

# Greetings
echo "Hello, World!"

Tip: You can copy and paste the code from this post into the text editor. In general, I approve of this, since it speeds up the process and avoids typing errors. But in the beginning, I would recommend typing out the commands and modifying the script manually. This will engage your brain and memory differently and help you memorize some of the standard steps and commands.

Save the text file you created to a file named hello.sh in the script project folder.

In Terminal, ensure that the working directory is your script project folder. Then enter the following:

> chmod +x hello.sh

This will make your script file executable. You are telling the system this file has code that can be run as a program rather than just data. We will look at what this means in detail later.

Now you can run or execute your script:

> ./hello.sh
Hello, World!

If you get a different result, please verify that you typed the script exactly as given above. Pay extra attention to spaces, quotes and other special characters. Then save the text file and try running the script again:

> ./hello.sh
Hello, World!

Even for a script as simple as this, there is a lot that happened here. We will look at the pieces and steps in detail in the next part.

Next: The Script File