Shellcheck and zsh

The sad news is that the shellcheck binary does not really know how to deal with zsh scripts:

% shellcheck script.zsh

In script.zsh line 1:
#!/bin/zsh
^-- SC1071: ShellCheck only supports sh/bash/dash/ksh scripts. Sorry!

You can force shellcheck to interpret a zsh script with the --shell option:

% shellcheck --shell=bash dirlist.sh

This might show errors, such as Quote to prevent word splitting/globbing which aren’t really errors in zsh so its usefulness will be limited.

If you want a specific script to be always interpreted as a bash script by shellcheck you can also add this line:

# shellcheck shell=bash

in the script, right after the shebang line.

This can be useful if you use zsh in bash emulation mode with

emulate -LR bash

Moving to zsh, part 8 – Scripting zsh

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

This is the final article in this series. (If I ever announce an eight part series again, please somebody intervene!) However, I am quite sure it will not be the last post on zsh

All the previous posts described how zsh works as an interactive shell. The interactive shell is of course the most direct way we use a shell and configuring the shell to your taste can bring a huge boost in usefulness and productivity.

The other, equally, important aspect of a shell is running script files. In the simplest perspective, script files are just series of interactive commands, but of course they will get complex very quickly.

sh, bash, or zsh?

Should you even script in zsh? The argument for bash has been that it has been pre-installed on every Mac OS X since 10.2. The same is true for zsh, with one exception: the zsh binary is not present on the Recovery system. It is also not present on a NetInstall or External Installation System, but these are less relevant in a modern deployment workflow, which has to work with Secure Boot Macs.

If you plan to run a script from Recovery, such as an installr or bootstrappr script or as part of an MDS workflow, your only choices are /bin/sh and /bin/bash. The current /bin/bash binary is 12 years old, and Apple is messaging its demise. I would not consider that a future proof choice. So, if your script may run in a Recovery context, i would recommend /bin/sh over either /bin/bash or /bin/zsh

Since installation packages can be run from the Recovery context as well, and you cannot really always predict in which context your package will be used, I would extend the recommendation to use /bin/sh for all installation scripts as well.

While sh is surely ubiquitous, it is also a ‘lowest common denominator’, so it is not a very comfortable scripting language to work in. I recommend using shellcheck to verify all your sh scripts for bashisms that might have crept in out of habit.

When you can ensure your script will only run on a full macOS installation, zsh is good choice over sh. It is pre-installed on macOS, and it offers better and safer language options than sh and some advantages over bash, too. Deployment scripts, scripts pushed from management systems, launch daemons and launch agents, and script you write to automate your admin workflows (such as building packages) would fall in this category.

You can also choose to stick with bash, but then you should start installing and using your own bash 5 binary instead of the built in /bin/bash. This will give you newer security updates and features and good feeling that when Apple does eventually yank the /bin/bash binary, your scripts will keep working.

Admins who want to keep using Python for their scripts are facing a similar problem. Once you choose to use a non-system version of bash (or python), it is your responsibility to install and update it on all your clients. But that is what system management tools are for. We will have to get used to managing our own tools as well as the users’ tools, instead of relying on Apple.

Shebang

To switch your script from using bash to zsh, you have to change the shebang in the first line from #!/bin/bash to #!/bin/zsh.

If you want to distinguish your zsh script files, the you can also change the script’s file extension from .sh to .zsh. This will be especially helpful while you transfer scripts from bash to zsh (or sh). The file extension will have no effect on which interpreter will be used to run the script. That is determined by the shebang, but the extension provides a visible clue in the Finder and Terminal.

zsh vs bash

Since zsh derives from the same Bourne shell family as bash does, most commands, syntax, and control structures will work just the same. zsh provides alternative syntax for some of the structures.

zsh has several options to control compatibility, not only for bash, but for other shells as well. We have already seen that options can be used to enable features specific for zsh. These options can significantly change how zsh interprets your scripts.

Because you can never quite anticipate in which environment your particular zsh will be launched in, it is good practice to reset the options at the beginning of your script with the emulate command:

emulate -LR zsh

After the emulate command, you can explicitly set the shell options your script requires.

The emulate command also provides a bash emulation:

emulate -LR bash

This will change the zsh options to closely emulate bash behavior. Rather than relying on this emulation mode, I would recommend actually using bash, even if you have to install and manage a newer version yourself.

Word Splitting in Variable Substitutions

Nearly all syntax from bash scripts will ‘just work’ in zsh as well. There are just a few important differences you have to be aware of.

The most significant difference, which will affect most scripts is how zsh treats word splitting in variable substitutions.

Recap: bash behavior

In bash substituted variables are split on whitespace when the substitution is not quoted. To demonstrate, we will use a function that counts the number of arguments passed into it. This way we can see whether a variable was split or not:

#!/bin/bash
export PATH=/usr/bin:/bin:/usr/sbin:/sbin

function countArguments() {
    echo "${#@}"
}

wordlist="one two three four five"

echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 5

echo "substitution with quotes"
countArguments "$wordlist"
# -> 1

In bash and sh the contents of the variable split into separate arguments when substituted without the quotes. Usually you do not want the splitting to occur. Hence the rule: “always quote variable substitutions!”

zsh behavior: no splitting

zsh will not split a variable when substituted. With zsh the contents of a variable will be kept in one piece:

#!/bin/zsh
emulate -LR zsh # reset zsh options
export PATH=/usr/bin:/bin:/usr/sbin:/sbin

function countArguments() {
    echo "${#@}"
}

wordlist="one two three four five"

echo "normal substitution, no quotes:"
countArguments $wordlist
# -> 1

echo "substitution with quotes"
countArguments "$wordlist"
# -> 1

The positive effect of this is that you do not have to worry about quoting variables all the time, making zsh less error prone, and much more like other scripting and programming languages.

Splitting Arrays

The wordlist variable in our example above is a string. Because of this it returns a count of 1, since there is only one element, the string itself.

If you want to loop through multiple elements of a list

In bash this happens, whether you want to or not, unless you explicitly tell bash not to split by quoting the variable.

In zsh, you have to explicitly tell the shell to split a string into its components. If you do this naïvely, by wrapping the string variable in the parenthesis to declare and array, it will not work:

wordlist="one two three"
wordarray=( $wordlist )

for word in $wordarray; do
    echo "->$word<-"
done

#output
->one two three<-

Note: the for loop echoes every item in the array. I have added the -> characters to make the individual items more visible. In the subsequent examples, I will not repeat the for loop, but only show its output. So the above example will be shortened to:

wordarray=( $wordlist )
->one two three<-

There are few options to do this right.

Revert to sh behavior

First, you can tell zsh to revert to the bash or sh behavior and split on any whitespace. You can do this by pre-fixing the variable substitution with an =:

wordarray=( ${=wordlist} )
->one<-
->two<-
->three<-

Note: if you find yourself using the = frequently, you can also re-enable sh style word splitting with the shwordsplit option. This will of course affect all substitutions in the script until you disable the option again.

setopt shwordsplit
wordarray=( $wordlist )
->one<-
->two<-
->three<-

This option can be very useful when you quickly need to convert a bash script to a zsh script. But you will also re-enable all the problems you had with unintentional word splitting.

Splitting Lines

If you want to be more specific and split on particular characters, zsh has a special substitution syntax for that:

macOSversion=$(sw_vers -productBuild) # 10.14.6
versionParts=${(s/./)macOSVersion}
->10<-
->14<-
->6<-

If you want to split on a newline character \n the syntax is slightly different:

citytext="New York
Rio
Tokyo"

cityarray=( ${(ps/\n/)citytext} )
->New York<-
->Rio<-
->Tokyo<-

Since newline is a common character to split text on, there is a short cut:

cityarray=( ${(f)citytext} )

Since the newline character is a legal character in file names, you should use zero-terminated strings where possible:

foundDirs=$(find /Library -type d -maxdepth 1 -print0)
dirlist=${(ps/\0/)foundDirs}

Again, there is a shortcut for this:

dirlist=${(0)foundDirs}

Array index starts at 1

Once you have split text into an array, remember, that in zsh array indices start at 1:

% versionList=( ${(s/./)$(sw_vers -productVersion)} )
% echo ${versionList[1]}
10
% echo ${versionList[2]}
14
% echo ${versionList[3]}
6

If you think this is wrong and absolutely require a zero-based index, you can set the KSH_ARRAYS shell option:

% setopt KSH_ARRAYS
% echo ${versionList[0]}
10
% echo ${versionList[1]}
14
% echo ${versionList[2]}
6
% echo ${versionList[3]}

Conclusion

Switching your scripts from bash to zsh requires a bit more work than merely switching out the shebang. However, since /bin/bash will still be present in Catalina, you do not have to move all scripts immediately.

Moving to sh instead of zsh can be safer choice, especially for package installation scripts.

In zsh, there always seems to be some option to disable or enable a particular behavior.

This concludes my series on switching to zsh on macOS. I hope you found it helpful.

After having worked with zsh for a few weeks, I already find some of its features indispensable. I am looking forward to discovering and using more features over time. When I do, I will certainly share them here.

Weekly News Summary for Admins — 2019-08-16

Longer summary this week to catch up on the last three weeks of traveling!

The last post celebrating the 100th email resulted in quite a few new subscribers. Welcome!

I posted my speaking and training class schedule for the next few months. It’d be great to meet you somewhere!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • Denise Yu: “I illustrated my three favourite binaries”
  • Pepijn Bruienne: “To my surprise we recently updated the Authorization Plugin documentation. Surprise because it’s not seen a lot of change in recent times, but you get TWO new callbacks, RemoveHintValue and RemoveContextValue!”
  • Mr. Macintosh: “2019–004 for High Sierra 10.13, Sierra 10.12 and BridgeOS have been re released! I will update the main article and reply to this post when I have the new build numbers.”
  • Rosyna Keller: “The updated Hardened Runtime docs are out! The overview includes more information on how to enable it in Xcode and explains that the hardened runtime is designed to stop certain classes of exploits.” (Thread)
  • Phil Stokes: “Thanks to @SentinelOne @patrickwardle, my free ebook of How To Reverse macOS Malware is now available. Learn to set up a safe Mac test lab, find malware samples, static + dynamic analysis. eBook is free; tools you need are free.”
  • Ronald Oussoren: “The version 6 branch of PyObjC has finally caught up with the current beta: tests pass in macOS 10.15 beta 5 with Python 3.8b3, including bindings for all new frameworks.”
  • Edward Marczak: “Support local meetups! Particularly in tech: we need more discussion between disparate teams. There’s undoubtedly a meetup near you that matches your style. Support can mean presenting, but also just showing up. Without you, these gatherings go away. With you, they get stronger.”

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Watch

To Listen

Just for Fun

  • Timo Elliott: “How to evaluate your existing enterprise infrastructure…” (click for cartoon)
  • Ed Marczak: “We have a long way to go in exploring this medium. ” (read linked thread)

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Autumn 2019 Schedule

As I am emerging from vacation, I made an overview of my agenda for the next few months and realized there are quite a few public events. It might be useful to share my public agenda. If you are coming to any of these events, I’d be really happy to meet you!

Sep 6: Moving to zsh Class

Half-day class covering zsh on macOS. We will cover:

  • why Apple is switching the default shell
  • how this affects you and your users
  • how to configure zsh to increase your Terminal productivity
  • how to transfer your bash configuration
  • scripting zsh

The class will be held in the Pro Warehouse training room in Amsterdam on Sep 6, 10:00–14:00. You can sign up at the Pro warehouse page.

You can combine this training with the next event.

Sep 6: Dutch MacAdmins Meeting/Pro Academy Opening

We are celebrating the opening of the Pro Academy by hosting the Dutch MacAdmins meeting. Participation is free. Among many other presentations I will show how (and why) to build a command line tool in Swift.

Presentations will be (mostly) in English, so the event is suitable for international guests. Participation is free and you can sign up on Eventbrite.

Oct 1–4: MacSysAdmin Gothenburg

I am really excited about presenting at MacSysAdmin again. My topic is “Moving to zsh,” where I will discuss the changes (not just the changed shell) in Catalina and what they mean for MacAdmins.

There are still a few tickets available, but this conference usually fills up. So don’t delay and go register if you haven’t done so yet!

Oct 30–31: Scripting macOS Class

This is our ‘entry to scripting’ class, where we teach the basics of shell scripting. This will be the second time we are teaching this class and it will be updated for Catalina. Most of the examples used to teach scripting come from real administrator workflows.

This is a two day class held in the Pro Warehouse training room in Amsterdam on October 30 and 31. You can sign up at the Pro warehouse page.

Nov 12–14: JamfNation User Conference

I am not going to JNUC myself, but my co-worker and director Mischa van der Bent is. He will be presenting on “Offboarding in a Modern Deployment Workflow”, a topic which will include, among many other things, our Erase&Install application.

Book Updates

I am working on updates with Catalina content for all three books. They should be published around the Catalina release date, maybe a little later.

As always, if you have already purchased one of my book, the update will be free in the Apple Books application on your iOS device or Mac. There is no need to wait for the update if you are interested now. Purchase now and get the update when it is ready!

I am also working on new books. I had been working on some interesting content, but the experiences from the scripting classes and the Catalina announcements have made me take a step back and re-evaluate the plans and progress so far. Re-writing will be necessary before I am happy enough to publish.

From the topics in all the events above, you should be able to deduce at least one of the topics, though.

Weekly News Summary for Admins — 2019-07-26

Another Update Week. Apple released iOS 12.4 and macOS 10.14.6. There were also security updates (2019–004) for Sierra and High Sierra, which were then removed because apparently they coould cause kernel panics on wake.

Also interesting: Apple has provided updates for some older iOS hardware. iOS 9.3.6 and iOS 10.3.4 will make sure the “GPS”GPS, date, and time continue to work properly.” Some of the affected devices were last sold in 2012 or 2013, so that is quite an effort by Apple to support older hardware.

Note: Due to work travel and vacation time, there will be no newsletter for the next three weeks. I will return in late August with a long news summary. Hope you get to enjoy your summer as well!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

100

This is the 100th email newsletter!

The news summary actually started a bit earlier as a weekly post on the weblog. I added the email newsletter later. This is the 111th or, according to Tolkien, eleventy-first news summary, but I’ll take either cause to celebrate. Also, I missed celebrating the 100th summary blog post, the 500th and 600th subscriber, and the two-year anniversary (both of the summary and the email newsletter). At least I didn’t miss this milestone.

Back in 2017, when I started the news summary, we were all working on Sierra and iOS 10, UAMDM and APFS were still in the future, the MacAdmins podcast was on episode 31, the infamous “Mac Pro press meeting” was recent news, and the MacAdmins Slack was just about to reach 10K users.

An average summary has 38 entries, so I have sent out close to 4000 links, posts, and articles.

I want to thank all you readers for, well, reading the summary, whether it is on the website or in the email. Also, for the many kind words I have received. Sometimes posting on the web feels like shouting into a void and you never know if any one is listening. Any feedback is great. Positive feedback is just wonderful.

I am not trying to solicit praise, but want to encourage you all to send a note to one of the many MacAdmin authors whose links appear in this newsletter.

Also, thanks to all of you who have recommended the news summary to someone else and helped spread the word around. Please keep doing that!

Most of all I want to thank, again, all the MacAdmins out there who share their experience and knowledge with all of us in the Slack, on Twitter and other social media, in open source projects, at conference workshops and presentations, and, most of all, with blog posts. The MacAdmins community is something special. It demonstrates a unique culture of openness, sharing and “being in this together.” I am so proud and honored to be a small part of it.

On to the next hundred!

#! On Scripting OS X

If I ever announce an eight-part series on my website again, somebody please intervene. The next and last post will show some issues you may encounter when you move your scripts from bash to zsh and how to solve them.

Even when this series is (finally) complete. I am sure there will be more articles about zsh over the next months, there is still a lot to discover and learn.

I will be giving a half-day ‘Moving to zsh’ class in Amsterdam on September 6. We will cover the ‘why’ and ‘how’ of zsh and show lots of practical examples how using zsh will improve your Terminal productivity. More info and sign-up on the Pro Warehouse webpage.

📰News and Opinion

🏜macOS 10.14.6 Mojave and iOS 12.4 Updates

🏝macOS 10.15 Catalina and iOS 13 beta

🐦MacAdmins on Twitter

  • John C. Welch: “Good job by the MS Edge for Mac folks on making it a part of the MS Autoupdater mechanism so quickly. Now if the Teams folks could get on board…”
  • Patrick Fergus: “.@AppleSupport HT210216 mentions using wired data transfers for migrating between two Lightning iOS devices. What wired migration cabling setups are supported with USB-C+Lightning and USB-C+USB-C device combinations?” (HT210216)
  • Eric Boyd: “For those of you testing all the Apple things, this is an important resource: Profiles and Logs – Bug Reporting – Apple Developer

## 🐞Bugs and Security

🔨Support and HowTos

🤖Scripting and Automation

🍏Apple Support

♻️Updates and Releases

📺To Watch

🎧To Listen

## 🎈Just for Fun

📚 Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh – part 7: Miscellanea

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

We have covered the general aspects of configuring your zsh environment and enabling some of its features to make your work more productive. However, there are zsh features that didn’t quite fit in earlier posts, but also don’t warrant a post of their own. So I am gathering them here.

multiIO

Terminal commands can take input from a file or a previous command (stdin) and have two different outputs: stdout and stderr. In bash you can redirect each of these to a single other destination.

For example, you can redirect the output of a command to a file:

% system_profiler SPHardwareDataType >hardwareinfo.txt

In zsh you can redirect to (and from) multiple sources. In the simplest form you can write the output to two files:

% system_profiler SPHardwareDataType >hardwareinfo.txt >computerinfo.txt

This is of course not a very realistic case. Since the pipe | is a form of redirection, you can combine output to a file with a pipe:

% system_profiler SPHardwareDataType >hardwareprofile.txt | cat

Instead of piping to cat, you can also redirect to stdout (or &1) as well as to a file:

% system_profiler SPHardwareDataType >&1 >hardwareprofile.txt

Note that the order of doing this is important. The construct >file.txt >&1 would redirect the output to file.txt and then redirect the output again to where stdout or 1 is going, so it would be redundant.

When combined with pipes and other commands multiIO can become very useful:

% system_profiler SPHardwareDataType >hardwareprofile.txt | awk '/Serial Number/ { print $4 }' >&1 >serialnumber.txt

You can use multiIO for input as well:

% sort </usr/share/calendar/calendar.freebsd </usr/share/calendar/calendar.computer

And while this not directly related, but somewhat close, in zsh, this

% <hardwareinfo.txt

is equivalent to more hardwareinfo.txt.

Recursive Globbing with **

You can use the ** to denote an arbitrary string that can span multiple directories in a path.

For example:

% echo Library/Preferences/**/com.apple.screensaver.*plist
Library/Preferences/ByHost/com.apple.screensaver.BBCCDDEE-AABB-CCDD-ABCD-00AABBCCDDEE.plist Library/Preferences/com.apple.screensaver.plist

In this case the ** matches nothing as well as /ByHost/.

Note: when used on large folder structures this glob can take a while. So use with care.

Connected array Variables

We already encountered the fpath variable in earlier posts. You can see its contents with the echo command:

% echo $fpath
/Users/armin/Projects/mac-zsh-completions/completions/ /Users/armin/Projects/dotfiles/zshfunctions /usr/local/share/zsh/site-functions /usr/share/zsh/site-functions /usr/share/zsh/5.3/functions

Interestingly enough, zsh also has an FPATH variable, which is a colon-separated list of directories:

% echo $FPATH
/Users/armin/Projects/mac-zsh-completions/completions/:/Users/armin/Projects/dotfiles/zshfunctions:/usr/local/share/zsh/site-functions:/usr/share/zsh/site-functions:/usr/share/zsh/5.3/functions

Since the fpath variable is an array, I only changed the fpath variable in my zshrc.I never set or changed the FPATH, yet it reflects the changes made to the fpath variable.

When you see the type of both variables, you get an idea that something is going on:

% echo ${(t)fpath}
array-special
% echo ${(t)FPATH}
scalar-special

The fpath and FPATH are connected in zsh. Changes to one affect the other. This allows use of more flexible and powerful array operations through the fpath ‘aspect’ of the value, but also provides compatibility to tools that expect the traditional colon-separated format in FPATH.

You will not be surprised to hear that zsh uses the same ‘magic’ with the PATH variable and its array counterpart path.

This means that you can continue to use path_helper to get your PATH from the files in /etc/paths and /etc/paths.d. (Well, you don’t have to, because on macOS this is done for all users in /etc/zprofile.) But then you can manipulate the path variable with array functions, like:

path+=~/bin

You get the useful aspects of both syntaxes.

Suffix Aliases

I learnt this one after writing the aliases part.

Suffix aliases take effect on the last part of a path, so usually the file extension. A suffix alias will assign a command to use when you just type a file path in the command line.

For example, you can a suffix alias for the txt file extension:

alias -s txt="open -t"

When you then type a path ending with .txt and no command, zsh will execute open -t /path/to/file.txt.

The open -t command opens a file in the default application set for the txt file extension in Finder. You probably want to set the suffix alias to bbedit or atom or something like that rather than open -t.

You can use other command line tools for the suffix alias:

alias -s log="tail -f"

Then, typing /var/log/install.log will show the last lines of that file and update the output when the file changes. If you prefer the graphical user interface, you can use the open -a command to assign suffix aliases to applications:

alias -s log="open -a Console"

You can even create a suffix alias using a different alias:

alias pacifist="open -a Pacifist"
alias -s pkg=pacifist

Together with the AutoCD option, this can improve your application-shell interactions a lot.

Bindkey for History Search

Most of the keyboard shortcuts in zsh work the same way as they do in bash. I have found one change that has proven quite useful:

^[[A' up-line-or-search # up arrow bindkey

^[[B' down-line-or-search # down arrow 

These two commands will change the behavior of the up and down arrow keys from just switching to the previous command, to searching. This means that when you start typing a command and then hit the up key, rather than just replacing what you already typed with the previous command, the shell will instead search for the latest command in the history starting with what you already typed.

There are many commands or ‘widgets’ you can assign to keystrokes with the bindkey command. You can find a list of default ‘widgets’ in the documentation.

Conclusion

This concludes the part of the series about configuring zsh. When I set out I wanted to recreate the environment I had built in bash. Along the way I found a few features in zsh that seemed worth adding to my toolkit.

After nearly two months of working in zsh, there are already some features I would miss terribly when switching back to bash or a plain, unconfigured zsh. Most important is the powerful tab-completion. But features like AutoCD, MultiIO, and flexible aliases, are useful tools as well.

The dynamic loading of functions from files in the fpath was initially confusing, but it allows configurations and functions to be split out into their own, which simplifies “modularizing” and sharing.

In the next (and last) post, I will cover the changes when scripting with zsh vs bash.

Weekly News Summary for Admins — 2019-07-19

Still a lot of fallout from the Zoom invulnerability. It took Apple three updates to MRT (so far) to eradicate all the differently branded varieties of the Zoom client web server. Makes me wonder how many Macs there were or are with multiple of these clients installed.

We also got new betas for 10.14.5 and 10.15 and the respective iOS versions. Some of the worst data-destroying bugs seem to be fixed or at least mitigated but I am still not comfortable moving my production devices to the betas. That’s what test devices are there for. I am getting quite excited about some of the features I have seen, both for end-users and administrators.

One of those features is that zsh will become the default shell for macOS. If you want to know what that means, how to transfer your shell configuration, workflows, and scripts from bash to zsh, and increase your Terminal productivity, we are doing a half-day training in Amsterdam on September 6. You can get more details and sign up on our webpage!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

Many,many thanks to Erik for his contributions to the Mac Admin community and I wish him all the best for the future!

MacAdmins on Twitter

  • Timo Perfitt: “So it begins. We just signed up to be an MDM vendor. MDS DEP deployment coming soon.”

Bugs and Security

macOS 10.15 Catalina and iOS 13

Support and HowTos

Scripting and Automation

Updates and Releases

To Watch

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Moving to zsh, part 6 – Customizing the zsh Prompt

Apple has announced that in macOS 10.15 Catalina the default shell will be zsh.

In this series, I will document my experiences moving bash settings, configurations, and scripts over to zsh.

This series has grown into a book: reworked and expanded with more detail and topics. Like my other books, I plan to update and add to it after release as well, keeping it relevant and useful. You can order it on the Apple Books Store now.

As I have mentioned in the earlier posts, I am aware that there are many solutions out there that give you a pre-configured ‘shortcut’ into lots of zsh goodness. But I am interested in learning this the ‘hard way’ without shortcuts. Call me old-fashioned. (“Uphill! In the snow! Both ways!”)

The default bash prompt on macOS is quite elaborate. It shows the username, the hostname, and the current directory.

Calypso:~ armin$

On the other hand, the default bash prompt doesn’t show the previous command’s exit code, a piece of information I find very useful. I have written before how I re-configured my bash prompt to have the information I want:

Of course, I wanted to recreate the same experience in zsh.

Minimal Zsh Prompt

The only (visual) difference to my bash prompt is the % instead of the $.

Note: creating a file ~/.hushlogin will suppress the status message at the start of each Terminal session in zsh as well as in bash (or any other shell).

Basic Prompt Configuration

The basic zsh prompt configuration works similar to bash, even though it uses a different syntax. The different placeholders are described in detail in the zsh manual.

zsh uses the same shell variable PS1 to store the default prompt. However, the variable names PROMPT and prompt are synonyms for PS1 and you will see either of those three being used in various examples. I am going to use PROMPT.

The default prompt in zsh is %m%#. The %m shows the first element of the hostname, the %# shows a # when the current prompt has super-user privileges (e.g. after a sudo -s) and otherwise the % symbol (the default zsh prompt symbol).

The zsh default prompt is far shorter than the bash default, but even less useful. Since I work on the local system most of the time, the hostname bears no useful information, and repeating it every line is superfluous.

Note: you can argue that the hostname in the prompt is useful when you frequently have multiple terminal windows open to different hosts. This is true, but then the prompt is defined by the remote shell and its configuration files on the remote host. In your configuration file, you can test if the SSH_CLIENT variable is set and show a different prompt for remote sessions. There are more ways of showing the host in remote shell sessions, for example in the Terminal window title bar or with different window background colors.

In our first iteration, I want to show the current working directory instead of the hostname. When you look through the list of prompt placeholders in the zsh documentation, you find %d, %/, and %~. The first two do exactly the same. The last substitution will display a path that starts with the user’s home directory with the ~, so it will shorten /Users/armin/Projects/ to ~/Projects.

Note: in the end you want to set your PROMPT variable in the .zshrc file, so it will take effect in all your zsh sessions. For testing, however, you can just change the PROMPT variable in the interactive shell. This will give you immediate feedback, how your current setup works.

% PROMPT='%/ %# '
/Users/armin/Projects/dotfiles/zshfunctions % 

% PROMPT='%~ %# '
~/Projects/dotfiles/zshfunctions % 

Note the trailing space in the prompt string, to separate the final % or # from the command entry.

I prefer the shorter output of the %~ option, but it can still be quite long, depending on your working directory. zsh has a trick for this: when you insert a number n between the % and the ~, then only the last n elements of the path will be shown:

% PROMPT='%2~ %# '
dotfiles/zshfunctions %                       

When you do %1~ it will show only the name of the working directory or ~ if it is the home directory. (This also works with %/, e.g. %2/.)

Adding Color

Adding a bit of color or shades of gray to the prompt can make it more readable. In bash you need cryptic escape codes to switch the colors. zsh provides an easier way. To turn the directory in the path blue, you can use:

PROMPT='%F{blue}%1~%f %# '

The F stands for ‘Foreground color.’ zsh understands the colors black, red, green, yellow, blue, magenta, cyan and white. %F or %f resets to the default text color. Furthermore, Terminal.app represents itself as a 256-color terminal to the shell. You can verify this with

% echo $TERM
xterm-256color

You can access the 256 color pallet with %F{0} through %F{255}. There are tables showing which number maps to which color:

So, since I want a dark gray for my current working dir in my prompt, I chose 240, I also set it to bold with the %B code:

PROMPT='%B%F{240}%1~%f%b %# '

You can find a detailed list of the codes for visual effects in the documentation.

Dynamic Prompt

I wrote an entire post on how to get bash to show the color-coded exit code of the last command. As it turns out, this is much easier in zsh.

One of the prompt codes provides a ‘ternary conditional,’ which means it will show one of two expressions, depending on a condition. There are several conditions you can use. Once again the details can be found in the documentation.

There is one condition for the previous commands exit code:

%(?.<success expression>.<failure expression>)

This expression will use the <success expression> when the previous command exited successfully (exit code zero) and <failure expression> when the previous command failed (non-zero exit code). So it is quite easy to build an conditional prompt:

% PROMPT='%(?.√.?%?) %1~ %# ' 
√ ~ % false
?1 ~ % 

You can get the character with option-V on the US or international macOS keyboard layout. The last part of the ternary ?%? looks confusing. The first ? will print a literal question mark, and the second part %? will be replaced with previous command’s exit code.

You can add colors in the ternary expression as well:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# ' 

Another interesting conditional code is ! which returns whether the shell is privileged (i.e. running as root) or not. This allows us to change the default prompt symbol from % to something else, while maintaining the warning functionality when running as root:

% PROMPT='%1~ %(!.#.>) ' 
~ > sudo -s
~ # exit
~ > 

Complete Prompt

Here is the complete prompt we assembled, with all the parts explained:

PROMPT='%(?.%F{green}√.%F{red}?%?)%f %B%F{240}%1~%f%b %# '
%(?.√.?%?) if return code ? is 0, show , else show ?%?
%? exit code of previous command
%1~ current working dir, shortening home to ~, show only last 1 element
%# # with root privileges, % otherwise
%B %b start/stop bold
%F{...} text (foreground) color, see table
%f reset to default textcolor

Right Sided Prompt

zsh also offers a right sided prompt. It uses the same placeholders as the ‘normal’ prompt. Use the RPROMPT variable to set the right side prompt:

% RPROMPT='%*'
√ zshfunctions %                    11:02:55

zsh will automatically hide the right prompt when the cursor reaches it when typing a long command. You can use all the other substitutions from the left side prompt, including colors and other visual markers in the right side prompt.

Git Integration

zsh includes some basic integration for version control systems. Once again there is a voluminous, but hard to understand description of it in the documentation.

I found a better, more specific example in the ‘Pro git’ documentation. This example will show the current branch on the right side prompt.

I have changed the example to include the repo name and the branch, and to change the color.

autoload -Uz vcs_info
precmd_vcs_info() { vcs_info }
precmd_functions+=( precmd_vcs_info )
setopt prompt_subst
RPROMPT=\$vcs_info_msg_0_
zstyle ':vcs_info:git:*' formats '%F{240}(%b)%r%f'
zstyle ':vcs_info:*' enable git

In this case %b and %r are placeholders for the VCS (version control system) system for the branch and the repository name.

There are git prompt solutions other than the built-in module, which deliver more information. There is a script in the git repository, and many of the larger zsh theme projects, such as ‘oh-my-zsh’ and ‘prezto’ have all kinds of git status widgets or modules or themes or what ever they call them.

Summary

You can spend (or waste) a lot of time on fine-tuning your prompt. Whether these modifications really improve your productivity is a matter of opinion.

In the next post, we will cover some miscellaneous odds and ends that haven’t yet really fit into any of preceding posts.

Weekly News Summary for Admins — 2019-07-12

Apple released an unexpected update to the entry-level MacBook Air and MacBook Pro 13“ this week. What the new release replaces is interesting, as well: the non-retina entry level MacBook Air, the no-Touchbar 13” MacBook Pro, and the 12″ MacBook are all discontinued.

The iMac and the (cylinder) Mac Pro are the only remaining Mac models without a T2 chip. All current Mac models except the iMac Pro and the Mac Pro now require macOS 10.14 Mojave as their minimum OS.

We also got an interesting vulnerability story with the popular videoconferencing software Zoom. The reaction from Zoom was slow, they did not address the issue until the researcher published the problem, but Apple reacted much faster, adding the problematic local web server, that Zoom installed to bypass some of the system’s user security to macOS’s Malware Removal Tool’s list of, well, malware within 24 hours.

Finally, I am going to give a half-day class based on the ‘Moving to zsh’ article series at our offices in Amsterdam on September 6. If you are wondering why Apple is changing the default shell, how it will affect you, and how you can increase your Terminal productivity with zsh, then sign up here!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

macOS Catalina and iOS 13

MacAdmins on Twitter

  • Sisyphus-J: “MacOS/iOS Entitlement database now updated for MacOS 15 and iOS 13 beta 3. with over 150 more daemons and 500 more entitlements.. including such fine gems as com.apple.rootless.kext-more-securer-management http://newosxbook.com/ent.jl”
  • Mustafa Hussain: “In macOS, when you drop a Calendar event into the Terminal, the Terminal adopts the event’s color. As it should.… ”
  • Steve Troughton-Smith: “Disasters like Zoom are why the Mac gets progressively more locked-down, and why commandline tools and daemons can’t be exempt from signing & security. By default, we should trust no software, and neither should the hardware it runs on…”
  • Alex Wild: “Apparently sailors used to use a log, tied to a measured rope & thrown overboard, to periodically measure the speed of a boat. They’d then write the data in a ”log-book“ Which is why we now use the word ”log“ to record data. Things I learned. https://www.etymonline.com/word/log”
  • Isaac T.: “Confirmed with Apple: sharing with SMB is broken in macOS Mojave 10.14.5. The problem: ”smbd“ deadlocks with multiple users connected to a share. Engineering knows and is working on it. Time to resolution: unknown. Here are the options: 1/” (thread)
  • Rich Mogull: ““Shadow IT” is merely how the fearful describe the reduction of friction and democratization of technology to better enable agile business.”

PSU MacAdmins Talks, Slides and Resources

Bugs and Security

Support and HowTos

To Watch

To Listen

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Build a shellcheck installer for macOS

Shellcheck is an invaluable tool for anyone who writes, well, shell scripts. The tool will point out common and less common errors, such as forgetting to quote a variable substitution, not putting that space before the closing ]], or not testing if a variable substitution used with rm -Rf might be empty.

Update 2019-11-20: The shellcheck project now has pre-compiled binaries available and I have updated instructions here.

You can use the tool’s website to check your code, but it is unwieldy, not to mention you or your organization might have concerns uploading all your code to some website, useful as it may be. You can install a shellcheck command line tool, but the instructions available for macOS will only work with Homebrew.

If you already have Homebrew installed, or don’t mind installing it and only need to install shellcheck on a single Mac, then you can use brew install shellcheck and skip ahead to the end, where I share how to use the shellcheck binary with BBEdit.

Relevant for MacAdmins

Homebrew may be a fine tool for individual users, but it is near-impossible to centrally manage a deployment of brew let alone a piece software installed with brew.

My attempts to build the shellcheck binary without homebrew lead to several dead-ends. Shellcheck is written in Haskell Cabal, which can be easily installed with… brew. There is also an official ghcup script, which is the ‘recommended way to install Haskell’, but this script fails on macOS because it requires a tool named xz. This is a (de-)compression tool, which is part of xz-utils, whose website does not explain how to install on macOS, but can be easily installed with… guess… brew

Even without brew, you have to install a number of tools that you probably don’t want hanging around on your production machine. I am sure cabal is a fine environment to build other tools as well, but I don’t need it on my production system.

I would recommend running these steps on a virtual machine. That will simplify ‘cleanup’ as well as the ‘start over’ process, should you need it.

How to build the shellcheck binary for macOS

Commands shown are correct at the time this article was written and may change in the future. Links to original sources are given, so you can check for changes.

Developer Tools

  • install Xcode or command line developer tools (CLI dev tools have the advantage of not being >6GB in size)
$ xcode-select --install

XZ Utils

Note: the macOS Archive Utility can deal with .xz archives just fine. There is really no need on macOS to have the xz-utils installed, but the GHC Cabal installer requires them.

$ cd ~/Downloads/xz-5.2.4
$ ./configure
$ sudo make install
  • check to see of xz was installed correctly
$ xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.4

Install Pandoc

We will require pandoc to build the man page for shellcheck.

Haskell Cabal

$ curl https://get-ghcup.haskell.org -sSf | sh
  • make sure the cabal environment is setup
$ source "$HOME/.ghcup/env"
$ cabal update

Shellcheck

$ git clone "https://github.com/koalaman/shellcheck.git"
  • build the shellcheck binary (this will take a while)
$ cd shellcheck
$ cabal install
  • this will build and install the shellcheck binary into ~/.cabal/bin, you can test if this worked with
$ ~/.cabal/bin/shellcheck --version
ShellCheck - shell script analysis tool
version: 0.6.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net

man page

  • there is also a man page in the repo, written in markdown format. The ReadMe says you can convert it to man format with pandoc:
$ pandoc -s -f markdown-smart -t man shellcheck.1.md -o shellcheck.1

Pkg it up

  • create a payload directory for the binary and man file and move the files there:
$ mkdir -p payload/local/bin
$ cp ~/.cabal/bin/shellcheck payload/local/bin/
$ mkdir -p payload/share/man/man1
$ cp shellcheck.1 payload/share/man/man1/
  • build the pkg installer from the payload with pkgbuild
$ pkgbuild --root payload --install-location /usr --identifier com.scriptingosx.pkg.shellcheck --version 0.6.0 shellcheck-0.6.0.pkg

Automation!

Of course, you can do this once and then forget about until the next update of shellcheck. While I wrote this post, I had to go through these steps more than once and got bored with it halfway through the second cycle. So, there is a script, which you can get at this Github repo.

Since this script will install xz-utils, cabal, and pandoc, I do not recommend to run this on your work machine. I use it with clean virtual machine. I suggest this workflow:

  • boot to clean vm
  • snapshot the vm
  • open Terminal
  • install developer command line tools: xcode-select --install
  • clone the script: git clone https://github.com/scriptingosx/BuildShellcheckPkg
  • run the script: cd BuildShellcheckPkg; ./buildShellCheckPkg.sh
  • confirm the prompts (once the cabal installation starts for good, the prompts are over)
  • wait…
  • wait…
  • start doing other work such as answering emails, MacAdmins Slack, Twitter,…
  • much later, remember there was something important, go to the vm, copy the brand new shellcheck pkg to your work machine
  • revert the vm to snapshot
  • buy ‘Packaging for Apple Administrators’ to show your gratitude and learn how to wield pkgbuild yourself

Using shellcheck

Using the shellcheck command line tool is very straightforward. Just pass one (or more) shell script files as arguments. There are some flags and options that you can read about in the --help option or the man page.

$ shellcheck serial.sh

In serial.sh line 11:
serial_number=$(serial) # stores output of function serial in variable
^-----------^ SC2034: serial_number appears unused. Verify use (or export if used externally).

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- serial_number appears unused. Ver...

Use shellcheck with BBEdit

My favored text editor – BBEdit – has a tool that will display the output of tools like shellcheck. This allows you to click on an error or warning and see and edit the code right there in a BBEdit window.

To get this, pipe the output of shellcheck into bbresults:

$ shellcheck -f gcc serial.sh | bbresults

I have created a function in my .bash_profile and .zshrc, so I don’t have to remember the details:

function bbshellcheck {
    shellcheck -f gcc "$@" | bbresults
}

If you do not know what the .bash_profile or functions are, then you can read about them here:

Note: you cannot use shellcheck to check zsh scripts. As of now it only works for sh or bash scripts. You can tell shellcheck to ignore the #!/bin/zsh shebang and interpret the script as using bash syntax, which should work in most cases, but will not always have good results.

$ shellcheck -s bash script.zsh