The fine people from MacAD.UK have started putting the sessions from this year’s conference online for every one to use.
The video of my presentation titled “The macOS School of Terminal Witchcraft and Wizardry” was uploaded earlier today. Go watch it!
You can find the slides and some notes on my session here.
Charles Edge is dominating the summary today with a list of posts on how best to replace the waning macOS Server services. Thank you!
We also got the WWDC announcement from Apple. (June 4–8 in San Jose, CA) If you applied to the ticket lottery (before March 22, 10am PDT): Good Luck!
If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)
I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)
If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!
No big news this week, which is a nice change.
The first iPhone SDK dropped ten years ago this week. Craig Hockenberry wrote a wonderful retrospective which is worth reading.
If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)
I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)
If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!
A strange thing happened last Friday (a few hours after the last news summary). Apple updated the kbase article HT208020 – the infamous one that killed imaging. Then the article was reverted back to the old version after a few hours. Presumably some news for 10.13.4 were released prematurely.
Mark Melaccio grabbed screen shots.
If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)
(I have re-included the links from last week.)
known_hosts among your admin peers – Robin LaurénI do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)
If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!
In my recent post I mentioned in passing, that you should be using double brackets [[…]] for tests in bash instead of single brackets.
This is the post where I explain why. I also talked about this briefly in my MacSysAdmin session: Scripting Bash
bashismDouble brackets were originally introduced in ksh and later adopted by bash and other shells. To use double brackets your shebang should be #!/bin/bash not #!/bin/sh.
Since sh on macOS is bash pretending to be sh, double brackets will still work with the wrong shebang, but then your script might break on other platforms where a different shell might be pretending to be sh. Consistent behavior across platforms is the main point why sh is still around, so don’t use double brackets in sh (or use bash to use double brackets).
I go into detail on why to use bash over sh in this post: On the Shebang
In shell scripts you usually use tests in if or while clauses. These are tedious to write in the interactive shell. The ‘and’ operator && will execute the following statement only if the preceding statement returns 0 (success). So you can use && to write simple if … then … clauses in a single line.
if [ -d Documents ]
then
echo "found docs"
fi
and
[ -d Documents ] && echo "found docs"
have the same effect. The second is much shorter, but as soon as the test or the command gets more complex you should revert to the longer syntax.
Alternatively, the ‘or’ operator || will only execute the following statement when the previous statement returns non-zero or fails:
[ -d Documents ] || echo "no docs"
is the same as
if [ ! -d Documents ]
then
echo "no docs"
fi
The single bracket [ is actually a command. It has the same functionality as the test command, except that the last argument needs to be the closing square bracket ]
$ [ -d Documents && echo "found docs"
-bash: [: missing `]'
~ $ [ -d Documents ] && echo "found docs"
found docs
$ test -d Documents && echo "found docs"
found docs
Note: in bash on macoS both test and [ are built-in commands, but as usual for built-in commands there are also executables /bin/test and /bin/[.
A single bracket test will fail when one of its arguments is empty and gets substituted to nothing:
$ a="abc"
$ b="xyz"
$ [ $a = $b ] || echo "unequal"
unequal
$ unset a
$ [ $a = $b ] || echo "unequal"
-bash: [: =: unary operator expected
unequal
You can prevent this error by quoting the variables (always a prudent solution).
$ [ "$a" = "$b" ] || echo "unequal"
unequal
Double brackets in bash are not a command but a part of the language syntax. This means they can react more tolerantly to ‘disappearing’ arguments:
$ [[ $a = $b ]] || echo "unequal"
unequal
You will also get an error if one of the arguments is substituted with a value with whitespace with single brackets, while double brackets can deal with this.
$ a="a"
$ b="a space"
$ [ $a = $b ] || echo "unequal"
-bash: [: too many arguments
unequal
$ [[ $a = $b ]] || echo "unequal"
unequal
Note: the = operator in sh and bash is for string comparison. To compare numerical values you need to use the -eq (equals), -ne (not equals), -gt (greater than), -ge (greater than or equal), -lt (less than), -le (less than or equal) operators. With double brackets you can also use two equals characters == for a more C like syntax. (or, better, use ((…)) syntax for arithmetic expressions)
Also, when using the = to assign variables, you cannot have spaces before and after the =, while the spaces are required for the comparison operator (both with single and double brackets):
a="a" # no spaces
b="b" # no spaces
[ "$a" = "$b" ] # spaces!
[[ $a = $b ]] # spaces!
Since the single bracket is a command, many characters it uses for its arguments need to be escaped to work properly:
$ [ ( "$a" = "$b" ) -o ( "$a" = "$c" ) ]
-bash: syntax error near unexpected token `"$a"'
$ [ \( "$a" = "$b" \) -o \( "$a" = "$c" \) ]
You could alternatively split this example into two tests: [ "$a" = "$b" ] || [ "$a" = "$c" ].
Double brackets interpret these characters properly. You can also use the (again more C like) && and || operators instead of -a and -o.
[[ ( $a = $b ) || ( $a = $c ) ]]
In general, you can work around most of the issues with single bracket syntax, but the double bracket syntax is more straight forward and hence more legible and easier to type.
Aside from the cleaner syntax, there are a few ‘bonus’ features you gain with double brackets.
With double brackets you can compare to * and ? wildcards, and bracket globbing […]:
$ a="Documents"
$ [[ $a = D* ]] && echo match
match
$ a=hat
$ [[ $a = ?at ]] && echo match
match
$ [[ $a = [chrp]at ]] && echo match
match
You can also use < and > to compare strings lexicographically:
$ a=cat
$ b=hat
$ [[ $a < $b ]] && echo sorted
sorted
And you get an operator =~ for regular expressions:
$ a=cat
$ b="the cat in the hat"
$ [[ $a =~ ^.at ]] && echo match
match
$ [[ $b =~ ^.at ]] && echo match
Note that you should not quote the globbing patterns or the regex pattern.
bash for shell scripting on macOSLast week, fellow MacAdmin Kyle Crawford has discovered that on macOS High Sierra the defaults will delete a property list file with invalid plist/XML syntax, even when you just attempt to read data. Erik Holtham has a more detailed OpenRadar bug.
Patrik Wardle has found the relevant code, which deletes the file when it doesn’t validate.
This is bad. Thanks to all involved for finding, documenting and sharing this.
This is new behavior in High Sierra. I am not yet sure which version of High Sierra this new behavior was introduced. The behavior makes sense in the context of an application attempting to read a settings file, but the defaults tool deleting arbitrary files is, of course, dangerous.
Update: This behavior has been fixed in 10.13.4. However, it is still good practice to avoid
defaultsfor anything other than actual preferences files.
As usual, don’t panic. This will only affect Macs running High Sierra and corrupt or broken files. However, if you have a script that accidently points the defaults command at a different file, it will delete that. So you have to use it with care.
It is probably a good practice to verify a file before you attempt to modify it with the defaults command in a script with the plutil command:
if ! plutil -lint path/to/file.plist; then
echo "broken plist"
exit 1
else
defaults path/to/file …
fi
defaultsAlternatively, you can and should use plutil or PlistBuddy to read and modify property list files.
Learn more about
plutil,PlistBuddyand other tools to read and write property lists in my book: “Property Lists, Preferences and Profiles for Apple Administrators”
plutil is unfortunately not really useful to read a single value from a property list file. The extract verb will show any value as its own plist. The plutil command is useful to edit existing plist files. (Read details on the plutil command.)
PlistBuddy, however is very useful for both reading an writing values to a property list file:
$ /usr/libexec/PlistBuddy berry.plist -c "print size"
$ /usr/libexec/PlistBuddy berry.plist -c "set size enormous"
PlistBuddy has the additional advantage of allowing to add or edit values nested deep in dict or array structures.
You can get more information on PlistBuddy in its man page or my book.
The interactive mode of PlistBuddy is also very useful.
defaults command is dead?No.
Apple has been warning us to not use defaults for generic property list file editing and parsing for quite a while in the defaults command’s man page:
WARNING: The defaults command will be changed in an upcoming major release to only operate on preferences domains. General plist manipulation utilities will be folded into a different command-line program.
As this warning states, the defaults tool reads and writes data to plist files through macOS’s preferences system. This has the advantage that the tool gets (and changes) the current value whether it is cached in memory or not. When an application is listening for notifications that a preference has changed (not many do) then it will be notified.
Files for preference domains are usually stored in /Library/Preferences/, ~/Library/Preferences or their ByHost subfolders. Sandboxed applications will have their preference plist files in their container.
There, however, many other files that are property list files which are not part of the user defaults system: launchd files, configuration profiles and AutoPkg` recipes to name just a few.
Mac Admins commonly use the defaults tool, despite Apple’s warning, to create, read and edit generic plist files. As mentioned above, plutil, PlistBuddy or direct manipulation through Obj-C, Python or Swift, are better choices for generic plist files.
You can learn about all these options in my book: “Property Lists, Preferences and Profiles for Apple Administrators”
This week I was travelling to London for MacAD.UK. So maybe I missed some interesting links. If I did, please let me know and I will include the link next week.
MacAD.UK was great. I had a great time meeting many people, some I already knew and some I had only met virtually so far. Great presentations, hope to see you all again next year!
If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)
Let me know if I missed one!
defaults bug in tweets:
defaults read command DELETES your plist if invalid“-@kcrawford Here’s the bug/reason why: -[CFPDSource copyPropertyListWithoutDrainingPendingChangesValidatingPlist:] in CoreFoundation calls unlink() if plist validation fails🐛🤣 ”I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)
If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!
Thanks to all who came to see my “macOS School of Terminal Witchcraft and Wizardry” presentation at MacAD.UK this morning. I am really enjoying the conference and all the other talks.
I have put up a page with the slides and links to articles with more information and explanations here.
There was some news this week about Apple refocusing the next major OS updates on bug fixing. Will this be (High) Sierra’s Snow Leopard?
Getting ready for my MacAD.UK presentation next week. Looking forward to meeting some of you there!
If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)
I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)
If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!
A discussion that comes up frequently on MacAdmin Slack and other admin discussions is:
Should commands in scripts have their full path hardcoded or not?
Or phrased slightly differently, should you use /bin/echo or just echo in your admin scripts?
I talked about this briefly in my MacSysAdmin session: Scripting Bash
When you enter a command without a path, e.g. echo, the shell will use the PATH environment variable to look for the command. PATH is a colon separated list of directories:
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
The shell will look through these directories in order for the given command.
You can read more detail about the PATH and environment variables in these posts:
The example PATH above is the default on macOS on a clean installation. Yours will probably look different – mine certainly does:
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/munki:/Users/armin/bin
Third party applications and tools can and will modify your PATH. You yourself might want to change your PATH in your shell profile.
But on top of that, the PATH will be different in different contexts. For example, open the Script Editor application, make a new script document, enter do shell script "echo $PATH" and run the script by hitting the run/play button.
The small AppleScript we just built runs the shell command echo from the AppleScript context. The result is
/usr/bin:/bin:/usr/sbin:/sbin
Note how the PATH in this context is different from the default macOS PATH in Terminal and also different from your PATH.
I also built a simple payload-free installer package which only runs the echo "Installer PATH: $PATH" command. You will have to search through /var/log/install.log to get the output:
installd[nnn]: ./postinstall: Installer PATH: /bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec
Which is yet another, different PATH.
The PATH may be different in different contexts that your script may run in.
That means we cannot be consistently certain if a command will be or which command will be found in a script in different contexts. This is, obviously, bad.
Mac system adminstrative scripts which can run in unusual contexts, such as the Login Window, NetInstall, the Recovery system or over Target Disk Mode and usually run with root privileges. You really want to avoid any uncertainty.
There are two solutions to make your scripts reliable in these varying contexts:
PATH in the scriptBoth are valid and have upsides and downsides. They are not exclusive and can be both used in the same script.
Update: 2020-08-25 changed some of the sample commands.
You can avoid the unreliability of the PATH by not using it. You will have to give the full path to every command in your script. So instead of
#!/bin/sh
systemsetup -settimezone "Europe/Amsterdam"
you have to use:
#!/bin/sh
/usr/sbin/systemsetup -settimezone "Europe/Amsterdam"
When you do not know the path to a command you can use the which command to get it:
$ which systemsetup
/usr/sbin/systemsetup
Note: the
whichcommand evaluates the path to a command in the current shell environment, which as we have seen before, is probably different from the one the script will run in. As long as the resultingPATHstarts with one of the standard directories (/usr/bin,/bin,/usr/sbin, or/sbin) you should be fine. But if a differentPATHis returned you want to verify that the command is actually installed in all contexts the script will run in.
Using full paths for the commands works for MacAdmin scripts because Mac administrative scripts will all run on some version of macOS (or OS X or Mac OS X) which are very consistent in regard to where the commands are stored. When you write scripts that are supposed to run on widely different falvors of Unix or Linux, then the location of certain commands becomes less reliable.
The downside of hardcoding all the command paths is that you will have to memorize or look up many command paths. Also, the extra paths before the command make the script less legible, especially with chained (piped) commands.
If you want to save effort on typing and maintenance, you can set the PATH explicitly in your script. Since you cannot rely on the PATH having a useful value or even being set in all contexts, you should set the entire PATH.
This should be the first line after the shebang in a script:
#!/bin/sh
export PATH=/usr/bin:/bin:/usr/sbin:/sbin
systemsetup -settimezone "Europe/Amsterdam"
Note: any environment variable you set in a script is only valid in the context of that script and any sub-shells or processes this script calls. So it will not affect the
PATHin other contexts.
This has the added benefit of providing a consistent and well known PATH to child scripts in case they don’t set it themselves.
The downside of this is that even with a known PATH you cannot be entirely sure which tool will be called by the script. If something installed a modified copy of echo in /usr/bin it would be called instead of the expected /bin/bash.
However, on macOS the four standard locations (/usr/bin, /bin, /usr/sbin, /sbin, as well as the less standard /usr/libexec) are protected by System Integrity Protection (SIP) so we can assume those are ‘safe’ and locked down.
But notice that I do not include /usr/local/bin when I set the PATH for my scripts, even though it is part of the default macOS PATH. The PATH seen in the installer context does not include /usr/local/bin, either.
/usr/local/bin is a standard location where third party solutions can install their commands. It is convenient to have this directory in your interactive PATH under the assumption that when you install a tool, you want to use it easily.
However, this could create conflicts and inconsistent results for administrative scripts. For example, when you install bash version 4, it will commonly be installed as /usr/local/bin/bash, which (with the standard PATH) overrides the default /bin/bash version 3.
Since you chose to install bash v4, it is a good assumption that you would want the newer version over the older one, so this is a good setting for the interactive shell.
But this might break or change the behavior of administrative scripts, so it is safe practice to not incluse /usr/local/bin in the PATH for admin scripts.
When you use commands from other directories (like /usr/libexec/PlistBuddy, or third party tools like the Munki or Jamf tools) then it is your choice whether you want to use full path for these commands or (when you use the commands frequently in a script) add their directory to the PATH in your script:
E.g. for Munki
export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki
or Jamf
export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/jamf/bin
Since the third party folders are not protected by SIP, it is safer to append them at the end of the PATH, so they cannot override built-in commands.
Another solution that is frequently used for single commands with long paths is to put the entire path to the command in a variable. This keeps the script more readable.
For example:
#!/bin/sh
# use kickstart to enable full Remote Desktop access
# for more info, see: http://support.apple.com/kb/HT2370
kick="/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart"
#enable ARD access
$kick -configure -access -on -users remoteadmin,localadmin -privs -all
$kick -configure -allowAccessFor -specifiedUsers
$kick -activate
Note: that you need to quote the variable when the path to the command contains spaces or other special characters.
As a system administrator it is important to understand the many contexts and environments that a script might be run in.
Whether you choose to write all command paths or explictly set the PATH in the script is a matter of coding standards or personal preference.
You can even mix and match, i.e. set the PATH to the ‘default four’ and use command paths for other commands.
My personal preference is the solution where I have to memorize and type less, so I set the PATH in the script.
Either way you have to be aware of what you are doing and why you are doing it.