Installing and Using Command Line Tools

There are many command line tools and scripts you can download and install that are very useful for Mac Admins.

(Just a representative list, certainly not complete.)

Some of these tools provide installer packages that deploy the tool in the proper directory – usually /usr/local/bin so that you and other users can use it. (/usr/local/bin is in the macOS default PATH.)

However, many of these tools, such as munkipkg or my own quickpkg just come as a git project folder, with no or few instructions to get it set up. The assumption is, that when you use these tools you are familiar enough with the shell to make them work.

There are actually several approaches to getting these tools to actually work for you, most with different upsides and downsides. This post will talk about a few of them.

Getting the Tool

Before you can choose a method to run the tool, you need to get it. Many admins share their scripts and tools through a hosted service like Github. My quickpkg tool, for example, is a python script hosted as an open Github repository.When you follow that link you will see the main project page. The page has a menu area up top, a file list in the middle and below an area where an introduction to the project (the ReadMe file) is shown. It is worth to read the ReadMe in case they have special installation instructions.

Download the Release

Git is a version management tool that lets you track changes throughout the coding process. Github is one popular service to host these projects online. Contributors of a project have the option of marking steps of the project as a ‘release.’ Releases are considered a tested and stable stop in between less reliable developmental steps.

Releases will be shown in the project’s ‘releases’ page (link in the middle of the page, above the file list). (quickpkg releases page

On the releases page you will see a list of releases with the newest on top. At the very least each release will have a snapshot of the project’s code as a zip or tar.gz archive. Some projects provide other archives or installers such as dmg or pkg as well.

Download the Current Project

Some projects do not manage releases. (You will see ‘0 releases’ in the tool bar.) Then you can still download the most recent version of the project. There is a large green ‘Clone or download’ button on the right area above the project’s file list for this. When you click that button it will expand to show some more options.

‘Download ZIP’ will simply download an archive of the current state of project, much like the release download would.

When you download the archives, either through the releases page or from the ‘Download ZIP’ button, the resulting project folder will not be connected with the Github project any more. If you just want to use the current version, then that is fine and will serve you well. If you want an updated version in the future you will simply download the newer version and replace the tool you already have.

If you rather use git to download and manage the code, then you can do that here, too. However, that is a topic for another post.

Using the Tool

However you get the project you will now have a directory with the tool and any supporting files. You can already change directory to this folder in Terminal (drag the folder on to the Terminal icon to open a new Terminal already changed to it) and run the tool directly:

$ cd ~/Projects/quickpkg/
$ ./quickpkg
usage: quickpkg [-h] [--scripts SCRIPTS] [--preinstall PREINSTALL]
                [--postinstall POSTINSTALL]
                [--ownership {recommended,preserve,preserve-other}]
                [--output OUTPUT] [--clean] [--no-clean] [--relocatable]
                [--no-relocatable] [--sign SIGN] [--keychain KEYCHAIN]
                [--cert CERT] [-v] [--version]
                item_path
quickpkg: error: too few arguments
This will do for tools that you use rarely. But for tools that you want to use frequently typing the path to the tool is quite cumbersome.

Put it in the PATH

The PATH environment variable lists the directories where the shell looks for commands. You could add the project directory of the tool you just added to the PATH, but that would be tedious to manage.

An easier solution is to copy the tool to /usr/local/bin. This is the designated directory for custom commands. /usr/local/bin is also in the default macOS PATH.

However, copying the tool has some downsides. When the tool get’s updated you will have to copy the newer version, as well. Also some tools may require additional resources or libraries that reside in its project directory.

Instead of moving the tool, you can create a symbolic link to the tool in /usr/local/bin.

I keep the project folders of tools in ~/Projects so I use the command:

$ sudo ln -s ~/Projects/quickpkg/quickpkg /usr/local/bin
$ ls -al /usr/local/bin/quickpkg 
lrwxr-xr-x  1 root  wheel /usr/local/bin/quickpkg -> /Users/armin/Projects/quickpkg/quickpkg
Since symbolic links use paths, this has the advantage that when you download a newer version of the project to the same location, the link will point to the new version.

Putting links to a tool in /usr/local/bin has a few downsides (or upsides, depending on your perspective):

  • you need to have administrator privileges to change /usr/local/binlinks/tools you add to /usr/local/bin affect all users on that Mac

Set your own PATH

When you want to have the tools only affect your shell environment you need to do a bit more work.

First you need to choose a location where you tools or links should live. I have created a directory ~/bin for that purpose.

$ mkdir ~/bin
When you don’t want anyone else on the Mac to see what you are doing in that directory, you can remove everyone else’s access to it:
$ chmod 700 ~/bin
If you want you can also hide the directory in the Finder:
$ chflags hidden ~/bin
(Use the same command with nohidden to make Finder show it again.)

(To test the following properly, you need to delete the symbolic link we created earlier in /usr/local/bin. If that still exists the shell will use that, since it comes earlier in the PATH.)

You can create a symbolic link to the in ~/bin with

$ ln -s ~/Projects/quickpkg/quickpkg ~/bin
However, these still will not work, since we need to add the ~/bin directory to your personal PATH.

To do that you need to add this line to your ~/.bash_profile or ~/.bashrc:

export PATH=$PATH:~/bin
(Read more about how to create a bash profile here and here. This assumes you are using bash, the default on macOS. Other shells will have other locations where you can change environment variables.)

Then open a new Terminal window or type source ~/.bash_profile so that the new profile is loaded in the current window and try running the command.

Success!

Weekly News Summary for Admins — 2018-03-30

What a week!

This would have been an interesting with many amazing articles and posts by fellow MacAdmins on different topics. But then there was Apple’s Education event and (finally) the spring updates (10.13.4, 11.3 etc.) dropped yesterday.

10.13.4 brings (yet again) many changes for macOS management, read the summaries and posts carefully.

PSU MacAdmins Conference 2018 registration is open!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Happy Easter! (Hope you get a long Easter week-end to catch up with the reading!)

#! On Scripting OS X

📰News and Opinion

Education Event

Spring Updates

🐞Bugs and Security

🔨Support and HowTos

For macOS 10.13.4

🤖Scripting and Automation

🍏Apple Support

Spring Updates

Security

♻️Updates and Releases

📺To Watch

🎧To Listen

📚Support

I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Thoughts on Books

Apple had an event yesterday at a school. You may have heard.

If you are not working in education tech the news probably seemed underwhelming. Don’t worry, Apple will have more events and some of them will have great news for you.

I am not going to discuss the entire event. I don’t currently do much work in education IT and when I do it is for Macs, so I don’t feel very qualified to address most of what was shown yesterday.

However, there was one topic on which I was hoping for news: iBooks. However, as it is often the case with Apple, it was neither the news I was expecting nor hoping for.

In case you didn’t know: I am interested in iBooks because I have published two books written with iBooks Author.

iBooks was introduced with the iPad in 2010 and the Mac version followed in 2013 (with Mavericks). Along the way Apple also introduced iBooks Author in 2012 which allows building rich media, interactive digital books tailored for the iPad. (Though they also work on Mac and iPhone.)

Since then the iBooks app, iBooks Store and iBooks Author have received a few updates. Overall, Apple’s attention to the books part of their software and services can only be described as ‘lacking.’

There were some signs in the recent iOS beta releases that Apple seems to be preparing to change the iBooks app’s name to simply ‘Books’, and iBooks Author has found its niche in education, so this education focused event seemed like a place for an announcement. The hopes were up!

Pages 7 vs iBooks Author

As I said, the announcement on the book publishing front is underwhelming. The new Pages (version 7.0) has improved ePub3 fixed layout support and some new templates.

This is not a new feature. Pages has had ePub export for a long time. So far it has focussed on free-flowing layout. Last year, Pages 6.3 gained the ability to create and export fixed-layout ePub formats. Back then I was really interested to have a solution other than iBooks Author. However, the feature set of Pages for fixed layout is quite rudimentary compared to iBooks Author. I have not had time to really test the latest version of Pages yet, but I can already confirm that some really important features are still missing.

iBooks Author forces you to structure your book with chapters and sections, but also builds an interactive table of contents automatically. iBooks Author provides interactive widgets for images, movies, and even Keynote presentations that go fullscreen on tap. Also scrollable sidebars which I use in my books for scripts and source text that is longer than fits on a screen. I can also create links in text to other parts and objects in the book.

None of these features are present in Pages.

In iBooks Author you can send a preview to iBooks on a connected device. In Pages I have to export as ePub and find some way to get it to the target device.

But the most damning thing is, there is no way to import an iBooks Author document or template into Pages. I’d have to start over and rebuild the book in a new layout.

Future of iBooks Author

However, the situation is not all bad. Apple has also stated that iBooks Author is not going anywhere. For now I would presume we are in a transition phase, where the ‘old software’ is kept around for compatibility and certain features until the replacement can live up to expectations. Apple has done that before with iMovie, QuickTime and even AppleWorks.

I’d rather have a brand new iBooks Author or a feature complete Pages sooner than later, but as long as iBook Author keeps working, well, I guess I can wait.

So, what am I going to do with my books? You might have guessed from my analysis: I am sticking with iBooks Author for the existing books.

I have about 500 pages in several books in various stages of completion. As long as there is no easy way to export them out of iBooks Author, they are staying in there.

iBooks Publishing Frustrations

However, there are some limitations of iBooks Author that I am quite aware of and that I have been looking for solutions to.

Since the topics of the books are very Mac specific, I felt that restricting myself to the iBooks Store and Apple platforms for publishing wouldn’t be much of a limitation. (Though I am still surprised that I repeatedly have to tell people iBooks exists on the Mac and you don’t need an iPad to buy and read iBooks.)

However, self-publishing in the iBooks store is not available in every region. Even some countries where an iBooks store is available don’t allow me to publish to them.

It is not easy or even possible for everyone to get a US Apple ID to purchase items in the US store. So far I have only gotten a handful of requests from readers with this problem, but I do feel sorry for every one of them (and am aware that there will be many more who just gave up without asking).

I’d also like to be able to write not just on a Mac but on an iPad or even the iPhone, whenever the inspiration (or boredom) strikes me. Right now I solve that by typing (and sometimes even dictating) my ideas into Notes and then copying them into iBooks Author later. However, this workflow does not allow me to review or rewrite parts of a book on the road.

Finally, as an admin and coder, I have gotten used to git and some sensible form of version control for the writing process would be nice, too.

The Leanpub Alternative?

So, to address these problems I have been looking for an alternative means of publication. Last week I was recommended Leanpub (thanks, Rob!). I believe their approach to publishing is well suited for technical books.

However, their workflow is entirely based on a markdown variant and builds free-flow ePub books. That means they are more flexible and can support many more reader platforms. On the other hand, the options for rich media and interaction are much more limited (if non-existent). I am not even sure if videos can work reliably in this format (though I plan to try).

On the other hand Leanpub provides better rates for publishers/authors and have a ‘publish early and often’ mentality. They actually allow (and encourage) publishing and selling an ‘incomplete’ book and pushing updates to customers. They even integrate git into the publishing workflow.

As far as I can tell they also sell to anywhere in the world.

Plans

So to summarize here are my specific (mid-term) plans:

  • I will keep ‘Packaging’ and ‘PR3’ in iBooks Author format and the iBooks Store for the forseeable future. Right now converting them to another format would take too much work and time that I’d rather spend on other projects. I will keep updating these books with fixes and new topics.
  • There is a third book that is more than half complete which I plan to keep in iBooks Author format for the same reason (this one has quite a number of screen capture videos, which would be a pain to convert to something else). I hope to get to a point where I can talk more about this project soon.
  • Finally, I have (yet) another project, which I believe is well suited for Leanpub. Last summer, I posted a series of ‘Terminal Primer’ articles and have kept working on that series with a ‘Terminal for Apple Adminstrators’ book in mind. I will (re-)build and publish this in Leanpub as an experiment. For now there is only a ‘stub’ page available, but you can sign up for updates on this project. Of course, I will publish updates on this project here and on Twitter as well.

Let me know what you think!

Weekly News Summary for Admins — 2018-03-23

Expect some extra news next week as Apple has announced a keynote, in a school in Chicago, no less!

MacAD.UK videos are being posted on YouTube, including a presentation by yours truly…

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🐞Bugs and Security

🔨Support and HowTos

Charles Edge has continued his posts on how to replace macOS Server:

Howard Oakley has been doing some interesting posts on the macOS unified log:

🤖Scripting and Automation

🍏Apple Support

♻Updates and Releases

📺To Watch

🎧To Listen

📚Support

I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Update 1.7 for ‘Packaging for Apple Administrators’

The advantage of self-publishing a digital book is that I can write and make updates. Some people have pointed out typos (thank you!) or had questions or comments about the book.

In some cases a question from a client or a MacAdmin Slack forum member will make me re-visit a section of the book and then add or re-write a paragraph or two.

The really great thing is that when you have already purchased the book, you will get the new improved update for free! And even if you have not bought the book yet, then you can know that when you buy it now, you will keep getting improvements in the future. It’s all just bits and bytes after all.

All of that said, the latest version of my book ‘Packages for Apple Administrators’ is now available on the iBooks Store.

I have fixed many typos and clarified a few sentences and paragraph. I have expanded and clarified the instructions for signing packages. There is also a new segment on the order of installation with distribution packages.

If you have already purchased books, all I ask in return for the free update, is to go to the iBooks Store and leave a review. The iBooks Store segregates reviews by territory, so every single one of them will be very important for other users to find and evaluate the book.

Thank you!

Weekly News Summary for Admins — 2018-03-16

Charles Edge is dominating the summary today with a list of posts on how best to replace the waning macOS Server services. Thank you!

We also got the WWDC announcement from Apple. (June 4–8 in San Jose, CA) If you applied to the ticket lottery (before March 22, 10am PDT): Good Luck!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

📰News and Opinion

🐞Bugs and Security

🔨Support and HowTos

🤖Scripting and Automation

♻️Updates and Releases

🎧To Listen

📚Support

I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Weekly News Summary for Admins — 2018-03-09

No big news this week, which is a nice change.

The first iPhone SDK dropped ten years ago this week. Craig Hockenberry wrote a wonderful retrospective which is worth reading.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

📰News and Opinion

🐞Bugs and Security

🔨Support and HowTos

🤖Scripting and Automation

♻️Updates and Releases

📚Support

I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Weekly News Summary for Admins — 2018-03-02

A strange thing happened last Friday (a few hours after the last news summary). Apple updated the kbase article HT208020 – the infamous one that killed imaging. Then the article was reverted back to the old version after a few hours. Presumably some news for 10.13.4 were released prematurely.

Mark Melaccio grabbed screen shots.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

MacAD.UK Session Links

(I have re-included the links from last week.)

🐞Bugs and Security

🔨Support and HowTos

🤖Scripting and Automation

🍏Apple Support

♻️Updates and Releases

📚Support

I do not have any ads on my webpage or this newsletter. However, if you want to support me and this website, then please consider buying one (or both) of my books. (Imagine it’s like a subscription fee, but you also get one or two useful books on top!)

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Single Brackets vs Double Brackets

In my recent post I mentioned in passing, that you should be using double brackets [[…]] for tests in bash instead of single brackets.

This is the post where I explain why. I also talked about this briefly in my MacSysAdmin session: Scripting Bash

Double Brackets are a bashism

Double brackets were originally introduced in ksh and later adopted by bash and other shells. To use double brackets your shebang should be #!/bin/bash not #!/bin/sh.

Since sh on macOS is bash pretending to be sh, double brackets will still work with the wrong shebang, but then your script might break on other platforms where a different shell might be pretending to be sh. Consistent behavior across platforms is the main point why sh is still around, so don’t use double brackets in sh (or use bash to use double brackets).

I go into detail on why to use bash over sh in this post: On the Shebang

Side note on syntax

In shell scripts you usually use tests in if or while clauses. These are tedious to write in the interactive shell. The ‘and’ operator && will execute the following statement only if the preceding statement returns 0 (success). So you can use && to write simple if … then … clauses in a single line.

if [ -d Documents ]
then
    echo "found docs"
fi

and

[ -d Documents ] && echo "found docs"

have the same effect. The second is much shorter, but as soon as the test or the command gets more complex you should revert to the longer syntax.

Alternatively, the ‘or’ operator || will only execute the following statement when the previous statement returns non-zero or fails:

[ -d Documents ] || echo "no docs"

is the same as

if [ ! -d Documents ]
then
    echo "no docs"
fi

What’s wrong with the single brackets?

The single bracket [ is actually a command. It has the same functionality as the test command, except that the last argument needs to be the closing square bracket ]

$ [ -d Documents && echo "found docs"
-bash: [: missing `]'
~ $ [ -d Documents ] && echo "found docs"
found docs
$ test -d Documents  && echo "found docs"
found docs

Note: in bash on macoS both test and [ are built-in commands, but as usual for built-in commands there are also executables /bin/test and /bin/[.

A single bracket test will fail when one of its arguments is empty and gets substituted to nothing:

$ a="abc"
$ b="xyz"
$ [ $a = $b ] || echo "unequal"
unequal
$ unset a
$ [ $a = $b ] || echo "unequal"
-bash: [: =: unary operator expected
unequal

You can prevent this error by quoting the variables (always a prudent solution).

$ [ "$a" = "$b" ] || echo "unequal"
unequal

Double brackets in bash are not a command but a part of the language syntax. This means they can react more tolerantly to ‘disappearing’ arguments:

$ [[ $a = $b ]] || echo "unequal"
unequal

You will also get an error if one of the arguments is substituted with a value with whitespace with single brackets, while double brackets can deal with this.

$ a="a"
$ b="a space"
$ [ $a = $b ] || echo "unequal"
-bash: [: too many arguments
unequal
$ [[ $a = $b ]] || echo "unequal"
unequal

Note: the = operator in sh and bash is for string comparison. To compare numerical values you need to use the -eq (equals), -ne (not equals), -gt (greater than), -ge (greater than or equal), -lt (less than), -le (less than or equal) operators. With double brackets you can also use two equals characters == for a more C like syntax. (or, better, use ((…)) syntax for arithmetic expressions)

Also, when using the = to assign variables, you cannot have spaces before and after the =, while the spaces are required for the comparison operator (both with single and double brackets):

a="a"           # no spaces
b="b"           # no spaces
[ "$a" = "$b" ] # spaces!
[[ $a = $b ]]   # spaces!

Since the single bracket is a command, many characters it uses for its arguments need to be escaped to work properly:

$ [ ( "$a" = "$b" ) -o ( "$a" = "$c" ) ]
-bash: syntax error near unexpected token `"$a"'
$ [ \( "$a" = "$b" \) -o \( "$a" = "$c" \) ]

You could alternatively split this example into two tests: [ "$a" = "$b" ] || [ "$a" = "$c" ].

Double brackets interpret these characters properly. You can also use the (again more C like) && and || operators instead of -a and -o.

 [[ ( $a = $b ) || ( $a = $c ) ]]

In general, you can work around most of the issues with single bracket syntax, but the double bracket syntax is more straight forward and hence more legible and easier to type.

Double bracket features

Aside from the cleaner syntax, there are a few ‘bonus’ features you gain with double brackets.

With double brackets you can compare to * and ? wildcards, and bracket globbing […]:

$ a="Documents"
$ [[ $a = D* ]] && echo match
match
$ a=hat
$ [[ $a = ?at ]] && echo match
match
$ [[ $a = [chrp]at ]] && echo match
match

You can also use < and > to compare strings lexicographically:

$ a=cat
$ b=hat
$ [[ $a < $b ]] && echo sorted
sorted

And you get an operator =~ for regular expressions:

$ a=cat
$ b="the cat in the hat"
$ [[ $a =~ ^.at ]] && echo match
match
$ [[ $b =~ ^.at ]] && echo match

Note that you should not quote the globbing patterns or the regex pattern.

Summary

  • you should use bash for shell scripting on macOS
  • when using bash, you should use double brackets instead of single brackets
  • double brackets are safer, easier to type and read, and also add few neat features