PR3 Book Launch Gift! — Command Line Reference

‘Property Lists, Preferences and Profiles for Apple Adminstrators’ launches today! Go get it before the iBooks Store runs out of bytes!

You can read more about the book in my pre-sale announcement post.

While writing the book. I built a list of common and important commands for the ‘Property Lists, Preferences and Profiles for Apple Administrators’ book. (Thanks to François for the idea!)

The list is included in the book resources as a PDF, so you can print it and hang it above your desk. However, you can also download it directly here.

(If you are a consultant, trainer or other service provider of some sorts, your customers might enjoy these, too. There is even white space on the second page for your contact and logo.)

Enjoy!

And when you finish the book, please leave a review on the iBooks Store!

Update to ‘Packaging for Apple Administrators’

Aside from the new book ‘Property Lists, Preferences and Profiles for Apple Administrators’ (available for pre-sale right now, goes on sale Monday) my first book also got some love. ‘Packaging for Apple Administrators’ got a new cover, to better fit with the cover for PR3 and also a few new internal additions.

iBooks should have already notified owners of the update. If you do not own it yet, go get it on the iBooks Store!

If you like the book, please leave a review!

Terminal and the Clipboard

Continuing in my informal series of Terminal articles, I’d like to visit two tools that help interact from the shell to a particular part of the macOS UI: the clipboard.

The clipboard is the ‘place’ where macOS stores anything you cut (⌘X) or copy (⌘C). Later the system reads from the clipboard (and possibly converts the data) when you paste (⌘V).

Terminal can trivially interact with the clipboard. You can select text in the Terminal and copy it, and then paste it elsewhere, you can also paste text in the command prompt itself.

(This is useful when you read a great iBook with many Terminal commands. You can simply copy the command from the iBook text and paste into Terminal.)

There are however, two commands, specific to macOS that connect the clipboard closer to other shell commands.

pbcopy will take the contents of stdin (usually text, but could be any stream of data) and put them in the clipboard. So, anything you pipe into pbcopy will end up in the clipboard, so you can paste it into a different application.

(NeXTSTEP was the operating system that Mac OS X was originally based on. What was called the ‘clipboard’ in Mac OS was called ‘pasteboard’ in NeXTSTEP. While all references visible to the user where changed to the Mac OS naming, you still find the old NeXTSTEP names in the ‘innards’ of macOS, hence pbcopy and pbpaste.)

Examples:

# easier than open, select all, copy
$ cat ideas.txt | pbcopy

# converts Word Doc to plain text and puts it in the clipboard
$ textutil -convert txt MyBook.docx -stdout | pbcopy

$ ipconfig getifaddr en0 | pbcopy

$ system_profiler SPHardwareDataType SPSoftwareDataType | pbcopy

$ uuidgen | pbcopy

I use the last command a lot when I need UUIDs for custom configuration profiles. It is much easier to pipe the output directly into the clipboard, than to select and copy the output.

One downside of this it, that you cannot see what is piped into the clipboard. You can easily make the new clipboard contents visible by typing pbpaste as the next command:

$ uuidgen | pbcopy
$ pbpaste
A524B454-5B42-4832-943D-896DF755FDEC

or

$ uuidgen | pbcopy; pbpaste
95DC9C0E-052E-4896-A4D3-1BB5EAECD93C

(Several UUIDs were wasted writing this article.)

pbpaste is the counterpart to pbcopy. pbpaste will take the contents of the clipboard, and if they are plain text write them to stdout. That alone can be useful when you want to visualize the clipboard but gets more powerful when you pipe or substitute it into other unix commands.

Examples:

# after copying the output of uuidgen
$ plutil -replace PayloadUUID -string $(pbpaste) MyConfigProfile.mobileconfig

# copy an html snippet from somewhere or
$ echo '<a href="https://scriptingosx.com">Scripting OS X</a>' | pbcopy
# then
$ pbpaste | textutil -stdin -format html -convert rtf -stdout | pbcopy

This last command will convert an HTML snippet into rich text (rtf).

New Book — Property Lists, Preferences and Profiles

As it happens so often, the plan changed.

The plan was to go from ‘Packaging’ right to the obvious sequel ‘Automated Packaging’. And I am working on that book. However, since AutoPkg uses property list files for the recipes, I needed to introduce property list file formats and all the tools, quirks and caveats.

So the sidenote on property lists grew into an appendix. And it kept growing. Then I thought it would be a waste to explain property list files but not talk about preference files. And once you explained preference files, you should also introduce configuration profiles, especially custom profiles. Profiles are property lists after all.

So I moved the chapter on property lists into a new iBook. And I added chapters on preferences and configuration profiles, leading to a cumbersome, but memorable title. (‘PR3’ for short.)

So, here we are, not a sequel, but a ‘sidequel’. I will of course continue writing ‘Automated Packaging’ once this book is published.

PR3 is not quite done yet. I am still getting great feedback from the proofreaders. (Thank you!) But all the small pieces that are still missing or need refinement should fall in to place next week and I aim to release on March 20. (Deep breaths!)

And there will be updates to the iBook (like with ‘Packaging’) as I find more typos and learn more in future.

As with ‘Packaging’ the main target audience is either an administrator who is new to the Mac platform or a Mac user who is new to Mac administration and management. However, the proofreaders have all told me, there are many nuggets and pieces of new information for experienced admins as well.

Until then you can download the first chapter on property list file formats from the iBooks Store!

You can also pre-order and get the book automatically, as as soon as I press the ‘Publish’ button!

Get it on iBooks

Overall, writing these books is a great experience. It is a great chance to dive deeply into a topic. It is also a great chance to give back to the community that is amazingly generous in sharing knowledge and experience. There are many more topics on my list that I would like to write books on.

On Bash History Substitution

There are some things in every profession or area of expertise that seem so obvious to you and most people around you. However, there was a point in your career where you did not know this yet and somebody taught you.

A brief discussion on the MacAdmins Slack yesterday led to such a moment. A bash shortcut that seems so obvious to me and probably to many who read this, but again, not everyone knows. And it was someone’s lucky ten thousand moment to learn an wonderful shell shortcut:

sudo !!

Re-visit your (shell) History

The bash shell remembers a certain number of commands that you entered. This way you can ‘recall’ them with the up arrow or the ctrl-R keystroke. ((hit ctrl-R and type some text, to get the first command in your history containing that text))

There are also history substitutions to save typing. And not only do they save typing but these automations, like tab-completion will make you type less errors.

Sudo Make me a Sandwich ((If you do not recognise that reference, you are one the ten thousand, again.))

So imagine you typed

$ profiles -P
profiles: this command requires root privileges

and the system helpfully reminds you that you need root privileges to run this. You could retype the entire command after sudo. You could also hit up-arrow, ctrl-a and type sudo[space][return], which is some improvement.

Or you just type

$ sudo !!
sudo profiles -P
Password:
There are no configuration profiles installed

The shell will substitute the !! (‘bangbang’) with the previous command. The shell will print the entire command after the substitution (this can be really helpful when things don’t work as expected) and then immediately execute the full command. ((Yes, technically, sudo !![return] is the same number of keystrokes as up-arrow, ctrl-a and sudo[space][return] but I still find it easier.))

Personally, I only ever use this with sudo. I could imagine there are other uses, maybe with ssh user@host -c.

You can also just type !! to repeat the last command, I find ‘up-arrow, return’ to be more intuitive (and one keystroke less).

Historical Arguments

The other history substitution I use is more specific, but still has its uses. The shell substitutes !$ with the last used argument. The catch here is that if the previous command had multiple arguments, !$ will be the very last argument on that line, not all the arguments.

Some examples:

$ mkdir -p MyNewInstaller/payload
$ cd !$
cd MyNewInstaller/payload

$ touch postinstall
$ chmod +x !$
chmod +x postinstall

$ pkgbuild --component /Volumes/Firefox/Firefox.app --install-location /Applications Firefox-51.0.1.pkg
$ cp !$ /Volumes/SoftwareArchive/Mozilla/
cp Firefox-51.0.1.pkg /Volumes/SoftwareArchive/Mozilla/

(Most pkgbuild examples are easily dated with the Firefox version.)

If the last command had no arguments, !$ will be the ‘zeroth’ argument, the command itself:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.3
BuildVersion:   16D32
$ !$
sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.3
BuildVersion:   16D32

More History

There are more bash history substitutions. I found a helpful StackOverflow post with examples here. Most are a bit too cumbersome to actually use the frequently.

Though, I myself had a ‘lucky ten thousand’ moment, since I did not know about !* (all the previous command’s arguments) and !^ (the previous command’s first argument) until today.

Path Through History

This is not directly related to the shell history, but `cd` also remembers your previous location in the file system and you can quickly jump back with `cd -`.

$ cd ~/Projects
$ cd ~/Library/AutoPkg/
$ cd -
/Users/armin/Projects
$ pwd
/Users/armin/Projects

Is Packaging Dead?

I wrote a book on Packaging, you may have heard. (Please buy it! If you have bought and read it, please leave a review!)

I have been asked whether it was wise to publish a book on Packaging now. There is lots of talk in the Mac Admin community about the future of admin tools. The general consensus is that Apple wants to move macOS further towards a locked down system like iOS (and watchOS and tvOS). We have already seen the first moves in that direction with System Integrity Protection. Some suspect the new file system APFS may break traditional imaging workflows.

Note: I used to work for Apple, but that was years ago. I have no inside knowledge and all this is speculation, based on my many years of experience working with and for the company. I hope I am right, but would be the first to admit that I have made very wrong predictions regarding Apple before.

Up to now, these changes are not too severe. Some (myself included) even approve of deprecating legacy practices such as imaging, given that the alternative solutions bring many other benefits. However, there is the spectre of the completely locked down macOS, which can only be managed the same as iOS can: with an MDM server.

I do not think this is Apple’s ultimate goal. I also believe it will take longer than many are afraid of, and it may be even impossible in the long term without breaking the nature of macOS fundamentally. I’d like to give a few examples:

Omnigroup has been a productive and successful developer for macOS since NeXTSTEP. They were on Mac OS X before there was a Mac OS X. Last year, the OmniGroup switched some of their applications in the Mac AppStore from the one-time purchase model to a free-download with trial and in-app purchase model. I understand why Omnigroup decided to do this. However, Apple has no VPP solution for in-App-Purchases so an administrator cannot deploy Omni software with an MDM anymore. OmniGroup still sells their apps directly, so you can buy a volume deployment license directly from OmniGroup and deploy without the Mac AppStore, VPP, or MDM.

Veertu is an interesting new virtualisation solution, which started out in the Mac App Store. It could do that because it uses Apple’s Hypervisor Framework, which is meant to provide the system resources necessary for virtual machines to SandBoxed apps. However, last year, Veertu removed their solution from the Mac App Store because the limitations were still too stringent. Similarly, VMware Fusion, Parallels and VirtualBox are all not in the Mac App Store and not deployable with VPP or MDM.

MATLAB is a powerful solution for building scripts and tools with mathematical and engineering libraries. Calling MATLAB an ‘app’ would be like calling your MacBook a ‘typewriter with a screen.’ Scientists and engineers not only use MATLAB to build complex mathematical and statistical solutions, they share those solutions as products or open source tools. Many people may not even be aware that they are using a tool running the MATLAB runtime in the background. MATLAB is one of many powerful math/numerical/statistics tools, such as Wolfram Mathematica, IBM SPSS, R, etc. None of which are in the Mac App Store or even possible with the current Mac App Store limitations.

These math tools are deeply entwined with the UNIX philosophy of tools passing data from one to the other to do specialised steps of processing. Some workflows I have encountered are Rube-Goldberg-Machine-like sequences of MATLAB, Python, and bash scripts with a Qt or Java UI. As monstrous as these ‘solutions’ may be, they are essential for many people’s day-to-day work.

The only Adobe application in the Mac App Store is Photoshop Elements. There are many more Adobe apps in the iOS App Store, but none of their core Creative Cloud applications. One of the reasons may be that many Mac users, not only rely on the applications, but on plug-ins for these applications.

For some solutions, like AutoCad, the plug-ins have plug-ins. There is no method to provide, sell and deploy plug-ins for applications in the Mac App Store. You can argue that App Extensions can fill that functionality, and may even be more flexible than app-specific plug-ins, but they have certainly not been adopted on a large scale yet.

While macOS has improved how users discover and install drivers for printers, there is no way of managing driver deployment in the MDM spec. Administrators still have to deal with downloading and deploying printer driver installers and settings. The same is true for any other hardware driver software. While you can argue that the vendors sometimes seem to go out of their way to make driver deployment as complex as possible, even the best of them have no way of providing their software in the Mac App Store or manage it with an MDM.

This is but a short list of examples. I am certain any admin can easily list a many more. All of them have in common that Mac administrators need to use package installers and/or shell scripts to install and manage these solutions. Sometimes you can re-use the vendor installer, but even then you need to know how to inspect and verify these files. In many cases you have to reverse engineer the installation and re-build your own packages. (How to do all of this is covered in my book: “Packaging for Apple Administrators”)

As long as the Mac App Store has no manageable (VPP) solutions for upgrades, subscriptions and in-App-Purchases, it has to be considered broken for client management. Any such feature which gets added to the Mac App Store (such as subscriptions last year) have to have a VPP solution, otherwise it is useless for managed deployments.

The Mac App Store excludes entire categories of software because of the sandbox requirement, software providers. Users and admins still need a way to deploy that software. If Apple were to go ahead and lock down macOS completely, users and admins would have to either jailbreak their Macs, or abandon the Mac platform.

The number of users who would be affected by this is paltry, compared to the hundreds of millions of iOS devices Apple sells each year. So, the cynic will, “what does Apple care if a few hundred thousand users leave?”

A locked down macOS would provide little benefit over iOS running on an iPad Pro. A locked down MacBook may have more processing power (barely), and more ports for connecting hardware, but the software to use them to their full extent would simply not be available.

Apple wants to lock down macOS to make it more secure, because that distinguishes macOS from the other desktop systems. On the other hand, Apple has an interest in keeping macOS open because it is not iOS.

Apple has its own set of Mac applications that have no iOS counterpart: Xcode, Logic Pro X and Final Cut Pro. These are the ‘Pro’ apps, the lighter apps: GarageBand and iMovie have iOS counterparts. Swift Playgrounds which may be considered ‘Xcode lite’ is iOS only.

There are good reasons for each of these not being on iOS. Final Cut Pro and Logic Pro X rely very much on external hardware for input, most of which is not available for iOS. Xcode relies on integration with many command line tools (llvm, git, etc.) to develop its full potential. There may be some projects within Apple that can be built without shell scripts, but I am sure they are in the minority. Just look at the postinstall scripts of any Apple installer package.

Interesting Sidenote: Xcode, GarageBand and Logic Pro bypass the Mac App Store limitations by installing additional components at first run. Apple is hitting the limitations of their own rules. There is also no way of deploying these additional components with an MDM.

There are many people at Apple who use their Macs for graphic design, hardware design, running complex computations, writing shell scripts, compiling complex software and hardware drivers, and testing on virtual machines, etc. Do you believe Apple wants to buy them Dells so they can run Photoshop, bash, AutoCAD, SPSS, or VMware? Apple also needs to deploy and manage the Macs in their retail stores. They are, after all, one of the largest deployments of their own OS.

So, assuming Apple is not planning to lock down macOS like iOS, what are they going to do?

They certainly will continue to restrict access by users and processes to security critical parts of the OS. They also want to gain (more) control of vectors where malicious software can inject itself into the system. Apple obviously wants to use MDM as the main conduit for Macs to be managed. The MDM spec already has the ability to deploy an custom package to a client. This is probably supposed to push client agents for management systems for features not included in the MDM spec (such as reporting).

Apple, third party MDM vendors and open source MDMs could build on this existing feature to provide package installation of software for non-App Store solutions. This alone would alleviate many concerns that administrators currently have. Maybe the packages will have to be signed with an Apple developer certificate or some other extra security step. (Learn how sign packages in my book.) This will require major re-architecting of many tools such as Munki, but those are do-able, maybe even desirable. The format the MDM spec is using to install software is, not surprisingly, a flat package installation file, so packaging skills will be needed more, rather than less in this scenario.

From my perspective, Apple’s choices are to obsolete the Mac platform, or to gradually keep improving security and working on the Mac App Store, VPP, DEP and MDM as management solutions while maintaining the openness that makes the Mac such a powerful and creative tool.

While I believe that ‘pro’ users of the Mac platform need to get accustomed to the fact that they are not Apple’s main focus anymore, I also refuse to believe that Apple is planning to obsolete the Mac entirely, because they would hurt themselves in the process.

The balance between security and openness will not be easy to find. The openness is what sets the Mac platform apart from iOS. The control and security is what sets macOS apart from the other desktop operating systems. There will be much back and forth between the ‘pro’ users and Apple. We as admins have to keep harping and nagging at Apple to do the right thing. (File bugs!)

The macOS open Command

You can learn more about using Terminal and the shell on macOS in my my book: “macOS Terminal and Shell” — Thank you!

Most Terminal users will know that

$ open .

will open the current working directory in a Finder window. (You, dear wonderful reader, know this because you read my previous post on Terminal-Finder Interaction.)

However, the open command can do so much more.

Folders

Trivially, it cannot merely open the current working directory, but any path:

$ open ~/Library/Preferences
$ open /etc
$ open ../..

This can be used as a quick way to navigate to hidden directories.

You can also open multiple folders at once:

$ open ~/Documents ~/Desktop ~/Downloads
$ open ~/D*

To clean up, you can option-click any close button in a Finder window to close all Finder windows. Or you can use the keyboard short cut ⌘⌥W.

Files

open can also open files. In general you can think of open as the command line equivalent of double-clicking a file or folder in Finder.

$ open document.pdf

will open document.pdf in the current working directory with the default application for PDF files (usually Preview). You can use this against multiple files as well:

$ open ~/Desktop/Screen\ Shot\ *.png

will open all screenshot files (if any) in a viewer in the default application (Preview).

Applications

If you have changed the default application that handles a file type or want to override the default application, you can use the -a option:

$ open -a Preview ~/Desktop/Screen\ Shot\ *.png
$ open -a TextEdit web.html

You can specify just the name of an application or the full path, i.e. /Applications/Preview.app. If you need to be specific, you can also specify an application’s bundle identifier with -b com.apple.Preview.

If you want to open a document but keep the application and the new document window in the background, use the -g option.

$ open -g ~/Desktop/Screen\ Shot\ *.png

Text Editors

There are two interesting special cases for designating applications:

$ open -e helloworld.swift

will open a file with TextEdit.

$ open -t helloworld.swift

will open a file with the default application for text files (.txt file extensions) You can use the Finder Info panel to change the default application or, if you want more fine grained control use RCDefaultApp. In the default macOS config these are the same, but you can of course change the default app to your favourite text editor. (Many text editors, like BBEdit and Atom, have their own CLI tool, but if they don’t, you can use open -t instead.)

You can even pipe text into open with the -f option:

$ ls -l ~ | open -f     # TextEdit, '-e' is implied
$ ls -l ~ | open -tf    # default application assigned to txt

You can set your $EDITOR environment variable: EDITOR='open -tnW'; export EDITOR and then command lines tools that expect text from an editor, like git commit, will get the text from open and thus your default text editor instead. The -n option will actually open a new (sometimes second) instance of the application and the command line tool will resume when you quit this new instance. This a somewhat awkward workflow for Mac users. Many text editors provide a command line tool that may work better in these cases. For BBEdit the correct $EDITOR value is bbedit -w --resume.

Showing Files in Finder

If you are working on a file in Terminal and want to locate it in Finder, open can do better than just opening the enclosing folder. It can select a given file as well:

$ open -R helloworld.swift

Will open a Finder window with the enclosing folder of helloworld.swift and select the file. (You can pass multiple files into open -R but it will only select the last file in the list.)

URLs

Finally there is one more useful thing you can open:

$ open https://scriptingosx.com   # default browser
$ open vnc://TestMac.local       # Screen Sharing
$ open x-man-page://open         # show man page in Terminal

and, as always, you can use the -a option to override the default application:

$ open -a Firefox https://scriptingosx.com

Header files

For the sake of being complete: you can also open header files quickly with open. The -h option will search and open the header file for a given class. There is an additional -s option to choose an SDK:

$ open -h NSTask
$ open -h NSTask -s OSX10.12
$ open -h UIView.h -s iPhoneOS10.2
$ open -a BBEdit -h NSTask

If the search term is ambiguous open will list all the options.