Category: Command Line

Terminal and the Clipboard

Continuing in my informal series of Terminal articles, I’d like to visit two tools that help interact from the shell to a particular part of the macOS UI: the clipboard.

The clipboard is the ‘place’ where macOS stores anything you cut (⌘X) or copy (⌘C). Later the system reads from the clipboard (and possibly converts the data) when you paste (⌘V).

Terminal can trivially interact with the clipboard. You can select text in the Terminal and copy it, and then paste it elsewhere, you can also paste text in the command prompt itself.

(This is useful when you read a great iBook with many Terminal commands. You can simply copy the command from the iBook text and paste into Terminal.)

There are however, two commands, specific to macOS that connect the clipboard closer to other shell commands.

pbcopy will take the contents of stdin (usually text, but could be any stream of data) and put them in the clipboard. So, anything you pipe into pbcopy will end up in the clipboard, so you can paste it into a different application.

(NeXTSTEP was the operating system that Mac OS X was originally based on. What was called the ‘clipboard’ in Mac OS was called ‘pasteboard’ in NeXTSTEP. While all references visible to the user where changed to the Mac OS naming, you still find the old NeXTSTEP names in the ‘innards’ of macOS, hence pbcopy and pbpaste.)

Examples:

# easier than open, select all, copy
$ cat ideas.txt | pbcopy

# converts Word Doc to plain text and puts it in the clipboard
$ textutil -convert txt MyBook.docx -stdout | pbcopy

$ ipconfig getifaddr en0 | pbcopy

$ system_profiler SPHardwareDataType SPSoftwareDataType | pbcopy

$ uuidgen | pbcopy

I use the last command a lot when I need UUIDs for custom configuration profiles. It is much easier to pipe the output directly into the clipboard, than to select and copy the output.

One downside of this it, that you cannot see what is piped into the clipboard. You can easily make the new clipboard contents visible by typing pbpaste as the next command:

$ uuidgen | pbcopy
$ pbpaste
A524B454-5B42-4832-943D-896DF755FDEC

or

$ uuidgen | pbcopy; pbpaste
95DC9C0E-052E-4896-A4D3-1BB5EAECD93C

(Several UUIDs were wasted writing this article.)

pbpaste is the counterpart to pbcopy. pbpaste will take the contents of the clipboard, and if they are plain text write them to stdout. That alone can be useful when you want to visualize the clipboard but gets more powerful when you pipe or substitute it into other unix commands.

Examples:

# after copying the output of uuidgen
$ plutil -replace PayloadUUID -string $(pbpaste) MyConfigProfile.mobileconfig

# copy an html snippet from somewhere or
$ echo '<a href="https://scriptingosx.com">Scripting OS X</a>' | pbcopy
# then
$ pbpaste | textutil -stdin -format html -convert rtf -stdout | pbcopy

This last command will convert an HTML snippet into rich text (rtf).

On Bash History Substitution

There are some things in every profession or area of expertise that seem so obvious to you and most people around you. However, there was a point in your career where you did not know this yet and somebody taught you.

A brief discussion on the MacAdmins Slack yesterday led to such a moment. A bash shortcut that seems so obvious to me and probably to many who read this, but again, not everyone knows. And it was someone’s lucky ten thousand moment to learn an wonderful shell shortcut:

sudo !!

Re-visit your (shell) History

The bash shell remembers a certain number of commands that you entered. This way you can ‘recall’ them with the up arrow or the ctrl-R keystroke. ((hit ctrl-R and type some text, to get the first command in your history containing that text))

There are also history substitutions to save typing. And not only do they save typing but these automations, like tab-completion will make you type less errors.

Sudo Make me a Sandwich ((If you do not recognise that reference, you are one the ten thousand, again.))

So imagine you typed

$ profiles -P
profiles: this command requires root privileges

and the system helpfully reminds you that you need root privileges to run this. You could retype the entire command after sudo. You could also hit up-arrow, ctrl-a and type sudo[space][return], which is some improvement.

Or you just type

$ sudo !!
sudo profiles -P
Password:
There are no configuration profiles installed

The shell will substitute the !! (‘bangbang’) with the previous command. The shell will print the entire command after the substitution (this can be really helpful when things don’t work as expected) and then immediately execute the full command. ((Yes, technically, sudo !![return] is the same number of keystrokes as up-arrow, ctrl-a and sudo[space][return] but I still find it easier.))

Personally, I only ever use this with sudo. I could imagine there are other uses, maybe with ssh user@host -c.

You can also just type !! to repeat the last command, I find ‘up-arrow, return’ to be more intuitive (and one keystroke less).

Historical Arguments

The other history substitution I use is more specific, but still has its uses. The shell substitutes !$ with the last used argument. The catch here is that if the previous command had multiple arguments, !$ will be the very last argument on that line, not all the arguments.

Some examples:

$ mkdir -p MyNewInstaller/payload
$ cd !$
cd MyNewInstaller/payload

$ touch postinstall
$ chmod +x !$
chmod +x postinstall

$ pkgbuild --component /Volumes/Firefox/Firefox.app --install-location /Applications Firefox-51.0.1.pkg
$ cp !$ /Volumes/SoftwareArchive/Mozilla/
cp Firefox-51.0.1.pkg /Volumes/SoftwareArchive/Mozilla/

(Most pkgbuild examples are easily dated with the Firefox version.)

If the last command had no arguments, !$ will be the ‘zeroth’ argument, the command itself:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.3
BuildVersion:   16D32
$ !$
sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.3
BuildVersion:   16D32

More History

There are more bash history substitutions. I found a helpful StackOverflow post with examples here. Most are a bit too cumbersome to actually use the frequently.

Though, I myself had a ‘lucky ten thousand’ moment, since I did not know about !* (all the previous command’s arguments) and !^ (the previous command’s first argument) until today.

Path Through History

This is not directly related to the shell history, but `cd` also remembers your previous location in the file system and you can quickly jump back with `cd -`.

$ cd ~/Projects
$ cd ~/Library/AutoPkg/
$ cd -
/Users/armin/Projects
$ pwd
/Users/armin/Projects

The macOS open Command

You can learn more about using Terminal and the shell on macOS in my my book: “macOS Terminal and Shell” — Thank you!

Most Terminal users will know that

$ open .

will open the current working directory in a Finder window. (You, dear wonderful reader, know this because you read my previous post on Terminal-Finder Interaction.)

However, the open command can do so much more.

Folders

Trivially, it cannot merely open the current working directory, but any path:

$ open ~/Library/Preferences
$ open /etc
$ open ../..

This can be used as a quick way to navigate to hidden directories.

You can also open multiple folders at once:

$ open ~/Documents ~/Desktop ~/Downloads
$ open ~/D*

To clean up, you can option-click any close button in a Finder window to close all Finder windows. Or you can use the keyboard short cut ⌘⌥W.

Files

open can also open files. In general you can think of open as the command line equivalent of double-clicking a file or folder in Finder.

$ open document.pdf

will open document.pdf in the current working directory with the default application for PDF files (usually Preview). You can use this against multiple files as well:

$ open ~/Desktop/Screen\ Shot\ *.png

will open all screenshot files (if any) in a viewer in the default application (Preview).

Applications

If you have changed the default application that handles a file type or want to override the default application, you can use the -a option:

$ open -a Preview ~/Desktop/Screen\ Shot\ *.png
$ open -a TextEdit web.html

You can specify just the name of an application or the full path, i.e. /Applications/Preview.app. If you need to be specific, you can also specify an application’s bundle identifier with -b com.apple.Preview.

If you want to open a document but keep the application and the new document window in the background, use the -g option.

$ open -g ~/Desktop/Screen\ Shot\ *.png

Text Editors

There are two interesting special cases for designating applications:

$ open -e helloworld.swift

will open a file with TextEdit.

$ open -t helloworld.swift

will open a file with the default application for text files (.txt file extensions) You can use the Finder Info panel to change the default application or, if you want more fine grained control use RCDefaultApp. In the default macOS config these are the same, but you can of course change the default app to your favourite text editor. (Many text editors, like BBEdit and Atom, have their own CLI tool, but if they don’t, you can use open -t instead.)

You can even pipe text into open with the -f option:

$ ls -l ~ | open -f     # TextEdit, '-e' is implied
$ ls -l ~ | open -tf    # default application assigned to txt

You can set your $EDITOR environment variable: EDITOR='open -tnW'; export EDITOR and then command lines tools that expect text from an editor, like git commit, will get the text from open and thus your default text editor instead. The -n option will actually open a new (sometimes second) instance of the application and the command line tool will resume when you quit this new instance. This a somewhat awkward workflow for Mac users. Many text editors provide a command line tool that may work better in these cases. For BBEdit the correct $EDITOR value is bbedit -w --resume.

Showing Files in Finder

If you are working on a file in Terminal and want to locate it in Finder, open can do better than just opening the enclosing folder. It can select a given file as well:

$ open -R helloworld.swift

Will open a Finder window with the enclosing folder of helloworld.swift and select the file. (You can pass multiple files into open -R but it will only select the last file in the list.)

URLs

Finally there is one more useful thing you can open:

$ open https://scriptingosx.com   # default browser
$ open vnc://TestMac.local       # Screen Sharing
$ open x-man-page://open         # show man page in Terminal

and, as always, you can use the -a option to override the default application:

$ open -a Firefox https://scriptingosx.com

Header files

For the sake of being complete: you can also open header files quickly with open. The -h option will search and open the header file for a given class. There is an additional -s option to choose an SDK:

$ open -h NSTask
$ open -h NSTask -s OSX10.12
$ open -h UIView.h -s iPhoneOS10.2
$ open -a BBEdit -h NSTask

If the search term is ambiguous open will list all the options.

Terminal–Finder Interaction

Mac Admins have to work a lot in Terminal. This seems counter-intuitive for an OS that is famed for its user interaction. I can’t talk for all admins, but for me the strength of macOS/OS X was always in the combination of ‘clicky’ UI and command line. When you know what you are doing, you can get the best of both worlds.

I remember an Apple marketing slogan: “The power of Unix, the simplicity of Mac” This is from OS X Lion, so more than five years old by now. The future will show how long Apple still values the ‘powerful’ Unix underpinnings. But for now, they are still available and I am going to use them.

All of that said, the CLI and the UI are not entirely separate areas in macOS, there is a lot of overlap and there are functions in Finder and Terminal that allow for quick interaction between the two.

Finder to Terminal

You can drag any folder from Finder to the Terminal application icon in the dock and Terminal will open a new window and change the working directory to the folder you dragged.

Movie 1: Drag Folder onto Terminal

You can drag the folder icon from the Finder window title bar, as well.

Movie 2: Drag Folder from Title Bar onto Terminal

When you drag any file into an open Terminal window, it will insert the full path to that file or folder with spaces and other special characters properly escaped.

Movie 3: Drag File onto Terminal

You can drag multiple files and Terminal will insert all of their paths, separated by spaces. For example you can type file[space] in Terminal and then drag multiple files into that window and hit return, to get information on the file type.

Movie 4: Drag Multiple Files onto Terminal

If you prefer, you can get the same effect with copy and paste. Just select the files in Finder, choose ‘Copy’ (⌘C), switch to Terminal and ‘Paste’ (⌘V).

Update: I knew I had forgotten one. Thanks to Elliot Jordan who pointed this one out on Twitter:

Dragging a folder into a Terminal window while holding the command (⌘) key will add cd before the path to a folder. When you command-drag a file it will cd to the enclosing folder of that file.

Movie 5: Command Drag a Folder to Terminal

Getting Finder path from Terminal

If you are already in Terminal and want to get the frontmost Finder window, we have to do some homework first. (I got the idea for this script from this post, though I have modified its solution somewhat.) This command

$ osascript -e 'tell app "Finder" to get posix path of ((target of window 1) as alias)'

will give us the correct path, but it has two downsides: a) it is awfully complex to type repeatedly and b) it fails with an error if no Finder window is open.

To avoid typing this long command every time, we have two options. You can either define the command as a function in your .bash_profile (or the respective profile for your preferred shell) or you can save it as a script in your $PATH.

To define it as a function, add this to the .bash_profile:

# prints the path of the front Finder window. Desktop if no window open
function pwdf () {
    osascript <<EOS
        tell application "Finder"
            if (count of Finder windows) is 0 then
                set dir to (desktop as alias)
            else
                set dir to ((target of Finder window 1) as alias)
            end if
            return POSIX path of dir
        end tell
EOS
}

# changes directory to frontmost Finder window
alias cdf='pwdf; cd "$(pwdf)"'

The pwdf function will just print the path to the frontmost Finder window to stdout.

There is also added an alias to quickly change directories to the frontmost Finder window. cdf will also print the path it is changing to, like the cd - command. (Which changes to the previous working directory.) This has to be defined as an alias, since scripts cannot change a shell’s working directory.

If you prefer to define the pwdf command from a script file use the following code:

Save this as a text file without extension in a folder in your $PATH, and set the executable bit with chmod +x /path/to/pwdf. You also need to remove (or comment) the function from your profile, since that would override the script.

Using the script form of osascript allows us to pass arguments into the AppleScript. (You can do that with function as well, but the syntax gets really messy quickly.) This script will list the paths to all open Finder windows with the -a|--all argument. Also when you provide a string as an argument it will search for a Finder window containing that string:

$ pwdf Pref
/Users/armin/Library/Preferences/

The open command

You can also go from Terminal to the Finder. This usually as simple as typing

$ open .

This will open the current working directory ‘.’ in a Finder window.

This is usually where most online ‘hints’ for the open command start and end. However, open has so much more to offer. So much more, in fact, that I will cover the open command in a separate article.

Editing Property Lists with plutil

I stumbled over these option this morning. I do not know when they were introduced, but I can see the options in 10.11 and 10.12. You can see them yourself with plutil -help. (The options are not listed in the man page.)

Note: Managing and editing Property List files and preferences is covered in much more detail and depth in my book “Property Lists, Preferences and Profiles for Apple Administrators“

Quick recap: plutil manipulates property list files. Its main use up to now was to convert between property list formats (mainly from binary plists to something readable)

$ plutil -convert xml1 /path/to/propertylist.plist

and to check wether the syntax is valid

$plutil -lint /path/to/propertylist.plist

On Sierra, when you run plutil -help you see some new options. These allow you to directly manipulate keys and values in a property list. This may be useful to replace PListBuddy and defaults to manipulate property lists.

When testing this I noticed one downside of plutil immediately: it cannot be used to create a new property list file. Copy this to create an empty plist file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict/>
</plist>

Inserting a new key/value pair

$ plutil -insert somekey -string somevalue test.plist
$ plutil -insert flag -bool YES test.plist
$ plutil -insert bestNumber -integer 1 test.plist
$ plutil -insert pi -float 3.141592 test.plist

This is pretty straightforward.

Inspecting a property list

You can use the -p option to check our progress:

$ plutil -p test.plist
{
  "newkey" => "newvalue"
  "pi" => 3.141592
  "bestNumber" => 1
  "flag" => 1
}

This uses a non-standard output format, and the help text warns to use this parse plists. But it will do to look at the content.

Note: you can use plutil -p to read the content of binary plists without converting!

Modifying values

You can modify values with the -replace option.

$ plutil -replace flag -bool NO test.plist

Note that you can create new entries with -replace:

$ plutil -replace otherkey -string othervalue test.plist

But you cannot overwrite an existing value with -insert.

Deleting values

Very straightforward:

$ plutil -remove otherkey test.plist

Arrays

You can insert an empty:

$ plutil -insert list -xml '<array/>' test.plist

or

$ plutil -insert list -json '[]' test.plist

and add items to the array

$ plutil -insert list.0 -string 'list item 1' test.plist
$ plutil -insert list.1 -string 'list item 2' test.plist
$ plutil -insert list.2 -string 'list item 3' test.plist

or do it all at once

$ plutil -replace list -json '[ "yes", "no", "maybe" ]' test.plist

Dictionaries

$ plutil -insert dictionary -xml '<dict/>' test.plist
$ plutil -replace dictionary -json '{}' test.plist
$ plutil -insert dictionary.key1 -string value1 test.plist
$ plutil -replace dictionary -json '{ "otherkey" : "othervalue" }' test.plist

Getting Values

It looks like -extract is meant to get values from a property list, but there is caveat. -extract will not merely get the value of a key in the property list but will write it to a new file! And by default if you do not provide an new output file path with the -o option it will overwrite the current file with the extracted data.

The proper, non-destructive syntax to use -extract is:

$ plutil -extract list xml1 -o - test.plist 
$ plutil -extract list json -o - test.plist

This will print a full property list file to stdout. The -o - option tells plutil to print to stdout. You can give a filename instead of the -.

Since the output is encumbered with the json or xml syntax, it will be hard to use this to get to property list values in shell scripts. However, it still may be useful to, well, extract property list data from a complex plist file.

Conclusion

Keep in mind that there now is an alternative to defaults and PlistBuddy. Not having to convert a plist before changing data might be helpful, as well as the possibility to manipulate arrays and dictionaries with key paths. (You still should always use defaults when working with preference plist files, since defaults will go through the preferences system and possibility notify a process to update data.)

If you are using python or a similar high level scripting language it will still be more effective to use the libraries for property lists.

SSH-installer

Mac System Administrators regularly have to build and test installer packages. One of the best ways to test a package is to install it on a virtual machine. You can then quickly revert the VM to a snapshot after the installation and try again.

Since most package building happens on the command line, the fastest way to copy and install a package is with ssh. You can use scp to copy the pkg file and then run the installer command in an ssh session.

Doing two separate commands over and over seemed quite tedious. so I built myself a short script that combined both. ssh-installer takes two arguments, the first is the remote host in the form [user@]hostname (user is optional) and the second in the pkg file you want to send and install. All other arguments will be passed into the installer command on the remote host. You will probably want to add -target / in most cases. I did not add the -target as a default in case you want to test installer’s other parameters.

This script is just a shortcut for a task I have to do several times while building and testing packages. It is not in any way meant to serve as a means of managing Mac clients. You really should use a decent management system for that.