Terminal Primer – Part 3 – Special Characters

It is vacation time here in the Scripting OS X headquarters and I will be traveling with family for most of August. The Weekly News Summaries are on also vacation, but to compensate for that I will be posting a draft of parts of my next book on macOS Terminal and bash. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

There are a group of characters that have special meaning in bash.

| & : ; ( ) < > ~ * @ ? ! $ # [ ] { } \ / ' " `

Also whitespace characters also need to be treated with care:

space, tab, return, and newline

The space character is a legal and often used character in file names on macOS. However, in bash and other shell commands , a space character (and other whitespace characters) separates a command from an argument and the arguments from each other.

When you try to enter a filename with a space, you will get an error:

$ cd /Library/Application Support
-bash: cd: /Library/Application: No such file or directory

To convince the shell that ‘/Library/Application Support’ belongs together, you can either ‘escape’ the space character or ‘quote’ the path.

Experienced users who have worked in a UNIX environment for a long time tend to avoid these special characters in filenames. However, as a system administrator, your users will probably not heed any rules you may want to impose. You will have to deal with many possible combinations.

Escaping Characters

The escape character in bash is the backslash \. A character that follows a backslash will be treated with no special meaning:

$ cd /Library/Application\ Support

In Finder, you can name files and folders nearly any way you want. When you encounter special characters from the list above you have to escape them with backslash. For a directory named ‘Project (Old & New)’ you would type:

$ cd Project\ \(Old\ \&\ New\)

All of this further confused by the fact that the shell will happily display the path with the unescaped special characters:

$ pwd
/Users/armin/Project (Old & New)

Separation Characters

In bash (and in Unix in general) files and directory names cannot contain a forward slash / since the character is used in paths to separate directories. However, Finder lets you name a file or folder with a forward slash, e.g. ‘Yes/No’.

On the other hand, Finder does not let you name a file or folder with a colon :. The underlying HFS+ file system uses the colon as a separator.

This conflict is solved by translating a / in the Finder (and underlying file system) to a colon : in the shell and vice versa.

A folder named ‘Yes/No/Maybe’ in Finder will appear as Yes:No:Maybe in the shell and you have to escape the colons when using the path in a command:

$ cd Yes\:No\:Maybe

Note: some characters that are legal on macOS might not be on file servers, which are usually hosted by other operating systems.

Quoting

As seen above, escaping characters can make the path quite unreadable. You can also place the name or path in quotes:

$ cd 'Project (Old & New)'

In bash you can use single quotes ' or double quotes " to quote paths.

Single quotes are more effective. Any character in single quotes is used as is, with no special function. Even the backslash character has no special function. The only character you cannot use in single quotes is the single quote itself.

Double quotes " are ‘weaker’ quoting. Double quotes remove the special function from all special characters except $, `, \, and !. Within double quotes you can use the backslash to escape $, ", `, and \ (but not the !).

Escape Strategies

In general, single quotes are most useful and easiest to use. However, you cannot use single quotes when the filename contains a single quote.

Double quotes still require some characters to be escaped with the backslash and cannot deal with an exclamation mark !.

Backslash escaping works in nearly all cases, but can be tricky to type right and is quite illegible.

name (in Finder) Backslash Escape single Quotes Double Quotes
My Great Folder My\ Great\ Folder 'My Great Folder' "My Great Folder"
“New” Files \"New\"\ Files '"New" Folder' "\"New\" Folder"
‘Old’ Stuff \'Old\'\ Stuff cannot escape ' "'Old' Stuff"
Important! Important\! 'Important!' cannot escape !
Bump \m/ Bump \\m\: 'Bump \m:' "Bump \\m:"
Do@Home Do\@Home 'Do@Home' "Do@Home"
Yes/No/Maybe Yes\:No\:Maybe 'Yes:No:Maybe' "Yes:No:Maybe"
Project (Old & New) Project\ \(Old\ \&\ New\) 'Project (Old & New)' "Project (Old & New)"
Profit$$$ Profit\$\$\$ 'Profit$$$' "Profit\$\$\$"

Quoting and Tab Completion

When typing paths, always use tab completion to be safe. Tab completion uses backslash escaping by default.

$ cd Proj⇥
$ cd Project\ \(Old\ \&\ New\)/

However, when you start a quoted path, tab completion will complete in quoted form.

$ cd 'Pro⇥
$ cd 'Projects (Old & New)'/

Tab completion is even smart enough to change the approach when the strategy you chose (i.e. double quotes) cannot work:

$ cd "Imp⇥
$ cd Important\!/

Quoting and Home Path

Since you generally use quoting to avoid bash changing characters, you cannot use the ~ to get a short cut to your home directory in quotes.

However, you can leave the ~ outside of the quotes and get the best of both worlds:

$ cd ~/'Project (Old & New)'

When you use double quotes, you can also use the $HOME environment variable:

$ cd "$HOME/Project (Old & New)"

We will cover environment variables later.

Terminal Primer – Part 2 – Navigating the File System

It is vacation time here in the Scripting OS X headquarters and I will be traveling with family for most of August. The Weekly News Summaries are also on vacation, but to compensate for that I will be posting a draft of parts of my next book on macOS Terminal and bash. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

Working Directory

Back to our earlier example, the pwd command. You entered the command pwd and the shell returned a result:

$ pwd
/Users/armin

Your output will be different. Unless your name is also armin, the path will end with your user name, not mine. Depending on the configuration of your Mac, your path might be entirely different.

pwd is short for ‘print working directory.’ It shows the folder this shell is currently working in, much like a Finder window displays the contents of a certain folder.

On macOS, Terminal will also show the working directory in the window title bar.

Paths

Modern file systems are hierarchical and have many folders and files nested in each other. In a Unix shell you denote a folder or file in another folder with the forward ‘/’ slash character. The complete series of folders to a file or folder is called its ‘path.’

For example, if there is a file hello.txt in your Documents folder in your home directory, its complete path or absolute path is:

/Users/armin/Documents/hello.txt

A path starts at the ‘root’ of the file system, which is shown as simply /. On macOS the root of the file system is the top level of the volume or disk with the active system, i.e. the current startup volume of the Mac.

Note: The file system root / should not be confused with the system user root.

This path tells me (and the shell) that there is a file ‘hello.txt’ in a folder ‘Documents’ in a folder ‘armin’ in a folder ‘Users’ on the root of the file system, the current startup volume.

Note: Finder in macOS and other graphical interfaces use the word ‘folder.’ Unix and other command line shells use the word ‘directory.’ They are the same.

Relative Paths

Addressing files and folders with their full path each and every time would be very tedious. To help with that a shell has a ‘current working directory.’

You can refer to files and folder relative to the shell’s current working directory. When using relative paths, there is no initial /.

hello.txt

refers to the file ‘hello.txt’ in the current working directory.

Documents/hello.txt

refers to the file ‘hello.txt.’ in the folder ‘Documents’ in the current working directory.

Relative paths do not have a leading / and are relative to the current working directory. Full paths start with a / and are based in the file system root. Full paths are also called ‘absolute’ paths.

It is surprisingly easy to lose track of the current working directory. You can ask the shell for the current working directory with the pwd command.

Mac:~ armin$ pwd
/Users/armin

Changing Directories

You can change the current working director with the cd command (change directory):

Mac:~ armin$ cd Documents
Mac:Documents armin$ pwd
/Users/armin/Documents

Use the cd command with a relative path Documents and the shell changes it working directory there. The pwd command prints the full path of the working directory.

You can also navigate to an absolute path:

Mac:Documents armin$ cd /Library
Mac:Library armin$ pwd
/Library
Mac:Library armin$ cd /
Mac:/ armin$ pwd
/

You may have noticed that the prompt displays the name of the current directory. This helps you remember your ‘bearings’ in terminal. Terminal on macOS will also display the current working directory in the window title bar.

You can open a second terminal window by selecting ‘New Window with Settings – Basic’ from the ‘New Window’ menu in the ‘Shell’ menu. It is easier to remember the ⌘N (command-N) keystroke. If you prefer you can also open new shells in a new Tab with ⌘T.

The second window or tab will start a second, new bash shell. This shell is entirely separate of the first bash shell. Different shells will have different working directories. They are very much like different windows showing different folder contents in the Finder.

Home Directory

The prompt in the second terminal window will show:

Mac:~ armin$

According to the prompt current working directory is ~?

The tilde ~ is a special character in bash and other shells. It is a shortcut for the user’s home directory or home folder. On macOS a user’s home directory is created in the /Users directory with the user’s short name as the directory name. So if my short name is ‘armin’ my home directory is /Users/armin.
So the ~ in the prompt says the current working directory is my home folder.

Mac:~ armin$ pwd
/Users/armin

Note: It is important that the shell will actually substitute a ~ with the home directory path before executing the command.

You can use the ~ with cd to quickly change the working directory to your home directory

$ cd ~
$ pwd
/Users/armin

You don’t have to use this, though, since cd without any arguments, will change to your home directory.

$ cd
$ pwd
/Users/armin

You can also start a path with ~:

$ cd ~/Documents
$ pwd 
/Users/armin/Documents

Since ~ is replaced with an absolute path to the home directory, paths that start with ~ are absolute as well.

Moving On Up

You can change the working directory into subdirectories and you can change to absolute paths.

You also often want to move up one level in the folder hierarchy, to the parent folder of the current folder.

Now you could remember your current location (or recall it with pwd) and cd to the absolute path of the parent manually. However, there is an easier way to do this:

$ cd ~/Documents/
$ pwd
/Users/armin/Documents
$ cd ..
$ pwd
/Users/armin

In bash (and most other shells) two periods .. represent the parent directory of the current working directory.

You can even use .. multiple times:

$ cd ~/Library/Preferences/ByHost
$ cd ../..
$ pwd
/Users/armin/Library

../.. designates the parent directory of the parent. ../../.. goes up three levels, etc.

Note: if you cd / and then cd .. nothing happens.

Moving Back

The shell remembers the last working directory, as well, even if you don’t. You can quickly switch back to the previous working directory with cd - (minus):

$ cd ~/Documents
$ cd /Library/Preferences
$ cd -
/Users/armin/Documents
Documents $ cd -
/Library/Preferences

To assist orientation, cd - will print the full path to the directory it is changing to.

Tab Completion

Typing paths is tedious and error prone. Errors in paths can lead to simple errors and frustration, and sometimes typos can have catastrophic results.

When you are typing commands, file names or paths in bash, then you can use the tab key ⇥ to save keystrokes and avoid typos.

For example, you can type:

$ cd ~/Doc⇥
and it will complete to
$ cd ~/Documents/ 

Tab completion will add a forward slash / to the end of a directory’s name so you can continue typing the next directory name. You can hit tab over and over at different parts of the command:

$ cd ~/Li⇥
$ cd ~/Library/
$ cd ~/Library/Appl⇥
$ cd ~/Library/Application\ S<beep>
$ cd ~/Library/Application\ Su⇥
$ cd ~/Library/Application\ Support/

When there are multiple options to complete, bash will complete as far as it is unambiguous and will play an alarm sound. When you then press tab for the second time, it will list all options:

$ cd ~/D⇥<beep>⇥
Desktop/   Documents/ Downloads/

Using tab-completion not only saves keystrokes and time, but also reduces the potential for typos and errors.

You may have noted that tab completion did something strange to the space in the path of the ‘Application Support’ folder. There are many special characters that you have to deal with in the shell. Space is one of them. We will learn how to deal with space and the other special characters in the next section.

Note: commands, arguments and file names in bash are case-sensitive. However, since the macOS filesystems (HFS+ and APFS) are case-insensitive by default, you may want to switch tab-completion to be case-insensitive as well , too.

Listing Directory Contents

The next command is ls, which is short for ‘list’.

$ cd ~
$ ls
Desktop     Downloads   Movies      Pictures
Documents  Library     Music       Public

This command will list the contents of the current working directory. The contents of your home directory may be different.

To use the space efficiently ls prints the files and folders in multiple columns (the number of columns depends on the size of your Terminal window).

This simple list can be very useful to lookup file and directory names. However, you cannot tell the difference between files and directories in this list.

You can tell ls to show an extra character to indicate the kind of an entry:
$ ls -F
Desktop/ Downloads/ Movies/ Pictures/
Documents/ Library/ Music/ Public/

This version of ls output will add a slash ‘/’ to directories and an asterisk ‘*’ to executable files. Normal files (documents) will have no extra character.

In this command the -F is an option for the ls command. Options are a special kind of argument. Options usually start with one or two a hyphen characters - and are followed by a letter. Generally options work like switches to turn certain features of the command on or off.

Since the shell is case-sensitive the options -F and -f are different.

Since ls is a very common command, it has many options. The -G option works similar to the -F option but marks the different kinds with colors:

You can use multiple options at once:

$ ls -G -F
Desktop/   Downloads/ Movies/    Pictures/
Documents/ Library/   Music/     Public/

With most commands you can combine multiple options after a single hyphen: ls -GF is the same as ls -G -F

You can also add an argument to ls:

$ ls ~/Desktop

will list the contents of the Desktop directory without changing the working directory. You can combine options and an argument. When you do that you usually put the options before the argument.

$ ls -GF ~/Desktop

Detailed List

The default output of ls is very terse. You can get more information by adding the -l (lower-case L) option.

$ ls -l
total 0
drwx------+  6 armin  staff   204 Jul 18 17:22 Desktop
drwx------+  3 armin  staff   102 Jun  6 11:24 Documents
drwx------+  3 armin  staff   102 Jun  6 11:24 Downloads
drwx------@ 50 armin  staff  1700 Jul 18 16:02 Library
drwx------+  3 armin  staff   102 Jun  6 11:24 Movies
drwx------+  3 armin  staff   102 Jun  6 11:24 Music
drwx------+  3 armin  staff   102 Jun  6 11:24 Pictures
drwxr-xr-x+  5 armin  staff   170 Jun  6 11:24 Public

This command lists one file or directory per line. The columns are (in order):

  • file type and mode
    • first character shows type: (d directory, l link, – file)
  • number of links
  • file owner
  • file group
  • file size in bytes
  • modification date and time
  • name

There are more options that you can use with the -l option. For example the -h option will show file sizes with B (bytes), K (kilobytes), M (megabytes) etc. instead of raw bytes.
You can also combine -l with -G and/or -F.

Invisible Files

Certain files and directories that are invisible in Finder are visible in the shell. The ~/Library folder has been hidden in the Finder since Mac OS X 10.7 Lion, but is easily visible with ls. By convention however, ls does not usually list files and directories that start with a dot or period .. You can make ls show these files with the -a option. Your home directory will already have a few ‘dot-files’:

$ ls -al ~
total 16
drwxr-xr-x+ 15 armin  staff   510 Jul 18 15:30 .
drwxr-xr-x   7 root   admin   238 Jul 19 08:35 ..
drwx------   4 armin  staff   136 Jul 19 14:55 .Trash
-rw-------   1 armin  staff   402 Jul 19 08:39 .bash_history
drwx------   3 armin  staff   102 Jul 13 09:16 .ssh
drwx------+  8 armin  staff   272 Jul 19 14:57 Desktop
drwx------+  3 armin  staff   102 Jun  6 11:24 Documents
drwx------+  3 armin  staff   102 Jun  6 11:24 Downloads
drwx------@ 50 armin  staff  1700 Jul 18 16:02 Library
drwx------+  3 armin  staff   102 Jun  6 11:24 Movies
drwx------+  3 armin  staff   102 Jun  6 11:24 Music
drwx------+  3 armin  staff   102 Jun  6 11:24 Pictures
drwxr-xr-x+  5 armin  staff   170 Jun  6 11:24 Public

In UNIX files and directories that start with a period are commonly used to store configuration data.
The first two entries in this list are . and ... We already know that .. represents the parent directory (in this case of the directory listed). The single period . is another convention that represents the current directory or (in this case the directory listed). This can be useful to see the file mode and owner of these directories right here.

Note: Finder also follows this convention and hides ‘dotfiles’ by default. You can learn more about hidden files and folders in this article.

Summary

So far we have encountered three commands to navigate the file system: pwd, cd, and ls

pwd prints the current working directory.

$ pwd
/Users/armin/Documents

cd changes the working directory to another.

You can use absolute paths (that begin with /):

$ cd /Library/Application\ Support
$ cd /

or paths relative to the current working directory (no leading /):

$ cd Documents

Two periods .. represent the parent directory:

$ cd ..

changes the working directory to the parent directory.

The tilde ~ represents your home directory. You can use it alone or to start a path to folders and files in your home directory.

$ cd ~
$ cd ~/Documents

The ls command lists the contents of the current working directory:

$ ls
Desktop        Downloads   Movies      Pictures
Documents  Library     Music       Public

The ls command has many options. The most commonly used are probably -l to show a detailed list of the files and folders and -a to also show the files and directories starting with a period, which are usually hidden.

Terminal Primer – Part 1 – First Steps in Terminal

It is vacation time here in the Scripting OS X headquarters and I will be traveling with family for most of August. The Weekly News Summaries are also on vacation, but to compensate for that I will be posting a draft of parts of my next book on macOS Terminal and bash. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

  • First Steps in Terminal (this post)
  • Navigating the File System
  • Special Characters
  • Commands
  • Managing Files

Terminal Application

On macOS you can find the Terminal application in /Applications/Utilities/. There is a shortcut in Finder’s Go menu to get to the Utilities folder or use the keystroke ⌘⇧U. Terminal has a distinct black icon with a white prompt.

However, since you are reading this book, you are planning to use Terminal regularly. In this case, you really want to add the Terminal application to your Dock.

First Prompt

Once you have opened Terminal, it will show you a new window, which is white with black text by default. A short message will show the last login and then a line with the default prompt:

Mac:~ armin$

At first it shows the computer name (as set in the ‘Sharing’ pane in System Preferences. After the colon ‘:’ it shows a tilde ‘~’. Then your user short name, followed by a dollar sign ‘$’. Finally a dark gray block, the cursor, waiting for your input.

Note: if you have used Terminal before, you may have changed the default look of the Terminal and prompt. The functionality remains the same, no matter how it looks.

You can enter your first command. Type the letters p, w and d and then hit the return key ↩︎

Mac:~ armin$ pwd
/Users/armin

You have typed the pwd command and executed it with the return key ↩. The return key confirms a command and runs it.

When you mistype a command you can delete the last character with the delete or backspace key ←. You can also move the cursor through the characters you entered with the left and right arrow keys.

When you hit return with a wrong command or a command with a typo the shell will complain that it cannot find the command:

Mac:~ armin$ pws
-bash: pws: command not found

Terminal Terminology

We have already encountered a few different terms which may be very confusing in the beginning. The bad news is there are more terms, the good news is that it will all make sense eventually.

All of these terms date back to the days when computing started with big mainframe computers. Since computers were scarce, complex and expensive, many users had to share access. They did this by typing and reading on a ‘terminal’ or ‘console’ — a dedicated device talking to the main frame. Terminals and consoles may have looked somewhat like like modern workstations and computers, but did not have their own CPU. They would just allow to enter and send commands and receive and display the results.

Very early in the history of computing, terminals used mechanical printers or typewriters to show the results, these were called tele-typewriters or tty. The protocol that the mainframe used to communicate with the typewriters, was named tty as well. The protocol and its name have remained, long after the mechanical typewriters are gone. Later the mechanical interfaces where replaced by terminals with electric keyboards and cathode ray screens.

As computers got smaller, more powerful and cheaper it became possible to have one on everybody’s desk. Even so, it was (and is) still convenient to run a shell, either locally or to connect to larger mainframes (servers). To do that you would open a program (or application) that ran a ‘virtual terminal.’

The Terminal application on macOS is such a virtual terminal program. To confuse things a little, macOS also has an application called Console. The Console application on macOS is not a virtual terminal or virtual console, but used to display and filter log files.

The terminal (whether virtual or real) only provides a means (virtual or mechanical) to enter and display text. It will display a ‘prompt’ which tells the user, that the system is ready and the user can enter a command, and a ‘cursor’ which shows where typed text will be displayed or inserted.

Shells

There is another program which interprets the text, executes code and sends the output generated back to the terminal.

This program (or class of programs) is called a ‘shell.’ A shell protects the user from the dangerous, complicated parts of the system and abstracts differences from one system to the other. Another way of looking at it, is that a shell protects the vulnerable, fragile parts of the the system from the user.

There are many different shell programs. One of the earliest shell programs was call just ‘shell’ or sh for short. Surprisingly, sh is still around after nearly fifty years.

On macOS the default shell is called bash for ‘bourne again shell.’ This comes from the fact that it was developed as a replacement for bsh or ‘Bourne shell,’ which was named after its main developer Stephen Bourne.

You will notice that unix shells and commands often exhibit a particular style of pun humor. ‘bash’ is merely the beginning.

Today, there are many different shells. Different shells have different ways of interpreting commands. The choice of shell is a personal preference and can be (as many things in computing) the cause of passionate argument.

Note: As system administrators, the choice of shell is not just determined by preference, but also practicality. There are a set of pre-installed shells on macOS and while it is possible to install additional shells, it increases complexity and management effort. Also when you are sharing commands and scripts with fellow administrators, bash is the commonly agreed upon shell.

bash is available for most platforms and operating systems and is the default shell on many systems. Because of this prevalence bash is a good choice for your first shell.
bash has been the default shell for Mac since Mac OS X 10.3 (Panther). It is also the default shell on most Linux distributions and was the choice for the Unix command shell on Windows 10.

Note: the bash that ships with macOS is version 3.2.x. There is a newer major version: bash4. However, since bash4 is licensed as GPLv3, Apple still only includes the older bash 3.2. You can download and install bash4 if you want to, but many system administrators stick with the pre-installed version. We will be covering bash 3.2 here.

If you are curious, you can list all available shells on macOS with the following command in Terminal:

$ cat /etc/shells
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/bash
/bin/csh
/bin/ksh
/bin/sh
/bin/tcsh
/bin/zsh

Warning: When entering commands, you have to watch that you type the command exactly as given, including spaces and other special characters. Terminal and shells are a ‘pro’ tool and assume that you know what you are doing. They are not forgiving to errors.

There are a few mechanisms that will usually protect you from ruining your system and data (and thus your day) entirely, but you need to always be careful and check.

Note: As we saw earlier the default prompt shows more information. However, this information (computer name, user name) is different for every Mac and user.
To keep things simple and short, when displaying terminal input and output, we will not show the entire prompt, but only the final character ‘$’. This designates the commands that you should enter in your terminal, without the ‘$’.
Subsequent lines without the leading ‘$’ show output that you should expect from this command. Sometimes the output in this book will be abbreviated to make it fit the layout.

Technically, ‘graphical user interfaces’ (GUI) which display files, folders and programs as icons in windows such as macOS, Windows, KDE, Gnome and even iOS and Android are also shells, since they shield the system internals from the user and vice versa.

However, usually the term ‘shell’ is used for interfaces where the user has to type commands, or ‘command line interfaces’ (CLI).

Shell Scripts

Command Line shells commonly have two major roles. The first is to interpret and execute commands entered in an interactive prompt and deliver the results back to the user. The second role is to interpret and process list of commands, called scripts.

While scripts basically use the same set of commands as the interactive shell, scripts can also use control statements, loops, and variables which makes them a related task, but much more complex.
In this book we will focus on the interactive part of bash.

Next Post: Navigating the File System

Vacation Time, Travel Time, Sneak Preview Time

It is vacation time here in the Scripting OS X headquarters and I will be enjoying time off with family for most of August.

Tomorrow’s weekly news summary will be the last until the end of August. I will keep track of interesting posts, and provide a big summary at the end of the month, though I will also be offline and lot probably miss something interesting. If you find any post or link you think is worthy, you can send to it me on Twitter or the MacAdmins forum (also scriptingosx) and I will make sure to include it.

However, I did not want to go entirely dark during the month. To keep you interested, I will post a few sections from the book I am currently working on over the next week. They are still a bit rough and unfinished.

These sections are from a chapter called ‘Terminal Primer’. They are much more basic than the series of Terminal tips and tricks I ran a few weeks ago. They are targeted to Mac Admins who are new to Terminal and bash. The book this chapter is from, will contain this primer, but also sections on how to use Terminal and bash effectively as a Mac Admin.

Yes, I know, I have promised a book on autopkg. Which I am also still working on. However, while writing the autopkg book I realized that Terminal and bash skills are fundamental for Mac Admins and I feel I need to get it out of the way first.

Now you may ask, “Aren’t there other skills you need to use autopkg effectively, such as property lists, Python, git and Github?” and I will answer, “I know, right!?”

The Terminal Primer sections will be posted (automatically scheduled) on this blog over the next few weeks. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

Terminal Primer

  • First Steps in Terminal
  • Navigating the File System
  • Special Characters
  • Commands
  • Managing Files

Have a great August!

Terminal and SSH Apps for macOS and iOS

This is an addendum to my series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

So far we have used ssh entirely from macOS’ built-in Terminal application. In most cases Terminal is entirely useful and sufficient. However, there are some really useful terminal/ssh applications from third parties. There are also applications for iOS you can use to connect to other computers with ssh.

(iOS and Mac AppStore links are affiliate links.)

Panic’s Prompt 2

Prompt 2 (vendor page) is my favorite iOS application to connect to a shell. (Together with Edovia’s Screens, it forms the backbone of my admin toolkit on iOS.)

Prompt supports external keyboards and keyboard shortcuts. It also supports key based authentication and agent forwarding. It can optionally sync server bookmarks, clips and credentials through the Panic Sync cloud server. Prompt can also detect SSH hosts in the local network that advertise via Bonjour.

Prompt 2 is a one time-purchase for iPhone and iPad, available at the AppStore.

Termius

Termius is a cross-platform SSH solution. It has apps for iOS, macOS, Android, Linux and Windows, so if you need to move across many of these platforms it may be very useful. (I have only tested iOS and macOS.) It is free for basic use, but you can unlock ‘premium’ features for a subscription fee. The premium features include a bookmark and key sync service, clippings, sftp and agent-forwarding.

It has some interesting features such as port forwarding and a the ability to send a public key to a server (like ssh-copy-id) on iOS.

iTerm2

iTerm2 is an alternative terminal application for macOS. It has many features that Terminal.app is lacking or added much later than iTerm2.

Cathode

Cathode is an alternative Terminal application for macOS and SSH client for iOS. Its main claim to fame is to emulate the appearance of old-style cathode ray displays, including 8-bit fonts, distortion and flicker. Cathode is a one-time purchase for macOS and iOS.

Mosh and Blink

mosh (mobile shell) is a protocol for remote shells that is designed for modern mobile connections. mosh can maintain connection to a server even when the client’s ip address and/or connection method changes (i.e. a device switches from Wi-Fi to cell data or vice versa). It uses ssh to establish the connection and security, but then uses its own connection. So you can use ssh keys to authenticate. However mosh requires a server component to be installed on the host.

The iOS mosh client is called Blink and commercially available on the AppStore. Though, since the project is open source, you could also compile the client yourself.

SSH Tunnels

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

We have learned so far that ssh is a really useful and flexible protocol. It can be used to connect securely to a remote shell, or to transfer files securely.

Rather than providing the shell itself, ssh provides a secure way to transmit data to and from the remote shell. In a similar way, ssh can be used to provide access to other remote services as well.

SSH Tunnels with Two Computers

Access to important services are usually blocked behind a firewall or router. Since ssh, when setup correctly, is quite secure, you can usually get access to a server with ssh even when other protocols are blocked. (Though some administrators move ssh access to a different port than the default 22.)

You can use ssh port forwarding or ‘tunneling’ to gain access to other services through ssh.

Imagine you want to use Screen Sharing to connect to a remote Mac (remote.example.com). Screen Sharing on macOS uses the VNC port 5900 to connect to a remote Mac. Since VNC itself is inherently insecure, (mac Screen Sharing adds a few things to make it more secure) this port is blocked by many firewalls.

However, I do have ssh access to remote.example.com. So, how do I tell both systems to ‘tunnel’ the screen sharing traffic through ssh?

(When you test this, remember to enable either ‘Screen Sharing’ or ‘Remote Management’ (i.e Apple Remote Desktop) access in the ‘Sharing’ pane in System Preferences on the remote Mac.)

The tunnel starts on my local machine and ends on remote.example.com at port 5900 (where the screen sharing service is listening on the remote Mac.)

The starting point also needs a port number, and I can basically choose freely. Port numbers under 1000 and over 49000 are reserved for the system and require root privileges. There are also many numbers that are commonly used by certain services (such as 5900 for VNC/Screen Sharing) and may already be in use. I will choose 5901 for the local port.

To connect the local port 5901 to port 5900 on the remote Mac use the following command:

$ ssh -N -L localhost:5901:localhost:5900 remote.example.com

(You can just try this with a second Mac or virtual machine in your network, even without a firewall.)

The syntax of this command is less than obvious. Let’s break it into pieces:

The -N option tells ssh that we do not want to invoke a remote shell or run a remote command.

The -L option creates a local port forwarding setup. This option takes a parameter with three or four parts, separated by colons :. The first pair (localhost:5901) are the tunnel start point. The second pair (localhost:5900) are the remote end point of the tunnel.

The second localhost is resolved on the remote host, so this means port 5900 on the remote host.

The last parameter states the remote host, to connect to, remote.example.com.

This commands tell ssh to connect to remote.example.com and establish a tunnel that transfers traffic from port 5901 on my computer to port 5900 on the remote computer.

Since the origin of my tunnel is usually on my local computer, the first localhost can be omitted, so you only see the origin port.

$ ssh -N -L 5901:localhost:5900 remote.example.com

When you execute the command nothing will happen. You will not even get a new prompt, because the ssh process is running until you cancel it with ctrl-C. Don’t cancel it yet, however, since it needs to run to provide the tunnel.

So, when you open the Screen Sharing application (from /System/Library/CoreServices/Applications/) and connect to localhost:5901 all traffic will be forwarded by sshto port 5900 on the remote Mac.

You can also use the open command to connect with Screen Sharing:

$ open vnc://localhost:5901

You should be able connect with Screen Sharing, even when port 5900 is blocked by a Firewall.

When you are done with the Screen Sharing session, you can end the ssh tunnel process in Terminal with ctrl-C.

SSH triangle

You can also use ssh port to use the remote host as a gateway or ‘jump host’ to a third computer. Imagine you want to use Screen Sharing to connect to secundus.example.com behind a firewall and you only have ssh connection to primus.example.com available. You can tell primus to point the endpoint of an ssh tunnel at secundus with:

$ ssh -N -L 5902:secundus.example.com:5900 primus.example.com

Note: secundus.example.com or whatever host or IP address you enter there will be resolved on the remote host. So you can use NAT IP addresses or .local host names here, even if they do not make sense in the network your work Mac is in. (They do have to make sense on the remote host, though, otherwise you will get an error.)

In the following examples the local IP address 192.168.1.200 or the Bonjour hostname Secundus.local will be resolved on the remote host, even if they don’t work on my local computer:

$ ssh -N -L 5902:192.168.1.200:5900 primus.example.com
$ ssh -N -L 5902:Secundus.local:5900 primus.example.com

Either way, you can then point Screen Sharing at localhost:5902 and it will connect through primus to Screen Sharing on secundus.

Keep in mind, that while the connection from the start point (on your Mac) to the host primus is secured by ssh the connection from primus to secundus is not.

Stumbling over HTTP hosts

In general you can use ssh port forwarding (or tunnels) for any service. Some services however, may introduce extra pitfalls.

For example, I wanted to use ssh port forwarding to gain access to my home router’s web interface. I can use ‘Back to My Mac’ to ssh into one of the iMacs at home, and thought it should be easy to connect to the router with an ssh tunnel:

$ ssh -N -L 8080:192.168.1.1:80 mac.example.com

This seemed to work, but whenever I tried to point a browser to localhost:8080 it couldn’t connect to the web page. The problem here is not the ssh tunnel but the the web server on the router. As part of the http request, the browser sends the name of the domain requested to the web server. This allows web servers to host different pages for different domains. With this request, the browser told the router it wanted the web page for localhost and the router replied with “I don’t serve pages for that host”… (Your router might behave differently.)

With curl I could convince the router to serve me the page with:

curl -H "Host: 192.168.1.1" http://localhost:8080

However, since navigating the web interface of the router with curl is out of the question I had to find a different solution.

Tunnel All the Things!

What if I could send all traffic through the iMac at home?

With the command

$ ssh -N -D 9001 remote.example.com 

I can create a tunnel from my computer (on port 9001) to the remote Mac that acts as a SOCKS proxy. Then I can set the Socks proxy to localhost:9001 in the proxy tab in the Network pane in System Preferences. You probably want to create a new network location for this setup. Then all network traffic will be securely routed through the ssh tunnel to my Mac at home where it can connect to the router.

This can also serve as a temporary VPN solution.

However it is somewhat painful to set up and maintain, so if you start using this more frequently, you probably need to look into a proper VPN service solution (some routers, ironically, provide one…).

Summary

  • you can bypass firewalls and other network blocks by tunnel traffic for any service through a ssh tunnel
  • the command describes which local port to connect to which port on the remote host
  • you can even tell the remote host to connect the end point to a third computer, behind the firewall
  • you can also create a SOCKS proxy with ssh to tunnel all traffic

Previous Post: Transferring Files with ssh

Transferring Files with SSH

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

In the previous posts we looked how to connect with ssh to a remote computer (host) and how to setup the keys necessary for a secure connection.

Despite the name ssh does not actually provide a shell or command line interface to the host itself. it ‘merely’ provides a secure connection to connect to the default shell on the host itself.

Even this basic use of ssh is already very useful and powerful. It allows to open one or more full command line sessions to remote computers as if we sat at their keyboard. You can also send individual commands and receive and process the results.

However, ssh has a few more powerful tools available.

Copy Files Remotely

You can also use the ssh connection to copy files to and from a remote host.

The command you use for this is scp (secure copy) and it use the same basic syntax as the cp command

$ scp source destination

But, since scp can copy from the local computer to a remote host or vice versa, you usually add a bit more information:

$ scp [[user@]host1:]source [[user@]host2:]destination

(The examples will use a file name hello.txt. To create one quickly, simply type echo "Hello SSH" > hello.txt in Terminal.)
To simplify this, a few examples:

$ scp hello.txt mac.example.com:"~/Documents/"
hello.txt                                     100%   10    10.8KB/s   00:00    

This will copy the local file sample.txt from the current working directory to the remote host’s ~/Documents/ directory. scp will show an ascii progress bar for every file copied. (though with these small files, you will not see much of it) You can suppress the progress display with the -q (quite) option.

For the destination, the colon : separates the hostname (DNS) from the file path.

This command will prompt for the user’s password on the remote host, unless you have added your public key to the remote host’s authorized_keys file.

Since no user name is given before the hostname (separated with an @) scp uses the username that you are logged in with on the local computer. If the remote user has a different name, use:

$ scp hello.txt user@mac.example.com:"~/Documents/"

We do not want the local shell to evaluate the ~ to the local home directory, but want the remote computer to evaluate ~ to the remote user’s home directory, so we have to quote the remote path.

Like cp, when the source is a file and the destination is a directory, then the file will be placed into the destination directory.

If the remote path does not exist, then scp will present an error:

$ scp hello.txt mac.example.com:"~/DoesNotExist/"
scp: /Users/armin/DoesNotExist/: No such file or directory

You can also rename the file while copying:

$ scp hello.txt mac.example.com:"~/Documents/ssh_test.txt"

You can copy files from the remote host to your local machine:

$ scp mac.example.com:"~/Documents/ssh_test.txt" .

In this case we passed . or the current working directory as the destination. You can also pass a local path:

$ scp mac.example.com:"~/Documents/ssh_test.txt" ~/Downloads

Use the -r option to copy all the contents of a directory:

$ scp -r ~/Projects/greatproject/ mac.example.com:"~/Documents/"

Remote-to-remote Copies

You can copy from one remote host to another.

There are two solutions for this. The first will copy the file to the local computer and then back up to the other remote host. You invoke this version of remote-to-remote with the -3 option.

$ scp -3 primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

(I am shortening the full domain names from primus.example.com and secundus.example.com to primus and secundus for simplicity.)

Under most circumstances copying a file down to your Mac and then back up to the other remote host is less than ideal. Imagine you are working from home with your laptop and want to copy a large file from one server at work to another.

The other option is to tell the source remote host to scp the to the other remote host. You could achieve this by sshing to the remote machine and running scp from there. Thankfully, scp is smart enough to attempt exactly that when you type

$ scp primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

This command will probably fail right now. It requires a few things to be set up to work:

  • either: client key authentication to be setup from primus to secundus
  • or: client key authentication from your local computer to primus and secundus with agent forwarding enabled

The first option is fairly easy to understand. scp will connect to primus and authenticate with your local client key. Then it will tell primus to connect to secundus, authenticating using primus’ client key and the copy the file. It basically works as if you sshed in to primus and ran scp without the extra typing.

There are drawbacks to this. If ssh-agent is not running on primus and does not have the passphrase stored yet, then primus cannot unlock its private key and authenticate to secundus.

Also you basically need to prepare all remote hosts to have keys exchanged between each other, which can be painful, if not impossible to manage.

Agent Forwarding

The second option, called ‘Agent Forwarding’, circumvents these problems. This will tell the first remote host (primus) to ask your local ssh-agent for a key to authenticate to secundus. The system does not actually transfer the private key, but asks ssh-agent on your local computer to encode the authentication challenge for primus.

That way you only need to manage the keys for all remote hosts on your local computer.

You can also use agent forwarding with normal ssh connections. It is not enabled by default, but you can enable it for an ssh session with the -o ForwardAgent=yes option:

$ ssh -o ForwardAgent=yes primus

Since this is hard to type, there is a shortcut:

$ ssh -A primus

When you are logged in to the remote host with agent forwarding enabled, you can then ssh from there to another remote host (secundus) and it will try to use the keys from your local computer’s ssh-agent to authenticate.

$ ssh -A primus
Last login: ...
primus$ ssh secundus
Last login: ...
secundus$ 

This can be a useful strategy if direct access to the second remote host (other.mac.com) is blocked with firewalls.

You can also use this for the scp remote-to-remote copy. Unfortunately, scp does not have a convenient -A option, so you have to use the long parameter form:

$ scp -o ForwardAgent=yes mac.example.com:"~/Documents/ssh_test.txt" other.example.com:"~/Documents/"

When you need agent forwarding regularly for a specific host, you can enable it by default for this particular host in your ~/.ssh/config file. Add the following lines:

Host primus.example.com
    ForwardAgent yes

Note: There are some security concerns with agent forwarding. A user with root access on the intermediate system can gain access to the connection to your local ssh-agent, thus gaining the ability to encrypt with your private keys. Be aware of this in security sensitive environments.

SFTP

If you have many files in a complex file structure , scp can be a little cumbersome. There is a special interactive mode that you can invoke with the sftp command (secure file transfer program).

$ sftp mac.example.com

Note: sftp(according to its man page) is ‘similar’ to ftp but not identical. It also should not be confused with ftps which is ftp over SSL.

Obviously, if you have key authentication setup, sftp will use that.

There are many interactive commands to navigate the local and remote file system and upload (put) and download (get) files. You can look at the details in the sftp man page. However, if you need to use sftp frequently, then you should use a graphical sftp application. There are a large number of SFTP client for macOS and iOS. Here is a list of some popular clients: (AppStore links are affiliate links.)

All of these tools connect to many other server protocols other than sftp. However, the advantage of sftp is not just the built-in security, but that you don’t need other software than sshd running on the server.

Summary

Previous Post: Client Verification

  • you can use scp [[user@]host:]source [[user@]host:]destination to copy files from or to a remote host over ssh
  • you can use agent forwarding to simplify key management in triangle setups
  • sftp help managing/transferring multiple files over ssh, there are many UI applications

Next Post: SSH Tunnels

Minimal Terminal Prompt

You may have noticed, I use Terminal a lot. As do many other Apple administrators. Personally, I like the Terminal to be non-flashy and out of the way with as little embellishment as possible.

That does not mean I don’t modify the standard Terminal settings. Over time I have actually modified them quite a lot and I thought it might be useful to share, what, how and why I changed them.

Of course the desire for minimalism is an entirely personal taste, but you can adapt these instructions to fit your tastes as well.

Window Settings

I use the ‘Basic’ window profile setting from Terminal app (Terminal > Preferences > Profiles) with a few minor changes. I use this trick to have randomly rotating light background colors. That way it is easier to find a specific session.

Colorful Background Windows

I have also changed the Terminal font to my monospaced favorite: Source Code Pro, 14pt.

Prompt

The default prompt for macOS is quite elaborate:

Calypso:~ armin$

It shows the computer name, the current working directory name and the user name. As I described in this post on environment variables, you can change the prompt with the PS1 environment variable. I like to have a short prompt:

export PS1="\W \$ "

This will only display the current working directory name \W and the prompt \$ (usually $ but # when sudoed to root). I have also added some control codes to color the directory dark gray and bold for some visual contrast. These codes are unfortunately quite illegible:

export PS1="\[\e[1;30m\]\W\[\e[m\] \\$ "
Minimal Terminal Prompt

Different Prompt for ssh

This works great when logging in locally. However, I want to be able to tell the difference between shell windows on the local computer and those that are sshed to a remote computer. You can check for that in your .bash_profile or .bashrc. The ssh process sets the SSH_CLIENT environment variable:

if [[ -z "$SSH_CLIENT" ]]; then
    # local connection, change prompt
    export PS1="\[\e[1;30m\]\W\[\e[m\] \\$ "
else
    # ssh connection, print hostname and os version
    echo "Welcome to $(scutil --get ComputerName) ($(sw_vers -productVersion))"
fi

So if the SSH_CLIENT variable is _un_set (if [[ -z) then the prompt is changed to the short, colored prompt. Otherwise the default prompt, which shows the computer name and user as well, remains and I print a welcome message with the computer name (scutil --get ComputerName) and the macOS version (sw_vers -productVersion).

Of course, the .bash_profile will have be installed on the remote Mac as well for this to work there.

You can test this by using ssh to connect to localhost:

The prompt is different when connected with SSH

Suppressing the ‘Last Login’

I also suppress the ‘Last Login: …’ message you get on every new session. You can do this by creating the file ~/.hushlogin. This message can of course serve as a security measure, but only when you actually pay attention it.

The .hushlogin file will also suppress the display of a ‘Motto of the Day,’ if present. (The contents of the file /etc/motd.

If you still want to know when your account last logged in, you can use the last command for a long list of logins or this command for just your last login:

$ last -1 $(whomai)

The Result

The minimal Terminal Window — Aaahhh!

SSH Keys, Part 2: Client Verification

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

In the previous post we discussed how ssh uses public and private keys to secure the data sent back and forth, between the remote computer (host) and your machine (client). The keys allow you to be sure that you are talking to the correct host and not an imposter.

However, how does the host know that you are who you claim to be?

Open Sesame…

The default way to authenticate yourself to the ssh host is to give it the correct username and password combination. This is and has been a common practice for computers, servers, websites for a long time and it works for ssh as well.

You will obviously need an account on the host for this to work. In larger organizations the computers will often be connected or bound to a directory service, to centralize the user and group management. If the computers are not bound to a central directory, as is often the case with mobile computers, you can use your management system to install one or more managed users on the system.

In the Sharing Preference Pane, you can control which users or groups of users can use ssh or ‘Remote Login’ to connect to this Mac. Behind the scenes the users and groups that can connect with ssh are in the com.apple.access_ssh group. You can find some scripts to manage the controls in this post.

Keys for the Client

Passwords, and user access control groups already provide a strong security, but have some disadvantages as well:

  • you have to re-enter the password on every connection. This is especially tedious when you send individual commands with the ssh hostname command syntax
  • automation is insecure because you have to provide the password in the script (in clear text)

With ssh we can use the same asymmetric keys to validate the client to the host, as we did to verify the host.

Sidenote: you will see a lot of actual keys and fingerprints in this post. While sharing the public key should not be a security issue, I deleted them all after finishing this post, just to be safe.

Creating a Key

First, you need to generate the key for your user. This is simple enough with ssh-keygen:

$ ssh-keygen -t rsa -b 2048 -C "Armin RSA 2017-07-06"
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/armin/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/armin/.ssh/id_rsa.
Your public key has been saved in /Users/armin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:jRydlFk3rRTpuKshFvnzon7UnzcYiHLQhKpQvLqPrl0 Armin RSA 2017-07-06
The key's randomart image is:
+---[RSA 2048]----+
|  .     . .+. ++ |
|   o   . +o. .o..|
|  . . . + o  + . |
| . . . o *  . o  |
|  o .   S o...   |
| . .   . =..o.   |
|  . E   =.+  oo. |
| o..   . .o+..o..|
|+oo.   .oo.o.  ..|
+----[SHA256]-----+

This command will create a new key pair of type ‘rsa’ (-t rsa) with a bit length of 2048 (-b 2048) and a comment of Armin RSA 2017-07-06.

RSA is the recommended type for best cross platform compatibility. Mac hosts and clients also support ECDSA, and ED25519. (DSA is in the list but considered insecure and obsolete.) When you connect to other, non-macOS, hosts, ask the administrators for the best key type to use.

The bit length of the key determines how complex the key is. Longer bit lengths are more secure, but require more CPU power for the encryption and decryption. Different key types can have different values for the bit length, read the ssh-keygen man page for details.

The comment is optional, but provides a way to add a label to a key which will help you identify it later. I usually add who created and when and if the key is for a specific purpose.

Next the ssh-keygen tool will ask where to store the key pair. Since this will be our ‘generic’ key, the default name and location ~/.ssh/id_rsa (or id_ecdsa, etc.) is exactly what we want. Sometimes you may have to create a key specifically for a single host, then you can choose a different name here, so you do not overwrite your other keys.

Next the tool will ask you for a passphrase. This will be used to encrypt (lock) the private key, so even if someone gets their hands on the private key file, they will not be able to use it. Since the private key represents your ssh identity it should be protected well. Use a password generator to create a long passphrase. Store the passphrase in a secure place (Password manager, you will need it again) and enter it here (twice).

Finally, the tool will show the files it generated (the private and public key), its fingerprint (with the comment) and a ascii art representation of the key, which is supposed to be more visually recognizable than the alphanumeric fingerprint.

Copying the Key to the Host

When we set up an ssh connection to a host we need to copy its public key into our ssh settings (the known_hosts file). Similarly we need to copy our public key to the host.

You have to copy the public key (i.e. id_rsa.pub) to the host and append its contents to the ~/.ssh/authorized_keys on the host in your user home directory. On some systems, the file may also be called authorized_keys2 and/or be in a different location.

On Mac OS X versions 10.11 and older you can use this command:

$ cat ~/.ssh/id_rsa.pub | ssh [user@]mac.example.com "cat >> ~/.ssh/authorized_keys"

This will pipe the contents of your public key file into ssh, which sends it to the remote host and appends (>>) it to the ~/.ssh/authorized_keys file. (This will fail if the ~/.ssh directory does not exist yet on the host.)

On many Linux systems and on macOS Sierra 10.12 there is a tool to help with this. You can use

$ ssh-copy-id [user@]mac.example.com

By default, the ssh-copy-id command will copy all available public keys to the host. You can tell it to use a specific key with the -i option. Either way, since the key is not yet in place, the command will need to provide you with the host’s password to authenticate to the host.

The ssh-copy-id command gives helpful additional information:

$ ssh-copy-id mac.example.com
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/armin/.ssh/id_ecdsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password:

Number of key(s) added:        1

Now try logging into the machine, with:   "ssh 'mac.example.com'"
and check to make sure that only the key(s) you wanted were added.

Now, when you attempt to connect to the remote host, you will be prompted to unlock your private key:

$ ssh mac.example.com
Enter passphrase for key '/Users/armin/.ssh/id_ecdsa': 
Last login: Thu Jul  6 09:00:28 2017 from [IP Address]
mac:~ armin$

and when you look at the contents of ~/.ssh/authorized_keys you will see your public key listed:

mac:~ armin$ more .ssh/authorized_keys 
ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAC3TR7ZYSjJO01JVNYMeXWNNgglNulQVuGGTnE2JiBULbs9In9uWrFWyqOXodi4XCs7lY92undONnN4wQKumsN/RgBnEDATKZZdh5OPI72ZGrmnPNXVUbU8mjzIJ4LpWfTu0x6nqR9zBu1LPNW9xPoQNwTogh7jIHxvEcYav1slKjeRkg== Armin ECDSA Key 2017-07-06

Note that the comment we added when creating the key is part of the public key (and its fingerprint). Considering that this file can contain multiple keys from different sources, the comment will help a lot in locating the proper key.

When a key is compromised or just not used any more, it should be removed from the host’s authorized_keys file. This file should be audited frequently.

Agent of Keys

When we connected to the remote host after copying the key, we did not have to enter the password for the host, because the key pair confirmed our identity. However, you still had to unlock the private key with the passphrase. You stored that somewhere secure, right? Without the passphrase the key is useless.

This does simplify passwords, because you can use the same key for multiple hosts, and you don’t have to memorize all those passwords. However, entering the passphrase over and over is tedious and error prone, too. Thankfully there is a solution to this. There is a process that can store the passphrase for the duration of your login session. The process is called ssh-agent. However you do not talk directly with this tool but use ssh-add to manage it instead. And on macOS you don’t even have to do that, because ssh-add and ssh-agent are integrated with the macOS Keychain.

The role of the ssh-agent is to hold on to your private key passphrases for the duration of your login session. On other systems (or with Keychain integration disabled) you have to add the passphrase once per login session with

$ ssh-add ~/.ssh/id_rsa

(this will prompt for the passphrase)

You can list the passphrases that ssh-agent is holding with

$ ssh-add -l

and clear all of them with

 $ ssh-add -D

Keychain Integration

Note: on macOS Sierra (10.12), Keychain integration is not enabled by default. To enable it, add these two lines at the top of ~/.ssh/config (create the file if necessary):

AddKeysToAgent yes
UseKeychain yes

With Keychain integration enabled, ssh-agent will also look for passphrases stored in the keychain and then remember them for the rest of the session. The passphrase is stored as an application password with the path to the private key in the name. So as long as the keychain is unlocked, it will find the passphrase in the keychain and use it to unlock the private key (and remember it for the rest of the login session).

On macOS Sierra, the passphrases are not stored in standard login keychain and not synced with iCloud.

Managing Keys with multiple clients

If you have many client Macs you use to connect to the same hosts, you have two choices:

  • you can create a new key pair in each client and add the public key to the host’s authorized_keys file
  • you can copy the key pair from one client to another

The first option can grow unwieldly quickly when you are managing several Macs and several users.

The second option might expose your private key when you copy them from one client to another. If you send the private key unencrypted over the network it can be intercepted by a man-in-the-middle attack. If you copy it on a USB stick to transfer, you should remember to securely delete any copies, after the transfer.

Your organization will (should) have rules on how to setup and manage keys.

Lockdown

Given proper key management, client verification with keys is safer than with username and passwords. Because of this some setups only allow ssh connections an authorized key.

In this case you cannot use ssh-copy-id to transfer and add your public key to the host. You will have to provide the public key to an administrator to add it for you. The authorized_keys file will probably be in a different, central and locked-down location.

Automation with ssh and keys

You can write scripts that use ssh to communicate with another host. This can yield some powerful workflows. However, if the script uses ssh repeatedly, you do not want to have to enter the password over and over. If the script runs within a user session, ssh-agent can provide the passphrase when necessary. But when a script has to run in background on a schedule without user interaction, ssh-agent will not be available.

In this case you can create a key with an empty passphrase. This key will be unlocked automatically. Obviously, such a key would need to be protected very well, and its use should be audited closely. A key with an empty passphrase is the security equivalent of leaving the house key under the doormat. However, for some automated workflows, it may be the best solution.

Summary

  • previous post: ssh uses public/private keys to verify the host and encrypt the connection
  • ssh can also use public/private keys to verify the user connecting to the host
  • you generate a key with ssh-keygen
  • the public key is added to the remote host’s ~/.ssh/authorized_keys
  • the private key remains on the client and is locked with a passphrase
  • ssh-agent and Keychain integration on macOS can simplify access to the private key’s passphrase

Next post: Transferring Files with ssh

SSH Keys, Part 1: Host Verification

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

In the earlier post, we talked about basic ssh setup and use.

We briefly mentioned ssh is securely encrypted. We encountered a cryptic prompt (pun intended) when connecting to the machine for the first time:

The authenticity of host 'mac.example.com (IP address)' can't be established.
ECDSA key fingerprint is SHA256:abcdefghhijklmonpqrstuvwxyz.
Are you sure you want to continue connecting (yes/no)? 

At that point we just wanted to connect and ignored this prompt. However, this is a crucial part of the ssh infrastructure and understanding it will help you use ssh securely and to its full potential.

Asymmetric Key Encryption

This is what ssh uses to encrypt traffic. It uses a two part encryption key to encode the data transmitted. The first part of the key is private and kept securely on the ssh host (the remote computer you want to log in to) and the second part is public and shared to the users who want to connect.

Data encrypted with one key can be decrypted with the other and vice versa. This way data can be kept secure between the client and the host.

When you connect for the first the remote host will send its public key to your computer. This is convenient, but opens the chance for malicious computer to impersonate the host you actually want to reach and set itself as a ‘man in the middle’, passing data back and forth and decrypting it in between.

OK, how does this work with SSH?

The key fingerprint the system displays on first connection is a ‘hash’ of the public key.

Using this fingerprint, you can verify that the DNS address or IP you are connecting to, is really the computer you want to connect to. To verify, you would need obtain the fingerprint from the host though some other means and compare.

On macOS the host keys are stored in /private/etc/ssh/ along with some other files required for ssh configuration. The key filenames have the format ssh_host_ABC_key where ABC is the key encryption type. On my Sierra Mac there are keys for dsa, ecdsa, rsa and ed25519. There two files for every type, the private key (no file extension) and the public key, with the .pub extension.

ECDSA (Elliptic Curve Digital Signature Algorithm – Wikipedia) is the default type of key ssh uses on macOS. The other keys are present for compatibility with other platforms and older versions of macOS/OS X. Usually the client and the server will negotiate which keys to use from the available options.

These keys are generated on every host, when the ssh server process starts for the first time. These keys are unique to this host.

When you look closely at the key files, you will see that the private keys can only be read by root, no other users. Private keys are like really important passwords and have to be kept safe. If someone can obtain both keys, they can impersonate this host.

The public keys are the part that are meant to be shared and can be read by any user.

Fingerprints

You can generate the fingerprint from a public key with the ssh-keygen command:

$ ssh-keygen -l -f /private/etc/ssh/ssh_host_ecdsa_key.pub 
256 SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo no comment (ECDSA)

ECDSA (Elliptic Curve Digital Signature Algorithm – Wikipedia) is the default type of key ssh uses on macOS. The other keys are present for compatibility with other platforms and older versions of macOS/OS X.

A security conscious admin would go to the remote machine in question (or ask another user/admin who has access to the server/machine) and generate the fingerprint. Then go back to the Mac from where he wants to remotely connect with ssh and compare the fingerprint shown on first connection:

$ ssh mac.example.com
The authenticity of host 'mac.example.com ([IP Address])' can't be established.
ECDSA key fingerprint is SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo.
Are you sure you want to continue connecting (yes/no)? 

Since the key and the fingerprint are unique, you can be sure you are talking to the correct remote machine and there is no other malicious computer impersonating the remote machine on the network.

Making a list, checking it twice…

Once you confirm the prompt to connect to a new host, its public key will be added to the file ~/.ssh/known_hosts. This file will contain one line per remote host. Each line will contain the hostname, the IP address, the key type and the public key data itself. You can view the file in a text editor, or you can use the ssh-keygen tool to search this file for a particular host:

$ ssh-keygen -F mac.example.com
# Host mac.example.com found: line 16 
mac.example.com,[IP Address] ecdsa-sha2-nistp256 AAAA1234...=

Once the key is stored in your known_hosts file, ssh can detect if the public key presented by a remote ssh host changes and warn you. The warning is not subtle:

$ ssh mac.example.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for mac.example.com has changed,
and the key for the corresponding IP address [IP Address]
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo.
Please contact your system administrator.
Add correct host key in /Users/armin/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/armin/.ssh/known_hosts:16
ECDSA host key for mac.example.com has changed and you have requested strict checking.
Host key verification failed.

In most cases this means that IP addresses have changed. In the worst case, someone is maliciously attempting to impersonate an ssh host in your network, probably in an attempt to gather login information.

As administrators, there is another common reason for this warning. When you re-install or re-image a client Mac, it will generate new ssh keys. So every time you re-image and re-connect to a test machine, your Mac will compare the old keys in known_hosts with a newly generated, different key on the freshly imaged Mac and bring up this warning.

You can remove the old keys from the ~/.ssh/known_hosts file directly in a text editor or use the ssh-keygen command to remove out of date keys:

$ ssh-keygen -R mac.example.com
# Host mac.example.com found: line 16
/Users/armin/.ssh/known_hosts updated.
Original contents retained as /Users/armin/.ssh/known_hosts.old

The hostname is used to identify hosts and keys in this file. If you connect to the client with different hostnames, DNS aliases, or you alternate between the ‘proper’ DNS name mac.example.com, partial DNS name mac, the Bonjour name (i.e. Mac.local), or the IP address, you may end up with multiple entries for the same host in known_hosts. Usually this is not a problem, but is useful to know when cleaning out stale keys.

If you have the public key for a host, you can pre-add it to your known_hosts file. The format you have to use is

mac.example.com,[IP Address] ecdsa-sha2-nistp256 AAAA1234...=

Don’t copy these keys

If you use ‘Golden Master’ imaging on macOS (you really shouldn’t, for several reasons, this being one of them), you might inadvertently copy the host ssh keys from your Golden Master to all the other clients. While this does not break how ssh works, it will undermine some of its security. You can simply delete the existing keys on the Golden Master and new keys will be created on the clients when needed.

Remember there are other files in /private/etc/ssh/ that are required for ssh to work which you should not delete.

Locking down even more

You can increase or decrease the host key security with the StrictHostKeyChecking setting in the ~/.ssh/config file for a particular user or the/private/etc/ssh/ssh_config` file for all users on a system.

The default value is ask which means that ssh will ask what to do with an unknown key from a host. This is the behavior described above.

You can increase security by changing this setting to yes which means ssh will refuse to connect to a host which is not listed in ~/.ssh/known_hosts:

$ ssh stranger.example.com
No ECDSA host key is known for stranger.example.com and you have requested strict checking.
Host key verification failed.

With this setting you will have to manually obtain a host’s public key and add it to your known_hosts file.

You can also set StrictHostKeyChecking to no. With this setting, unknown hosts will automatically be added to the known_hosts file and changed public keys will automatically be replaced. This could be useful in a very dynamic, low-security environment (i.e. a classroom where Macs are re-imaged frequently). However, it does undermine the security of ssh and you will not notice any attempts to insert malicious hosts. In general you should not use this setting.

Recap

  • the ssh host (remote machine) has a set of encryption keys in /private/etc/ssh/
  • the public key is sent to the client. A fingerprint of the public key is presented with a prompt wether you should connect. You should use the public key fingerprint to verify the host’s authenticity on first connect.
  • generate the fingerprint from the public key on the host with ssh-keygen -l -f keyfile.pub
  • the public keys of approved hosts are stored in ~/.ssh/known_hosts
  • use ssh-keygen -R host to remove old or unused keys
  • you can control this behavior with the StrictHostKeyChecking setting in the config file

When you verify the fingerprint of the host, you can be sure of its identity. The host however, cannot verify a client’s identity in this setup. This is why you still have to enter a password every time you log in with ssh remotely.

We will discuss how to improve on that in the next post : SSH Keys, Part 2: Client Verification