Terminal Primer – Part 6 – Symbolic Links

If you like this series and want to learn Terminal and the shell on macOS in more detail, get my book “macOS Terminal and Shell

Symbolic Links

When you get a detailed list of the file system root / you will see a few interesting entries: (output abbreviated)

$ ls -l /
lrwxr-xr-x@    1 root  wheel    etc -> private/etc
lrwxr-xr-x@    1 root  wheel    tmp -> private/tmp
lrwxr-xr-x@    1 root  wheel    var -> private/var

These are symbolic link files. Symbolic links are identified by the first character l on a long ls. Symbolic links are redirections to other files or directories. /etc, /tmp and /var are standard directories for certain kinds of files in Unix systems, but in this case these directories are actually located in /private.

Note: The reason for this is historical and goes back to NeXTStep. This setup could allow the /private folder to be mounted from another disk or file share separate from the rest of the system. This is not used in macOS anymore, but the directory structure remains.

Aside from the long ls you can use the readlink command to determine where a symbolic links points to:

$ readlink /etc
private/etc

or the stat -l command:

$ stat -l /etc
lrwxr-xr-x 1 root wheel 11 Nov 17 07:50:53 2016 /etc -> private/etc

A symbolic link contains a path pointing to the original file or directory. Most operations such as reading or changing are directed to the original, rather than the symbolic link. ls /tmp and ls /private/tmp will show you both the contents of the original /private/tmp.

An example for a symbolic link to a file is the file /usr/share/dict/words (a list of english words, which can be surprisingly useful to have around) which points to a file web2 in the same directory. Symbolic links can be used as a means to ‘switch’ between files without having to change around filenames and configuration files.

When you read the file /usr/share/dict/words the read command will be redirected to /usr/share/dict/web2:

$ cat /usr/share/dict/words | wc 
  235886  235886 2493109
$ cat /usr/share/dict/web2 | wc 
  235886  235886 2493109

Note: the wc command counts words, lines and bytes in a file.

Symbolic links can be relative or absolute. However, most of the time they are relative, since you do not want them pointing to different files, depending on which volumes the system is booted from. Relative paths are resolved relative to the link itself, not the current working directory. The above link for /etc points to the relative path private/etc so to the sub-directory etc in the directory private in the same location as the symbolic link itself.

To create a symbolic link use the ln -s command.

$ mkdir directory
$ touch directory/file
$ ln -s directory/file
$ readlink file
directory/file

This series of commands will create a directory, and en empty file in directory and then a link to that file in the current working directory. When you want the symbolic link to have a different name, you can give that as a second argument:

$ ln -s directory/file second_link
$ readlink second_link 
directory/file

The first argument is not really the path to a file or directory, but the path the symbolic link points to. When this path is relative it will be resolved relative to the location of the symbolic link. So if you wrote:

$ ln -s directory/file directory/link_to_file

A symbolic link named link_to_file will be created in directory but it will point to directory/directory/file instead of the file next to it. When you try to read from the symbolic link pointing to nowhere, you will get an error:

$ cat directory/link_to_file 
cat: directory/link_to_file: No such file or directory

The reference you pass when creating the symbolic link has to be relative to the where the symbolic link is created. The correct command would have been:

$ ln -s file directory/link_to_file

When the second argument is a directory itself, a link named the same as the file it refers to will be created:

$ mkdir another_dir
$ ln -s ../directory/file another_dir

will create a link named file pointing to ../directory/file. Since you have to give the path to the target relative to the where the link is created you have to add the ../ to ‘go up a level’ out of the another_dir directory and then back into directory.

Note: you can create a symbolic link that points to non-exiting path. Also when the original file or directory gets deleted, a ‘dangling’ symbolic link will remain.

When you use rm or mv on a symbolic link, only the link will be affected, not the original item.

When you run cp on a symbolic link to a file, the contents of the original will be copied:

$ cp another_dir/file filecopy
 $ stat -l filecopy
-rw-r--r-- 1 armin staff 0 Sep  4 14:51:44 2017 filecopy

However, when you recursively copy a directory tree containing symbolic links, they will be copied as symbolic links:

$ cp -R another_dir/ copy_dir
$ ls -l copy_dir/
total 8
lrw-r--r--  1 armin  staff  17 Sep  4 14:52 file -> ../directory/file

Usually the destination of a copy will be in a vastly different location and this will break the links:

$ cp -R another_dir/ /Users/Shared/copy_dir
$ ls -l /Users/Shared/copy_dir
total 8
lrwxr-xr-x  1 armin  wheel  17 Sep  4 15:19 file -> ../directory/file
$ cat /Users/Shared/copy_dir/file 
cat: /Users/Shared/copy_dir/file: No such file or directory

You can use cp‘s -L option to make the copy process resolve symbolic links and copy their contents instead:

$ cp -RL another_dir/ /Users/Shared/another_dir
$ ls -l /Users/Shared/another_dir
total 8
-rwxr-xr-x  1 armin  wheel  17 Sep  4 15:19 file

Other commands will have similar options to control the behavior when encountering symbolic links. However, their names are not standardized in anyway and you will have to consult the man page to find out the details.

Symbolic Links vs Finder Aliases

In macOS Finder, you can create aliases with the menu item ‘Make Alias’ from the ‘File’ or context menu. Finder Aliases have much the same role as symbolic links, but a few significant differences in behavior:

  • the shell and most command line tools cannot resolve Finder Aliases and treat them as files
  • Finder Aliases will ‘follow’ the original when it is moved or renamed.
  • However, when the original is deleted and replaced by an item of the same name, a Finder Alias will resolve to the new item.
  • When a Finder alias points to an item on a file share, double clicking the alias in Finder will attempt to connect to the file share, if it not already connected.

Finder will display Aliases and symbolic links with a small arrow in the corner of the icon. Both symbolic links and Finder Aliases have a ‘Show Original’ menu item in the ‘File’ menu or context menu.

There are no commands to create or resolve Finder aliases in Terminal, but you can use AppleScript with the osascript command:

$ osascript -e 'tell app "Finder" to make new alias to posix file "/Users/armin/Documents" at posix file "/Users/armin/Desktop"'

This will create a new Finder alias to ‘Documents’ in the ‘Desktop’ folder. The user who is running this command has to be logged in to the Mac, so that osascript can connect to the Finder to run this command.

To find out the original of a Finder alias, you can use:

$ osascript -e 'tell application "Finder" to get POSIX path of ( (original item of (POSIX file "/Users/armin/Desktop/Documents" as alias) ) as alias)'

Note: there is a lot of type casting (as alias) in this command. To further confuse matters the data type alias in AppleScript is not the same as a Finder alias. It is a special data type in AppleScript that references a file or folder.

Parse Binary Property Lists in Finder Metadata

For more info on plutil and everything property list related read my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’

macOS and Finder use extended attributes to store plenty of extra information about files. For example, when you download a file in Safari, it stores when a file was downloaded and which website and download URL was used.

As an example, I downloaded the latest Firefox disk image. When you look at the downloaded file in the Terminal, you see an @ after the file mode which tells us this file has additional extended attributes:

$ ls -l ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg

We can further inspect the extended attributes with the -@ option:

$ ls -l@ ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg
    com.apple.metadata:kMDItemDownloadedDate          53 
    com.apple.metadata:kMDItemWhereFroms         203 
    com.apple.quarantine          57 

This shows the three extended attributes attached to this file, their names and their sizes in bytes.

When you double-click the dmg file to mount it, the system will store the checksum and file system check result in more extended attributes:

$ ls -l@ ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg
    com.apple.diskimages.fsck         20 
    com.apple.diskimages.recentcksum          81 
    com.apple.metadata:kMDItemDownloadedDate          53 
    com.apple.metadata:kMDItemWhereFroms         203 
    com.apple.quarantine          57 

To inspect the contents of the extended attributes in further detail, we have to use the xattr command: xattr -l filename will show all extended attributes, or you can use xattr -pl attributename filename to get just a particular one:

$ xattr -pl com.apple.quarantine ~/Downloads/Firefox\ 55.0.3.dmg 
com.apple.quarantine: 0083;59a6b982;Safari;E56EFC36-29AB-4F77-89E6-F4264336060F

The contents of the quarantine attribute is a string with some numbers (presumably a hash) the application that downloaded it (Safari) and a UUID.

The downloaded date however, looks a lot different:

$ xattr -pl com.apple.metadata:kMDItemDownloadedDate ~/Downloads/Firefox\ 55.0.3.dmg 
com.apple.metadata:kMDItemDownloadedDate:
00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 BF 56 F1 02  |bplist00..3A.V..|
00000010  53 AF D9 08 0A 00 00 00 00 00 00 01 01 00 00 00  |S...............|
00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 13                                   |.....|
00000035

This attribute is some binary data (xattr automatically uses binary representation when it detects a nil byte. The first few bytes in the binary data are bplist which tells us we are working with binary plist format. Unfortunately the rest of the binary plist data is quite unreadable for humans.

We can use xxd to convert the binary representation into actual data and plutil to print the plist:

$ xattr -px com.apple.metadata:kMDItemDownloadedDate ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -p -
[
  0 => 2017-08-30 13:11:30 +0000
]

Note that the options for xattr changed from -pl to -px which forces the output to be binary data only.

And the same command for the ‘WhereFroms’:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -p -
[
  0 => "https://download-installer.cdn.mozilla.net/pub/firefox/releases/55.0.3/mac/en-US/Firefox%2055.0.3.dmg"
  1 => "https://www.mozilla.org/en-US/firefox/new/?scene=2"
]

This uses plutil’s -p option to just print the data in a human readable form. You can also have plutil convert the plist data into XML:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -convert xml1 -o - -
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
    <string>https://download-installer.cdn.mozilla.net/pub/firefox/releases/55.0.3/mac/en-US/Firefox%2055.0.3.dmg</string>
    <string>https://www.mozilla.org/en-US/firefox/new/?scene=2</string>
</array>
</plist>

or JSON:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Download.0.3.dmg | xxd -r -p | plutil -convert json -r -o - -
[
  "https:\/\/download-installer.cdn.mozilla.net\/pub\/firefox\/releases\/55.0.3\/mac\/en-US\/Firefox%2055.0.3.dmg",
  "https:\/\/www.mozilla.org\/en-US\/firefox\/new\/?scene=2"
]

For more info on plutil and everything property list related read my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’

Terminal Primer – Part 5 – Managing Files

If you like this series and want to learn Terminal and the shell on macOS in more detail, get my book “macOS Terminal and Shell

Managing Files

We already know how to navigate and read the file system with cd and ls.
Now we want to actually do something to the files.

Create an Empty File

Sometimes it can be useful to quickly create an empty file. You can use the touch command to do this.

$ touch emptyfile

If you use the touch command on a file that already exists it will update the file’s modification date and change nothing else.
macOS (and other Unix-like operation systems) sometimes uses the existence of a file in a certain directory as a flag for configuration.
For example, when a file named .hushlogin exists at the root of a user’s home directory, the ‘Last Login: …’ message when you open a new shell (terminal window) is suppressed.

$ touch ~/.hushlogin

will create this file and subsequent new Terminal windows will not show this message. To return to showing the message, you will have to delete the file.

Deleting Files

To delete a file use the rm (remove) command:

$ rm document.txt

You have to use special care with the rm command.

In the Finder, deleted files are moved to the Trash, which is actually the invisible directory ~/.Trash. There the file will remain until the user chooses ‘Empty Trash’. Only then is the file removed from disk.
The command line has no such safety net.

When you delete a file (or directory) with the rm command it is gone. You have to be especially careful with files with special characters. If a filename has a space in it, and you run the rm command without escaping or quotes, then you will get an error or even worse, might delete the wrong file.

For example:

$ rm My Important Document.txt

Will delete the three files My, Important, and Document.txt, if they exist. If they do not exist it will show errors.

Use escape sequences or quotation marks to protect from spaces and other special characters in and directory names:

$ rm 'My Important Document.txt'

Tab-completion will also protect from improperly typed or escaped file names. If the tab-completion will not work, even though you believe you have the right file or path then something went awry and you have to step back and verify your working directory and paths.

To delete the .hushlogin file we created above, you use rm

$ rm ~/.hushlogin

Once the file is removed, new terminal windows will show the ’Last Login: …” message again.

You can add the -i option to the rm command which will ask for confirmation before actually deleting a file.

$ rm -i ~/.hushlogin 
remove /Users/armin/.hushlogin? y

Creating Directories

To create a new empty directory (or folder) you use the mkdir command.

$ mkdir scratchspace

you can give the mkdir command multiple arguments when you want or need to create multiple directories at once

$ mkdir folder1 folder2 folder3

When you create a nested directories, all the directories in between already have to exist:

$ mkdir LevelA
$ mkdir LevelA/LevelB/LevelC
mkdir: LevelA/LevelB: No such file or directory 

When you need to create nested directory hierarchies like this, you can use mkdir’s -p option:

$ mkdir -p LevelA/LevelB/LevelC

This will create all three folders at once, if they do not already exist.

Moving and Renaming

You can move a file or directory using the mv command. The mv command needs two arguments: the source and the destination.

$ touch testfile
$ mkdir testdir
$ mv testfile testdir
$ ls testdir
testfile

This mv command reads as ‘move the file testfile to the directory testdir’. To move it back to the current working directory you can use the ‘.’ short cut.

$ mv testdir/testfile .

The mv command can also rename a file:

$ mv testfile samplefile

Moving and renaming is considered the same in the shell.

Warning: When a file already exists in the destination, mv will mercilessly overwrite the destination file. There is no way to retrieve an overwritten file.

You have to take care to type the proper paths in the shell. It is a very unforgiving environment.
To make mv a bit safer add the -i option which will prompt to confirm when it will overwrite a file:

Warning: when you use mv to move between volumes, the source file will be removed after it is moved to the destination. This is different from the behavior in Finder, where the default drag action between volumes is copy.

Filename Extensions

On macOS and other operating systems it is common to denote the file type with an extension. The extension is a standard alphanumeric code separated from the rest of the file’s name by a dot or period. E.g. .txt, .pdf or .mobileconfig.

In bash, the filename extension is part of the filename. There is no special treatment for the extension.
On macOS, however, Finder usually hides the file extension from the user. You can control the display of the file extension in the ‘Advanced’ tab of Finder Preferences. You can also control this setting for each individual file in its Info panel.

Finder will also warn when you attempt to change the file extension, since it might change which application is used to open a file. (You can also disable this in the Finder Preferences.) bash has no such warning mechanism.

$ mv hello.txt hello.md

Note: A feature specific to macOS is that some directories will have filename extensions and Finder will display them as if they were files, not folders. These folders are called ‘packages’ or ‘bundles.’ The most common example are applications with the .app extension.
Packages and bundles are used to hide complex file and data structures from users. Another example is the ‘Photos Library’ (or ‘iPhotos Library’ on older systems) which hides a big and complex folder structure.
You can choose ‘Show Package Contents’ from the context menu in Finder to drill down further into the internal structure of a package or bundle.
bash and other shells are not really aware of packages or bundles and treat them like normal directories.
You can read more detail on Bundles and Packages on the Apple Developer Page.
Note to the Note: the name ‘packages’ is also used for package installer files (with the .pkg extension). These are different uses of the same word.

Copying

The command to copy is cp. It follows similar syntax as the mv command:

$ cp source destination

So you can copy the samplefile we created earlier:

$ cp samplefile newsamplefile

You can also copy to a directory:

$ cp samplefile testdir

Warning: the cp command will mercilessly overwrite existing files with the same name!

When you run cp again it will overwrite the existing copy:

$ cp samplefile testdir

As with the rm command overwritten files are lost. There is no way to retrieve overwritten files.

The -i option shows a prompt to confirm whenever a file will be overwritten:

$ cp -i samplefile testdir
overwrite testdir/samplefile? (y/n [n]) n
not overwritten

When you try to copy a directory, you will get an error message:

$ cp testdir newdir
cp: testdir is a directory (not copied).

Since the command to copy a file or directory would look exactly the same, cp expects an extra option to be certain you know what you are doing. The -R option (for recursive) will tell cp to recursively copy all files and sub-directories (and their contents) of a folder.

$ cp -R testdir newdir

This will create a copy of testdir and all its contents with the name newdir.

Warning: the option for recursive copying is -R (uppercase R). There is a legacy option -r (lowercase r) which seems to do the same thing. However, there is a difference in behavior mentioned in the cp man page:

Historic versions of the `cp` utility had a `-r` option.  This implementation supports that option; however, its use is strongly discouraged, as it does not correctly copy special files, symbolic links, or fifo's.
If the destination directory already exists the way you write the path of the source directory will influence the behavior of cp.

When the path to the source does not end with a /, cp will create a copy of the directory in the destination directory:

$ mkdir dirA
$ cp -R testdir dirA
$ ls dirA
testdir

When the path of the source directory ends with a /, cp will copy all the contents of the source directory to the destination folder:

$ ls testdir
samplefile
$ mkdir dirB
$ cp -R testdir/ dirB
$ ls dirB
samplefile

Warning: when you use tab-completion to complete paths to directories the / is always appended! You will need to consider whether you want to keep the trailing / or not.

You can add more source arguments to a cp command, the last argument will be the destination:

$ cp -R samplefile otherfile hello.txt bigfolder

Wildcards (Globbing)

Note: In early versions of Unix wildcard substitution was the responsibility of a program called glob, short for ‘global command.’ Because of this the action of replacing wildcards with actual paths and filenames was and still is called globbing.

When you have to address or manage many files at once, it can be slow, tedious and ineffective to address each file individually. bash provides wildcard characters to make that easier.

There are two commonly used wildcard characters: * and ?

The asterisk or star * will match zero or more characters. It can be placed anywhere in a path.

The question mark ? will match any character, but there has to be a character.

By default filenames that start with a period ‘.’ are not matched, unless you specifically start the string with a dot .*

It is important to keep in mind that bash will build a list of filenames that match the wildcards and substitute this list in place of the name with the wildcard(s) before executing the command.

When you enter

$ cd ~
$ ls D*

The D* will be replaced with the list of filenames that match (Desktop Documents Downloads) and then executed:

$ ls Desktop Documents Downloads

This can lead to some unforeseen consequences. For example say you are in a folder with some files and directories:

$ ls -F
dirA/  dirB/  dirC/  file1  file2  file3

And you run

$ cp file? dir?

The wildcards will be expanded to

$ cp file1 file2 file3 dirA dirB dirC

Which means that the three files as well as dirA and dirB will be copied into dirC, since treats the last argument as the destination and all previous arguments will be copied.

You can use wildcards in paths, so /Users/*/ will expand into all directories in the /Users folders.

However, /Users/*/Desktop will expand into a list of all users’ Desktop folders. Note that the first list contains /Users/Shared while the second does not contain /Users/Shared/Desktop, because that directory does not exist!

Warning: Wildcards can be extremely useful, but also very dangerous. They have to be handled with utmost caution, especially with potentially destructive commands such as mv, cp, and rm.

You can always test the result of wildcard expansion with the echo command:

$ echo /Users/*/
/Users/Guest/ /Users/Shared/ /Users/armin/
$ echo /Users/*/Desktop
/Users/Guest/Desktop /Users/armin/Desktop

You can also hit the escape key twice and bash will show the expansion, if there are any:

$ echo /Users/*<esc><esc>
Guest/  Shared/ armin/

Finally, bash has a third globbing or wildcard character, but it is a bit more complex. You can provide a list of possible characters between square brackets.

[bcr]at

will match

bat, cat or rat, but not Bat, Cat or Rat

Since shell commands are case-sensitive, you may have to provide both cases, if you want to match:

[bBcCrR]at

No matter how many characters are in the square brackets, they will match to exactly one character:

[bB][aei]t matches bat, Bat, bet, Bet, bit, or Bit

Deleting Directories

We have been creating and copying a lot of files. It is time to clean up. We already know the rm command to remove files. However, when you try to use rm to delete a directory you get:

$ rm newdir
rm: newdir: is a directory

There is a command rmdir which is the destructive equivalent of mkdir. However, rmdir can only remove empty directories:

$ rmdir newdir
rmdir: newdir: Directory not empty

You can use the * wildcard to delete all files in newdir:

$ rm newdir/*

Note: the * wildcard will not expand to filenames starting with a period. You may have to explicitly delete dot files as well:

$ touch newdir/.dotfile
$ rm newdir/*
$ rmdir newdir
rmdir: newdir: Directory not empty
$ rm newdir/.*
rm: "." and ".." may not be removed
$ rmdir newdir

This will work as long there are only files or empty directories in newdir. When the directory you want to delete contains an entire hierarchy of files and directories, then this approach will be cumbersome.

For this the rm command has the -R option which will recursively delete all contents and subdirectories.

$ rm -R testdir

Since there is no way to recover a file deleted by rm you should always use this command with care, especially when using the -R option.

You can add the -i option when using -R as well, but then you will be prompted for every single file and subdirectory, which can be very tedious and counter-productive.

Note: unlike the cp command, the -r and -R option for the rm command are synonyms. However, for consistency’s sake and to build muscle memory. I would recommend making a habit of using the -R syntax for both commands.

Terminal Primer – Part 4 – Commands

If you like this series and want to learn Terminal and the shell on macOS in more detail, get my book “macOS Terminal and Shell

Arguments

So far we have use three commands: pwd, cd, and ls

These commands are already quite different.

pwd is a single word command. You enter it and it prints information (the working directory to the terminal).

cd, however, requires additional information from you: where do you want to change to? The cd command requires an argument:

$ cd ~/Documents

(You can enter cd without an argument, and it will change to your home directory, but usually you want an argument.)

The command itself cd and the argument ~/Documents are separated by a space.

Some commands can have more than one argument. In that case all arguments are separated from each other by a space. (Or more. bash doesn’t care about multiple spaces.)

This is why we have to treat spaces in paths and filenames so carefully, because otherwise the shell might interpret the path as two or more arguments.

Finally ls has an optional argument. When you just write ls. it will list the contents of the current working directory. When you give an argument it will list the contents of that path. The ls command also has several options that modify its behavior.

When a shell command is written in documentation optional arguments are usually enclosed in square brackets:

ls [-options] [path]

Mandatory arguments, on the other hand, are shown without the square brackets.
When you enter an ls command with completely wrong options (surprisingly difficult, since its options cover nearly the entire alphabet, and some extra characters as well.) it will print a “usage” line:

$ ls --a
ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

The extra ... after the optional file command tells us, that you can give ls more than one path argument:

$ ls ~/Desktop ~/Documents

Read the Manual

When you want detailed information on a command, there are a few approaches.

Because of the long and diverse history of shells, bash and macOS in particular, not all commands support all of these options. Behavior here can be very inconsistent.

First, as we just saw with ls, some commands will print a brief usage note, when you enter something that the command cannot parse.

With some commands you can provoke the usage message with the -h or --help option:

$ sw_vers -h
Usage: sw_vers [-productName|-productVersion|-buildVersion]

The usage message is commonly very brief and usually does not explain all the options.

To get more detailed in information on command you can read its man page. man pages are documentation, often very detailed, stored in an file format optimized for display in ASCII terminals.

To get the man page for a command run the man command:

$ man ls

This will take over the current Terminal window and display the information.

This special display mode is actually controlled by another command called less. There many key commands you can use for navigation in this mode.

Key
q exit and return to command line prompt
up/down arrow scroll up/down by a line
space or z scroll down by a page
w scroll up a page
g top of document
G (shift-g) end of document
/word<return> find next occurrence of word in document
n find next occurrence of search term
N find previous occurrence of search term
h help

You can also scroll in this mode with the mouse wheel or two-finger scrolling on a trackpad.

You can also open man pages in terminal from the Help menu. When you enter a shell command in the help search field of Terminal it will suggest a man page, when one is available. When you select a suggested man page, it will open in a new yellow window.

You can modify the appearance of the man page window by changing the ‘Man Page’ window profile in Terminal’s Preferences.

You can also open a man page by selecting text and choosing ’Open man page from the context menu.

Some commands are ‘built-in’ to the bash shell. These do not always have man pages. Requesting the man page for a built-in command will show the man page for builtin instead.

cd is one example for a built-in command.

You can get documentation for built-in commands with

$ command help cd

Finding commands

We just learned that some commands, like cd, are ‘built-in’ to the shell. Others are not, so what and where are they?

All commands are files in the file system. They have a special file privilege set which makes them executable. Obviously, you cannot make any file executable, it has to have some form of code which makes sense so the system can interpret it as commands.

If you want to know where a given command resides in the file you can use the which command

$ which ls
/bin/ls
$ which sw_vers
/usr/bin/sw_vers

However, you do not have to type /bin/ls every time you want to execute ls. How does the shell know where to look?

The shell has an environment variable called PATH which contains a list of directories where it will look for commands that are typed without an absolute path. You can print the contents of this variable with the echo command:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Note: commands and variable names in the shell are case-sensitive. It is convention that environment variables are written in all-caps. You have to use the correct case for the PATH variable to get or set the proper value.

When you are new to shell and bash, there is a lot to process in this simple command, so let’s take this apart piece by piece:

The echo command simply tells the shell to print something to the terminal, so

$ echo hello
hello

prints ‘hello’ back to the terminal. This alone is rarely useful, but can be used to get at the results of another process.

$ echo $(( 6 * 7 ))
42

The $(( … )) means ‘evaluate this arithmetically,’ so this command prints the result of this arithmetic to the terminal.

In general in bash the $ stands for ‘substitute contents.’ echo $PATH means: print the contents of the PATH variable.

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

When you forget the $ and just write

$ echo PATH
PATH

bash will interpret PATH as a literal string and prints it to the terminal.

The actual contents of the PATH variable is a list of directories separated by colons.

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The order of the directories in the PATH is important as the shell will stop looking when it finds a command.

When you enter a command without a path, e.g. ls, bash will start looking for the command executable in /usr/local/bin, then in /usr/bin, and then in /bin, where it will find an executable ls, stop looking and execute that.

Note: if there were another executable named ls in a later directories it would not be used, since the shell will stop looking at the first match it finds. Changing the order of the standard directories in the PATH or even inserting other directories earlier in the PATH can lead to unexpected behavior.

The PATH on your system may be different when you have extra software installed. Xcode, Server.app, Xquartz, munki, Python3 and many other software packages insert paths to their command directories in the search path.

Note: some software solutions will attempt to modify the PATH on a system to make their commands available to the shell, other will place the commands or links to the commands in /usr/local/bin to make them available (e.g. text editors like BBEdit or Atom).

We will look at strategies to on how and why to modify the search path later.

Some third party solutions will instruct you to modify the PATH to include their commands rather than doing it during the installation.

Running Other Commands

When you need to execute a command or script that is not in the PATH, you have to type the full or relative path to the command:

$ /usr/libexec/PlistBuddy
Usage: PlistBuddy [-cxh] <file.plist>

or

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport --getinfo

These are commands that are usually considered too uncommon or maybe even dangerous to put in the standard search paths.

When you start using and writing custom-built scripts and commands, you can use relative paths:

$ test/my_great_script.sh

or

$ scripts/postinstall

When you need to execute a command or script in the current working directory, you have to start the command with ./, so the shell knows to not look in the search path.

$ ./buildMyProject.sh

Remember the . is a shortcut representing the current working directory.

Tab-completion for Commands

You can use tab-completion for commands as well. This will speed up your typing and prevent typing errors.

You can use this to get a list of all the commands available in the shell. At an empty command prompt hit the tab-key twice. Then shell will warn you that there are many completions (more than a thousand, depending on your version and configuration of macOS.

You can also use this command to list all tab-completions:

$ compgen -c

Note: compgen is the command that bash runs to determine which commands are available for tab-completion. You usually would not interface with it directly.

Terminal Primer – Part 3 – Special Characters

If you like this series and want to learn Terminal and the shell on macOS in more detail, get my book “macOS Terminal and Shell

There are a group of characters that have special meaning in bash.

| & : ; ( ) < > ~ * @ ? ! $ # [ ] { } \ / ' " `

Also whitespace characters also need to be treated with care:

space, tab, return, and newline

The space character is a legal and often used character in file names on macOS. However, in bash and other shell commands , a space character (and other whitespace characters) separates a command from an argument and the arguments from each other.

When you try to enter a filename with a space, you will get an error:

$ cd /Library/Application Support
-bash: cd: /Library/Application: No such file or directory

To convince the shell that ‘/Library/Application Support’ belongs together, you can either ‘escape’ the space character or ‘quote’ the path.

Experienced users who have worked in a UNIX environment for a long time tend to avoid these special characters in filenames. However, as a system administrator, your users will probably not heed any rules you may want to impose. You will have to deal with many possible combinations.

Escaping Characters

The escape character in bash is the backslash \. A character that follows a backslash will be treated with no special meaning:

$ cd /Library/Application\ Support

In Finder, you can name files and folders nearly any way you want. When you encounter special characters from the list above you have to escape them with backslash. For a directory named ‘Project (Old & New)’ you would type:

$ cd Project\ \(Old\ \&\ New\)

All of this further confused by the fact that the shell will happily display the path with the unescaped special characters:

$ pwd
/Users/armin/Project (Old & New)

Separation Characters

In bash (and in Unix in general) files and directory names cannot contain a forward slash / since the character is used in paths to separate directories. However, Finder lets you name a file or folder with a forward slash, e.g. ‘Yes/No’.

On the other hand, Finder does not let you name a file or folder with a colon :. The underlying HFS+ file system uses the colon as a separator.

This conflict is solved by translating a / in the Finder (and underlying file system) to a colon : in the shell and vice versa.

A folder named ‘Yes/No/Maybe’ in Finder will appear as Yes:No:Maybe in the shell and you have to escape the colons when using the path in a command:

$ cd Yes\:No\:Maybe

Note: some characters that are legal on macOS might not be on file servers, which are usually hosted by other operating systems.

Quoting

As seen above, escaping characters can make the path quite unreadable. You can also place the name or path in quotes:

$ cd 'Project (Old & New)'

In bash you can use single quotes ' or double quotes " to quote paths.

Single quotes are more effective. Any character in single quotes is used as is, with no special function. Even the backslash character has no special function. The only character you cannot use in single quotes is the single quote itself.

Double quotes " are ‘weaker’ quoting. Double quotes remove the special function from all special characters except $, , <code>\</code>, and <code>!</code>. Within double quotes you can use the backslash to escape <code>$</code>, <code>"</code>,, and \ (but not the !).

Escape Strategies

In general, single quotes are most useful and easiest to use. However, you cannot use single quotes when the filename contains a single quote.

Double quotes still require some characters to be escaped with the backslash and cannot deal with an exclamation mark !.

Backslash escaping works in nearly all cases, but can be tricky to type right and is quite illegible.

name (in Finder) Backslash Escape single Quotes Double Quotes
My Great Folder My\ Great\ Folder 'My Great Folder' "My Great Folder"
“New” Files \"New\"\ Files '"New" Folder' "\"New\" Folder"
‘Old’ Stuff \'Old\'\ Stuff cannot escape ' "'Old' Stuff"
Important! Important\! 'Important!' cannot escape !
Bump \m/ Bump \\m\: 'Bump \m:' "Bump \\m:"
Do@Home Do\@Home 'Do@Home' "Do@Home"
Yes/No/Maybe Yes\:No\:Maybe 'Yes:No:Maybe' "Yes:No:Maybe"
Project (Old & New) Project\ \(Old\ \&\ New\) 'Project (Old & New)' "Project (Old & New)"
Profit$$$ Profit\$\$\$ 'Profit$$$' "Profit\$\$\$"

Quoting and Tab Completion

When typing paths, always use tab completion to be safe. Tab completion uses backslash escaping by default.

$ cd Proj⇥
$ cd Project\ \(Old\ \&\ New\)/

However, when you start a quoted path, tab completion will complete in quoted form.

$ cd 'Pro⇥
$ cd 'Projects (Old & New)'/

Tab completion is even smart enough to change the approach when the strategy you chose (i.e. double quotes) cannot work:

$ cd "Imp⇥
$ cd Important\!/

Quoting and Home Path

Since you generally use quoting to avoid bash changing characters, you cannot use the ~ to get a short cut to your home directory in quotes.

However, you can leave the ~ outside of the quotes and get the best of both worlds:

$ cd ~/'Project (Old & New)'

When you use double quotes, you can also use the $HOME environment variable:

$ cd "$HOME/Project (Old & New)"

Next: Commands

Terminal Primer – Part 2 – Navigating the File System

If you like this series and want to learn Terminal and the shell on macOS in more detail, get my book “macOS Terminal and Shell

Working Directory

Back to our earlier example, the pwd command. You entered the command pwd and the shell returned a result:

$ pwd
/Users/armin

Your output will be different. Unless your name is also armin, the path will end with your user name, not mine. Depending on the configuration of your Mac, your path might be entirely different.

pwd is short for ‘print working directory.’ It shows the folder this shell is currently working in, much like a Finder window displays the contents of a certain folder.

On macOS, Terminal will also show the working directory in the window title bar.

Paths

Modern file systems are hierarchical and have many folders and files nested in each other. In a Unix shell you denote a folder or file in another folder with the forward ‘/’ slash character. The complete series of folders to a file or folder is called its ‘path.’

For example, if there is a file hello.txt in your Documents folder in your home directory, its complete path or absolute path is:

/Users/armin/Documents/hello.txt

A path starts at the ‘root’ of the file system, which is shown as simply /. On macOS the root of the file system is the top level of the volume or disk with the active system, i.e. the current startup volume of the Mac.

Note: The file system root / should not be confused with the system user root.

This path tells me (and the shell) that there is a file ‘hello.txt’ in a folder ‘Documents’ in a folder ‘armin’ in a folder ‘Users’ on the root of the file system, the current startup volume.

Note: Finder in macOS and other graphical interfaces use the word ‘folder.’ Unix and other command line shells use the word ‘directory.’ They are the same.

Relative Paths

Addressing files and folders with their full path each and every time would be very tedious. To help with that a shell has a ‘current working directory.’

You can refer to files and folder relative to the shell’s current working directory. When using relative paths, there is no initial /.

hello.txt

refers to the file ‘hello.txt’ in the current working directory.

Documents/hello.txt

refers to the file ‘hello.txt.’ in the folder ‘Documents’ in the current working directory.

Relative paths do not have a leading / and are relative to the current working directory. Full paths start with a / and are based in the file system root. Full paths are also called ‘absolute’ paths.

It is surprisingly easy to lose track of the current working directory. You can ask the shell for the current working directory with the pwd command.

Mac:~ armin$ pwd
/Users/armin

Changing Directories

You can change the current working directory with the cd command (change directory):

Mac:~ armin$ cd Documents
Mac:Documents armin$ pwd
/Users/armin/Documents

Use the cd command with a relative path Documents and the shell changes it working directory there. The pwd command prints the full path of the working directory.

You can also navigate to an absolute path:

Mac:Documents armin$ cd /Library
Mac:Library armin$ pwd
/Library
Mac:Library armin$ cd /
Mac:/ armin$ pwd
/

You may have noticed that the prompt displays the name of the current directory. This helps you remember your ‘bearings’ in terminal. Terminal on macOS will also display the current working directory in the window title bar.

You can open a second terminal window by selecting ‘New Window with Settings – Basic’ from the ‘New Window’ menu in the ‘Shell’ menu. It is easier to remember the ⌘N (command-N) keystroke. If you prefer you can also open new shells in a new Tab with ⌘T.

The second window or tab will start a second, new bash shell. This shell is entirely separate of the first bash shell. Different shells will have different working directories. They are very much like different windows showing different folder contents in the Finder.

Home Directory

The prompt in the second terminal window will show:

Mac:~ armin$

According to the prompt current working directory is ~?

The tilde ~ is a special character in bash and other shells. It is a shortcut for the user’s home directory or home folder. On macOS a user’s home directory is created in the /Users directory with the user’s short name as the directory name. So if my short name is ‘armin’ my home directory is /Users/armin.
So the ~ in the prompt says the current working directory is my home folder.

Mac:~ armin$ pwd
/Users/armin

Note: It is important that the shell will actually substitute a ~ with the home directory path before executing the command.

You can use the ~ with cd to quickly change the working directory to your home directory

$ cd ~
$ pwd
/Users/armin

You don’t have to use this, though, since cd without any arguments, will change to your home directory.

$ cd
$ pwd
/Users/armin

You can also start a path with ~:

$ cd ~/Documents
$ pwd 
/Users/armin/Documents

Since ~ is replaced with an absolute path to the home directory, paths that start with ~ are absolute as well.

Moving On Up

You can change the working directory into subdirectories and you can change to absolute paths.

You also often want to move up one level in the folder hierarchy, to the parent folder of the current folder.

Now you could remember your current location (or recall it with pwd) and cd to the absolute path of the parent manually. However, there is an easier way to do this:

$ cd ~/Documents/
$ pwd
/Users/armin/Documents
$ cd ..
$ pwd
/Users/armin

In bash (and most other shells) two periods .. represent the parent directory of the current working directory.

You can even use .. multiple times:

$ cd ~/Library/Preferences/ByHost
$ cd ../..
$ pwd
/Users/armin/Library

../.. designates the parent directory of the parent. ../../.. goes up three levels, etc.

Note: if you cd / and then cd .. nothing happens.

Moving Back

The shell remembers the last working directory, as well, even if you don’t. You can quickly switch back to the previous working directory with cd - (minus):

$ cd ~/Documents
$ cd /Library/Preferences
$ cd -
/Users/armin/Documents
Documents $ cd -
/Library/Preferences

To assist orientation, cd - will print the full path to the directory it is changing to.

Tab Completion

Typing paths is tedious and error prone. Errors in paths can lead to simple errors and frustration, and sometimes typos can have catastrophic results.

When you are typing commands, file names or paths in bash, then you can use the tab key ⇥ to save keystrokes and avoid typos.

For example, you can type:

$ cd ~/Doc⇥
and it will complete to
$ cd ~/Documents/ 

Tab completion will add a forward slash / to the end of a directory’s name so you can continue typing the next directory name. You can hit tab over and over at different parts of the command:

$ cd ~/Li⇥
$ cd ~/Library/
$ cd ~/Library/Appl⇥
$ cd ~/Library/Application\ S<beep>
$ cd ~/Library/Application\ Su⇥
$ cd ~/Library/Application\ Support/

When there are multiple options to complete, bash will complete as far as it is unambiguous and will play an alarm sound. When you then press tab for the second time, it will list all options:

$ cd ~/D⇥<beep>⇥
Desktop/   Documents/ Downloads/

Using tab-completion not only saves keystrokes and time, but also reduces the potential for typos and errors.

You may have noted that tab completion did something strange to the space in the path of the ‘Application Support’ folder. There are many special characters that you have to deal with in the shell. Space is one of them. We will learn how to deal with space and the other special characters in the next section.

Note: commands, arguments and file names in bash are case-sensitive. However, since the macOS filesystems (HFS+ and APFS) are case-insensitive by default, you may want to switch tab-completion to be case-insensitive as well , too.

Listing Directory Contents

The next command is ls, which is short for ‘list’.

$ cd ~
$ ls
Desktop     Downloads   Movies      Pictures
Documents  Library     Music       Public

This command will list the contents of the current working directory. The contents of your home directory may be different.

To use the space efficiently ls prints the files and folders in multiple columns (the number of columns depends on the size of your Terminal window).

This simple list can be very useful to lookup file and directory names. However, you cannot tell the difference between files and directories in this list.

You can tell ls to show an extra character to indicate the kind of an entry:
$ ls -F
Desktop/ Downloads/ Movies/ Pictures/
Documents/ Library/ Music/ Public/

This version of ls output will add a slash ‘/’ to directories and an asterisk ‘*’ to executable files. Normal files (documents) will have no extra character.

In this command the -F is an option for the ls command. Options are a special kind of argument. Options usually start with one or two a hyphen characters - and are followed by a letter. Generally options work like switches to turn certain features of the command on or off.

Since the shell is case-sensitive the options -F and -f are different.

Since ls is a very common command, it has many options. The -G option works similar to the -F option but marks the different kinds with colors:

You can use multiple options at once:

$ ls -G -F
Desktop/   Downloads/ Movies/    Pictures/
Documents/ Library/   Music/     Public/

With most commands you can combine multiple options after a single hyphen: ls -GF is the same as ls -G -F

You can also add an argument to ls:

$ ls ~/Desktop

will list the contents of the Desktop directory without changing the working directory. You can combine options and an argument. When you do that you usually put the options before the argument.

$ ls -GF ~/Desktop

Detailed List

The default output of ls is very terse. You can get more information by adding the -l (lower-case L) option.

$ ls -l
total 0
drwx------+  6 armin  staff   204 Jul 18 17:22 Desktop
drwx------+  3 armin  staff   102 Jun  6 11:24 Documents
drwx------+  3 armin  staff   102 Jun  6 11:24 Downloads
drwx------@ 50 armin  staff  1700 Jul 18 16:02 Library
drwx------+  3 armin  staff   102 Jun  6 11:24 Movies
drwx------+  3 armin  staff   102 Jun  6 11:24 Music
drwx------+  3 armin  staff   102 Jun  6 11:24 Pictures
drwxr-xr-x+  5 armin  staff   170 Jun  6 11:24 Public

This command lists one file or directory per line. The columns are (in order):

  • file type and mode
    • first character shows type: (d directory, l link, – file)
  • number of links
  • file owner
  • file group
  • file size in bytes
  • modification date and time
  • name

There are more options that you can use with the -l option. For example the -h option will show file sizes with B (bytes), K (kilobytes), M (megabytes) etc. instead of raw bytes.
You can also combine -l with -G and/or -F.

Invisible Files

Certain files and directories that are invisible in Finder are visible in the shell. The ~/Library folder has been hidden in the Finder since Mac OS X 10.7 Lion, but is easily visible with ls. By convention however, ls does not usually list files and directories that start with a dot or period .. You can make ls show these files with the -a option. Your home directory will already have a few ‘dot-files’:

$ ls -al ~
total 16
drwxr-xr-x+ 15 armin  staff   510 Jul 18 15:30 .
drwxr-xr-x   7 root   admin   238 Jul 19 08:35 ..
drwx------   4 armin  staff   136 Jul 19 14:55 .Trash
-rw-------   1 armin  staff   402 Jul 19 08:39 .bash_history
drwx------   3 armin  staff   102 Jul 13 09:16 .ssh
drwx------+  8 armin  staff   272 Jul 19 14:57 Desktop
drwx------+  3 armin  staff   102 Jun  6 11:24 Documents
drwx------+  3 armin  staff   102 Jun  6 11:24 Downloads
drwx------@ 50 armin  staff  1700 Jul 18 16:02 Library
drwx------+  3 armin  staff   102 Jun  6 11:24 Movies
drwx------+  3 armin  staff   102 Jun  6 11:24 Music
drwx------+  3 armin  staff   102 Jun  6 11:24 Pictures
drwxr-xr-x+  5 armin  staff   170 Jun  6 11:24 Public

In UNIX files and directories that start with a period are commonly used to store configuration data.
The first two entries in this list are . and ... We already know that .. represents the parent directory (in this case of the directory listed). The single period . is another convention that represents the current directory or (in this case the directory listed). This can be useful to see the file mode and owner of these directories right here.

Note: Finder also follows this convention and hides ‘dotfiles’ by default. You can learn more about hidden files and folders in this article.

Summary

So far we have encountered three commands to navigate the file system: pwd, cd, and ls

pwd prints the current working directory.

$ pwd
/Users/armin/Documents

cd changes the working directory to another.

You can use absolute paths (that begin with /):

$ cd /Library/Application\ Support
$ cd /

or paths relative to the current working directory (no leading /):

$ cd Documents

Two periods .. represent the parent directory:

$ cd ..

changes the working directory to the parent directory.

The tilde ~ represents your home directory. You can use it alone or to start a path to folders and files in your home directory.

$ cd ~
$ cd ~/Documents

The ls command lists the contents of the current working directory:

$ ls
Desktop        Downloads   Movies      Pictures
Documents  Library     Music       Public

The ls command has many options. The most commonly used are probably -l to show a detailed list of the files and folders and -a to also show the files and directories starting with a period, which are usually hidden.

Next: Special Characters

Terminal Primer – Part 1 – First Steps in Terminal

If you like this series and want to learn Terminal and the shell on macOS in more detail, get my book “macOS Terminal and Shell

Terminal Application

On macOS you can find the Terminal application in /Applications/Utilities/. There is a shortcut in Finder’s Go menu to get to the Utilities folder or use the keystroke ⌘⇧U. Terminal has a distinct black icon with a white prompt.

However, since you are reading this book, you are planning to use Terminal regularly. In this case, you really want to add the Terminal application to your Dock.

First Prompt

Once you have opened Terminal, it will show you a new window, which is white with black text by default. A short message will show the last login and then a line with the default prompt:

Mac:~ armin$

At first it shows the computer name (as set in the ‘Sharing’ pane in System Preferences. After the colon ‘:’ it shows a tilde ‘~’. Then your user short name, followed by a dollar sign ‘$’. Finally a dark gray block, the cursor, waiting for your input.

Note: if you have used Terminal before, you may have changed the default look of the Terminal and prompt. The functionality remains the same, no matter how it looks.

You can enter your first command. Type the letters p, w and d and then hit the return key ↩︎

Mac:~ armin$ pwd
/Users/armin

You have typed the pwd command and executed it with the return key ↩. The return key confirms a command and runs it.

When you mistype a command you can delete the last character with the delete or backspace key ←. You can also move the cursor through the characters you entered with the left and right arrow keys.

When you hit return with a wrong command or a command with a typo the shell will complain that it cannot find the command:

Mac:~ armin$ pws
-bash: pws: command not found

Terminal Terminology

We have already encountered a few different terms which may be very confusing in the beginning. The bad news is there are more terms, the good news is that it will all make sense eventually.

All of these terms date back to the days when computing started with big mainframe computers. Since computers were scarce, complex and expensive, many users had to share access. They did this by typing and reading on a ‘terminal’ or ‘console’ — a dedicated device talking to the main frame. Terminals and consoles may have looked somewhat like like modern workstations and computers, but did not have their own CPU. They would just allow to enter and send commands and receive and display the results.

Very early in the history of computing, terminals used mechanical printers or typewriters to show the results, these were called tele-typewriters or tty. The protocol that the mainframe used to communicate with the typewriters, was named tty as well. The protocol and its name have remained, long after the mechanical typewriters are gone. Later the mechanical interfaces where replaced by terminals with electric keyboards and cathode ray screens.

As computers got smaller, more powerful and cheaper it became possible to have one on everybody’s desk. Even so, it was (and is) still convenient to run a shell, either locally or to connect to larger mainframes (servers). To do that you would open a program (or application) that ran a ‘virtual terminal.’

The Terminal application on macOS is such a virtual terminal program. To confuse things a little, macOS also has an application called Console. The Console application on macOS is not a virtual terminal or virtual console, but used to display and filter log files.

The terminal (whether virtual or real) only provides a means (virtual or mechanical) to enter and display text. It will display a ‘prompt’ which tells the user, that the system is ready and the user can enter a command, and a ‘cursor’ which shows where typed text will be displayed or inserted.

Shells

There is another program which interprets the text, executes code and sends the output generated back to the terminal.

This program (or class of programs) is called a ‘shell.’ A shell protects the user from the dangerous, complicated parts of the system and abstracts differences from one system to the other. Another way of looking at it, is that a shell protects the vulnerable, fragile parts of the the system from the user.

There are many different shell programs. One of the earliest shell programs was call just ‘shell’ or sh for short. Surprisingly, sh is still around after nearly fifty years.

On macOS the default shell is called bash for ‘bourne again shell.’ This comes from the fact that it was developed as a replacement for bsh or ‘Bourne shell,’ which was named after its main developer Stephen Bourne.

You will notice that unix shells and commands often exhibit a particular style of pun humor. ‘bash’ is merely the beginning.

Today, there are many different shells. Different shells have different ways of interpreting commands. The choice of shell is a personal preference and can be (as many things in computing) the cause of passionate argument.

Note: As system administrators, the choice of shell is not just determined by preference, but also practicality. There are a set of pre-installed shells on macOS and while it is possible to install additional shells, it increases complexity and management effort. Also when you are sharing commands and scripts with fellow administrators, bash is the commonly agreed upon shell.

bash is available for most platforms and operating systems and is the default shell on many systems. Because of this prevalence bash is a good choice for your first shell.
bash has been the default shell for Mac since Mac OS X 10.3 (Panther). It is also the default shell on most Linux distributions and was the choice for the Unix command shell on Windows 10.

Note: the bash that ships with macOS is version 3.2.x. There is a newer major version: bash4. However, since bash4 is licensed as GPLv3, Apple still only includes the older bash 3.2. You can download and install bash4 if you want to, but many system administrators stick with the pre-installed version. We will be covering bash 3.2 here.

If you are curious, you can list all available shells on macOS with the following command in Terminal:

$ cat /etc/shells
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/bash
/bin/csh
/bin/ksh
/bin/sh
/bin/tcsh
/bin/zsh

Warning: When entering commands, you have to watch that you type the command exactly as given, including spaces and other special characters. Terminal and shells are a ‘pro’ tool and assume that you know what you are doing. They are not forgiving to errors.

There are a few mechanisms that will usually protect you from ruining your system and data (and thus your day) entirely, but you need to always be careful and check.

Note: As we saw earlier the default prompt shows more information. However, this information (computer name, user name) is different for every Mac and user.
To keep things simple and short, when displaying terminal input and output, we will not show the entire prompt, but only the final character ‘$’. This designates the commands that you should enter in your terminal, without the ‘$’.
Subsequent lines without the leading ‘$’ show output that you should expect from this command. Sometimes the output in this book will be abbreviated to make it fit the layout.

Technically, ‘graphical user interfaces’ (GUI) which display files, folders and programs as icons in windows such as macOS, Windows, KDE, Gnome and even iOS and Android are also shells, since they shield the system internals from the user and vice versa.

However, usually the term ‘shell’ is used for interfaces where the user has to type commands, or ‘command line interfaces’ (CLI).

Shell Scripts

Command Line shells commonly have two major roles. The first is to interpret and execute commands entered in an interactive prompt and deliver the results back to the user. The second role is to interpret and process list of commands, called scripts.

While scripts basically use the same set of commands as the interactive shell, scripts can also use control statements, loops, and variables which makes them a related task, but much more complex.
In this book we will focus on the interactive part of bash.

Next Post: Navigating the File System

Vacation Time, Travel Time, Sneak Preview Time

It is vacation time here in the Scripting OS X headquarters and I will be enjoying time off with family for most of August.

Tomorrow’s weekly news summary will be the last until the end of August. I will keep track of interesting posts, and provide a big summary at the end of the month, though I will also be offline and lot probably miss something interesting. If you find any post or link you think is worthy, you can send to it me on Twitter or the MacAdmins forum (also scriptingosx) and I will make sure to include it.

However, I did not want to go entirely dark during the month. To keep you interested, I will post a few sections from the book I am currently working on over the next week. They are still a bit rough and unfinished.

These sections are from a chapter called ‘Terminal Primer’. They are much more basic than the series of Terminal tips and tricks I ran a few weeks ago. They are targeted to Mac Admins who are new to Terminal and bash. The book this chapter is from, will contain this primer, but also sections on how to use Terminal and bash effectively as a Mac Admin.

Yes, I know, I have promised a book on autopkg. Which I am also still working on. However, while writing the autopkg book I realized that Terminal and bash skills are fundamental for Mac Admins and I feel I need to get it out of the way first.

Now you may ask, “Aren’t there other skills you need to use autopkg effectively, such as property lists, Python, git and Github?” and I will answer, “I know, right!?”

The Terminal Primer sections will be posted (automatically scheduled) on this blog over the next few weeks. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

Terminal Primer

  • First Steps in Terminal
  • Navigating the File System
  • Special Characters
  • Commands
  • Managing Files

Have a great August!

Terminal and SSH Apps for macOS and iOS

This is an addendum to my series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

So far we have used ssh entirely from macOS’ built-in Terminal application. In most cases Terminal is entirely useful and sufficient. However, there are some really useful terminal/ssh applications from third parties. There are also applications for iOS you can use to connect to other computers with ssh.

(iOS and Mac AppStore links are affiliate links.)

Panic’s Prompt 2

Prompt 2 (vendor page) is my favorite iOS application to connect to a shell. (Together with Edovia’s Screens, it forms the backbone of my admin toolkit on iOS.)

Prompt supports external keyboards and keyboard shortcuts. It also supports key based authentication and agent forwarding. It can optionally sync server bookmarks, clips and credentials through the Panic Sync cloud server. Prompt can also detect SSH hosts in the local network that advertise via Bonjour.

Prompt 2 is a one time-purchase for iPhone and iPad, available at the AppStore.

Termius

Termius is a cross-platform SSH solution. It has apps for iOS, macOS, Android, Linux and Windows, so if you need to move across many of these platforms it may be very useful. (I have only tested iOS and macOS.) It is free for basic use, but you can unlock ‘premium’ features for a subscription fee. The premium features include a bookmark and key sync service, clippings, sftp and agent-forwarding.

It has some interesting features such as port forwarding and a the ability to send a public key to a server (like ssh-copy-id) on iOS.

iTerm2

iTerm2 is an alternative terminal application for macOS. It has many features that Terminal.app is lacking or added much later than iTerm2.

Cathode

Cathode is an alternative Terminal application for macOS and SSH client for iOS. Its main claim to fame is to emulate the appearance of old-style cathode ray displays, including 8-bit fonts, distortion and flicker. Cathode is a one-time purchase for macOS and iOS.

Mosh and Blink

mosh (mobile shell) is a protocol for remote shells that is designed for modern mobile connections. mosh can maintain connection to a server even when the client’s ip address and/or connection method changes (i.e. a device switches from Wi-Fi to cell data or vice versa). It uses ssh to establish the connection and security, but then uses its own connection. So you can use ssh keys to authenticate. However mosh requires a server component to be installed on the host.

The iOS mosh client is called Blink and commercially available on the AppStore. Though, since the project is open source, you could also compile the client yourself.

SSH Tunnels

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

We have learned so far that ssh is a really useful and flexible protocol. It can be used to connect securely to a remote shell, or to transfer files securely.

Rather than providing the shell itself, ssh provides a secure way to transmit data to and from the remote shell. In a similar way, ssh can be used to provide access to other remote services as well.

SSH Tunnels with Two Computers

Access to important services are usually blocked behind a firewall or router. Since ssh, when setup correctly, is quite secure, you can usually get access to a server with ssh even when other protocols are blocked. (Though some administrators move ssh access to a different port than the default 22.)

You can use ssh port forwarding or ‘tunneling’ to gain access to other services through ssh.

Imagine you want to use Screen Sharing to connect to a remote Mac (remote.example.com). Screen Sharing on macOS uses the VNC port 5900 to connect to a remote Mac. Since VNC itself is inherently insecure, (mac Screen Sharing adds a few things to make it more secure) this port is blocked by many firewalls.

However, I do have ssh access to remote.example.com. So, how do I tell both systems to ‘tunnel’ the screen sharing traffic through ssh?

(When you test this, remember to enable either ‘Screen Sharing’ or ‘Remote Management’ (i.e Apple Remote Desktop) access in the ‘Sharing’ pane in System Preferences on the remote Mac.)

The tunnel starts on my local machine and ends on remote.example.com at port 5900 (where the screen sharing service is listening on the remote Mac.)

The starting point also needs a port number, and I can basically choose freely. Port numbers under 1000 and over 49000 are reserved for the system and require root privileges. There are also many numbers that are commonly used by certain services (such as 5900 for VNC/Screen Sharing) and may already be in use. I will choose 5901 for the local port.

To connect the local port 5901 to port 5900 on the remote Mac use the following command:

$ ssh -N -L localhost:5901:localhost:5900 remote.example.com

(You can just try this with a second Mac or virtual machine in your network, even without a firewall.)

The syntax of this command is less than obvious. Let’s break it into pieces:

The -N option tells ssh that we do not want to invoke a remote shell or run a remote command.

The -L option creates a local port forwarding setup. This option takes a parameter with three or four parts, separated by colons :. The first pair (localhost:5901) are the tunnel start point. The second pair (localhost:5900) are the remote end point of the tunnel.

The second localhost is resolved on the remote host, so this means port 5900 on the remote host.

The last parameter states the remote host, to connect to, remote.example.com.

This commands tell ssh to connect to remote.example.com and establish a tunnel that transfers traffic from port 5901 on my computer to port 5900 on the remote computer.

Since the origin of my tunnel is usually on my local computer, the first localhost can be omitted, so you only see the origin port.

$ ssh -N -L 5901:localhost:5900 remote.example.com

When you execute the command nothing will happen. You will not even get a new prompt, because the ssh process is running until you cancel it with ctrl-C. Don’t cancel it yet, however, since it needs to run to provide the tunnel.

So, when you open the Screen Sharing application (from /System/Library/CoreServices/Applications/) and connect to localhost:5901 all traffic will be forwarded by sshto port 5900 on the remote Mac.

You can also use the open command to connect with Screen Sharing:

$ open vnc://localhost:5901

You should be able connect with Screen Sharing, even when port 5900 is blocked by a Firewall.

When you are done with the Screen Sharing session, you can end the ssh tunnel process in Terminal with ctrl-C.

SSH triangle

You can also use ssh port to use the remote host as a gateway or ‘jump host’ to a third computer. Imagine you want to use Screen Sharing to connect to secundus.example.com behind a firewall and you only have ssh connection to primus.example.com available. You can tell primus to point the endpoint of an ssh tunnel at secundus with:

$ ssh -N -L 5902:secundus.example.com:5900 primus.example.com

Note: secundus.example.com or whatever host or IP address you enter there will be resolved on the remote host. So you can use NAT IP addresses or .local host names here, even if they do not make sense in the network your work Mac is in. (They do have to make sense on the remote host, though, otherwise you will get an error.)

In the following examples the local IP address 192.168.1.200 or the Bonjour hostname Secundus.local will be resolved on the remote host, even if they don’t work on my local computer:

$ ssh -N -L 5902:192.168.1.200:5900 primus.example.com
$ ssh -N -L 5902:Secundus.local:5900 primus.example.com

Either way, you can then point Screen Sharing at localhost:5902 and it will connect through primus to Screen Sharing on secundus.

Keep in mind, that while the connection from the start point (on your Mac) to the host primus is secured by ssh the connection from primus to secundus is not.

Stumbling over HTTP hosts

In general you can use ssh port forwarding (or tunnels) for any service. Some services however, may introduce extra pitfalls.

For example, I wanted to use ssh port forwarding to gain access to my home router’s web interface. I can use ‘Back to My Mac’ to ssh into one of the iMacs at home, and thought it should be easy to connect to the router with an ssh tunnel:

$ ssh -N -L 8080:192.168.1.1:80 mac.example.com

This seemed to work, but whenever I tried to point a browser to localhost:8080 it couldn’t connect to the web page. The problem here is not the ssh tunnel but the the web server on the router. As part of the http request, the browser sends the name of the domain requested to the web server. This allows web servers to host different pages for different domains. With this request, the browser told the router it wanted the web page for localhost and the router replied with “I don’t serve pages for that host”… (Your router might behave differently.)

With curl I could convince the router to serve me the page with:

curl -H "Host: 192.168.1.1" http://localhost:8080

However, since navigating the web interface of the router with curl is out of the question I had to find a different solution.

Tunnel All the Things!

What if I could send all traffic through the iMac at home?

With the command

$ ssh -N -D 9001 remote.example.com 

I can create a tunnel from my computer (on port 9001) to the remote Mac that acts as a SOCKS proxy. Then I can set the Socks proxy to localhost:9001 in the proxy tab in the Network pane in System Preferences. You probably want to create a new network location for this setup. Then all network traffic will be securely routed through the ssh tunnel to my Mac at home where it can connect to the router.

This can also serve as a temporary VPN solution.

However it is somewhat painful to set up and maintain, so if you start using this more frequently, you probably need to look into a proper VPN service solution (some routers, ironically, provide one…).

Summary

  • you can bypass firewalls and other network blocks by tunnel traffic for any service through a ssh tunnel
  • the command describes which local port to connect to which port on the remote host
  • you can even tell the remote host to connect the end point to a third computer, behind the firewall
  • you can also create a SOCKS proxy with ssh to tunnel all traffic

Previous Post: Transferring Files with ssh