So far in this series of posts on ssh on macOS:

• Quick Introduction to SSH for Mac Admins
• SSH Keys, Part 1: Host Verification
• SSH Keys, Part 2: Client Verification
• Transferring files with SSH
• SSH Tunnels (this post)

Please consider supporting Scripting OS X by buying one of my books!

We have learned so far that ssh is a really useful and flexible protocol. It can be used to connect securely to a remote shell, or to transfer files securely.

Rather than providing the shell itself, ssh provides a secure way to transmit data to and from the remote shell. In a similar way, ssh can be used to provide access to other remote services as well.

SSH Tunnels with Two Computers

Access to important services are usually blocked behind a firewall or router. Since ssh, when setup correctly, is quite secure, you can usually get access to a server with ssh even when other protocols are blocked. (Though some administrators move ssh access to a different port than the default 22.)

You can use ssh port forwarding or ‘tunneling’ to gain access to other services through ssh.

Imagine you want to use Screen Sharing to connect to a remote Mac (remote.example.com). Screen Sharing on macOS uses the VNC port 5900 to connect to a remote Mac. Since VNC itself is inherently insecure, (mac Screen Sharing adds a few things to make it more secure) this port is blocked by many firewalls.

However, I do have ssh access to remote.example.com. So, how do I tell both systems to ‘tunnel’ the screen sharing traffic through ssh?

(When you test this, remember to enable either ‘Screen Sharing’ or ‘Remote Management’ (i.e Apple Remote Desktop) access in the ‘Sharing’ pane in System Preferences on the remote Mac.)

The tunnel starts on my local machine and ends on remote.example.com at port 5900 (where the screen sharing service is listening on the remote Mac.)

The starting point also needs a port number, and I can basically choose freely. Port numbers under 1000 and over 49000 are reserved for the system and require root privileges. There are also many numbers that are commonly used by certain services (such as 5900 for VNC/Screen Sharing) and may already be in use. I will choose 5901 for the local port.

To connect the local port 5901 to port 5900 on the remote Mac use the following command:

$ ssh -N -L localhost:5901:localhost:5900 remote.example.com

(You can just try this with a second Mac or virtual machine in your network, even without a firewall.)

The syntax of this command is less than obvious. Let’s break it into pieces:

The -N option tells ssh that we do not want to invoke a remote shell or run a remote command.

The -L option creates a local port forwarding setup. This option takes a parameter with three or four parts, separated by colons :. The first pair (localhost:5901) are the tunnel start point. The second pair (localhost:5900) are the remote end point of the tunnel.

The second localhost is resolved on the remote host, so this means port 5900 on the remote host.

The last parameter states the remote host, to connect to, remote.example.com.

This commands tell ssh to connect to remote.example.com and establish a tunnel that transfers traffic from port 5901 on my computer to port 5900 on the remote computer.

Since the origin of my tunnel is usually on my local computer, the first localhost can be omitted, so you only see the origin port.

$ ssh -N -L 5901:localhost:5900 remote.example.com

When you execute the command nothing will happen. You will not even get a new prompt, because the ssh process is running until you cancel it with ctrl-C. Don’t cancel it yet, however, since it needs to run to provide the tunnel.

So, when you open the Screen Sharing application (from /System/Library/CoreServices/Applications/) and connect to localhost:5901 all traffic will be forwarded by sshto port 5900 on the remote Mac.

You can also use the open command to connect with Screen Sharing:

$ open vnc://localhost:5901

You should be able connect with Screen Sharing, even when port 5900 is blocked by a Firewall.

When you are done with the Screen Sharing session, you can end the ssh tunnel process in Terminal with ctrl-C.

SSH triangle

You can also use ssh port to use the remote host as a gateway or ‘jump host’ to a third computer. Imagine you want to use Screen Sharing to connect to secundus.example.com behind a firewall and you only have ssh connection to primus.example.com available. You can tell primus to point the endpoint of an ssh tunnel at secundus with:

$ ssh -N -L 5902:secundus.example.com:5900 primus.example.com

Note: secundus.example.com or whatever host or IP address you enter there will be resolved on the remote host. So you can use NAT IP addresses or .local host names here, even if they do not make sense in the network your work Mac is in. (They do have to make sense on the remote host, though, otherwise you will get an error.)

In the following examples the local IP address 192.168.1.200 or the Bonjour hostname Secundus.local will be resolved on the remote host, even if they don’t work on my local computer:

$ ssh -N -L 5902:192.168.1.200:5900 primus.example.com
$ ssh -N -L 5902:Secundus.local:5900 primus.example.com

Either way, you can then point Screen Sharing at localhost:5902 and it will connect through primus to Screen Sharing on secundus.

Keep in mind, that while the connection from the start point (on your Mac) to the host primus is secured by ssh the connection from primus to secundus is not.

Stumbling over HTTP hosts

In general you can use ssh port forwarding (or tunnels) for any service. Some services however, may introduce extra pitfalls.

For example, I wanted to use ssh port forwarding to gain access to my home router’s web interface. I can use ‘Back to My Mac’ to ssh into one of the iMacs at home, and thought it should be easy to connect to the router with an ssh tunnel:

$ ssh -N -L 8080:192.168.1.1:80 mac.example.com

This seemed to work, but whenever I tried to point a browser to localhost:8080 it couldn’t connect to the web page. The problem here is not the ssh tunnel but the the web server on the router. As part of the http request, the browser sends the name of the domain requested to the web server. This allows web servers to host different pages for different domains. With this request, the browser told the router it wanted the web page for localhost and the router replied with “I don’t serve pages for that host”… (Your router might behave differently.)

With curl I could convince the router to serve me the page with:

curl -H "Host: 192.168.1.1" http://localhost:8080

However, since navigating the web interface of the router with curl is out of the question I had to find a different solution.

Tunnel All the Things!

What if I could send all traffic through the iMac at home?

With the command

$ ssh -N -D 9001 remote.example.com 

I can create a tunnel from my computer (on port 9001) to the remote Mac that acts as a SOCKS proxy. Then I can set the Socks proxy to localhost:9001 in the proxy tab in the Network pane in System Preferences. You probably want to create a new network location for this setup. Then all network traffic will be securely routed through the ssh tunnel to my Mac at home where it can connect to the router.

This can also serve as a temporary VPN solution.

However it is somewhat painful to set up and maintain, so if you start using this more frequently, you probably need to look into a proper VPN service solution (some routers, ironically, provide one…).

Summary

• you can bypass firewalls and other network blocks by tunnel traffic for any service through a ssh tunnel
• the command describes which local port to connect to which port on the remote host
• you can even tell the remote host to connect the end point to a third computer, behind the firewall
• you can also create a SOCKS proxy with ssh to tunnel all traffic

Previous Post: Transferring Files with ssh

So far in this series of posts on ssh on macOS:

• Quick Introduction to ssh for Mac Admins
• SSH Keys, Part 1: Host Verification
• SSH Keys, Part 2: Client Verification
• Transferring files with ssh (this post)
• SSH Tunnels (upcoming)

Please consider supporting Scripting OS X by buying one of my books!

In the previous posts we looked how to connect with ssh to a remote computer (host) and how to setup the keys necessary for a secure connection.

Despite the name ssh does not actually provide a shell or command line interface to the host itself. it ‘merely’ provides a secure connection to connect to the default shell on the host itself.

Even this basic use of ssh is already very useful and powerful. It allows to open one or more full command line sessions to remote computers as if we sat at their keyboard. You can also send individual commands and receive and process the results.

However, ssh has a few more powerful tools available.

Copy Files Remotely

You can also use the ssh connection to copy files to and from a remote host.

The command you use for this is scp (secure copy) and it use the same basic syntax as the cp command

$ scp source destination

But, since scp can copy from the local computer to a remote host or vice versa, you usually add a bit more information:

$ scp [[user@]host1:]source [[user@]host2:]destination

(The examples will use a file name hello.txt. To create one quickly, simply type echo "Hello SSH" > hello.txt in Terminal.)
To simplify this, a few examples:

$ scp hello.txt mac.example.com:"~/Documents/"
hello.txt                                     100%   10    10.8KB/s   00:00    

This will copy the local file sample.txt from the current working directory to the remote host’s ~/Documents/ directory. scp will show an ascii progress bar for every file copied. (though with these small files, you will not see much of it) You can suppress the progress display with the -q (quite) option.

For the destination, the colon : separates the hostname (DNS) from the file path.

This command will prompt for the user’s password on the remote host, unless you have added your public key to the remote host’s authorized_keys file.

Since no user name is given before the hostname (separated with an @) scp uses the username that you are logged in with on the local computer. If the remote user has a different name, use:

$ scp hello.txt user@mac.example.com:"~/Documents/"

We do not want the local shell to evaluate the ~ to the local home directory, but want the remote computer to evaluate ~ to the remote user’s home directory, so we have to quote the remote path.

Like cp, when the source is a file and the destination is a directory, then the file will be placed into the destination directory.

If the remote path does not exist, then scp will present an error:

$ scp hello.txt mac.example.com:"~/DoesNotExist/"
scp: /Users/armin/DoesNotExist/: No such file or directory

You can also rename the file while copying:

$ scp hello.txt mac.example.com:"~/Documents/ssh_test.txt"

You can copy files from the remote host to your local machine:

$ scp mac.example.com:"~/Documents/ssh_test.txt" .

In this case we passed . or the current working directory as the destination. You can also pass a local path:

$ scp mac.example.com:"~/Documents/ssh_test.txt" ~/Downloads

Use the -r option to copy all the contents of a directory:

$ scp -r ~/Projects/greatproject/ mac.example.com:"~/Documents/"

Remote-to-remote Copies

You can copy from one remote host to another.

There are two solutions for this. The first will copy the file to the local computer and then back up to the other remote host. You invoke this version of remote-to-remote with the -3 option.

$ scp -3 primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

(I am shortening the full domain names from primus.example.com and secundus.example.com to primus and secundus for simplicity.)

Under most circumstances copying a file down to your Mac and then back up to the other remote host is less than ideal. Imagine you are working from home with your laptop and want to copy a large file from one server at work to another.

The other option is to tell the source remote host to scp the to the other remote host. You could achieve this by sshing to the remote machine and running scp from there. Thankfully, scp is smart enough to attempt exactly that when you type

$ scp primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

This command will probably fail right now. It requires a few things to be set up to work:

• either: client key authentication to be setup from primus to secundus
• or: client key authentication from your local computer to primus and secundus with agent forwarding enabled

The first option is fairly easy to understand. scp will connect to primus and authenticate with your local client key. Then it will tell primus to connect to secundus, authenticating using primus’ client key and the copy the file. It basically works as if you sshed in to primus and ran scp without the extra typing.

There are drawbacks to this. If ssh-agent is not running on primus and does not have the passphrase stored yet, then primus cannot unlock its private key and authenticate to secundus.

Also you basically need to prepare all remote hosts to have keys exchanged between each other, which can be painful, if not impossible to manage.

Agent Forwarding

The second option, called ‘Agent Forwarding’, circumvents these problems. This will tell the first remote host (primus) to ask your local ssh-agent for a key to authenticate to secundus. The system does not actually transfer the private key, but asks ssh-agent on your local computer to encode the authentication challenge for primus.

That way you only need to manage the keys for all remote hosts on your local computer.

You can also use agent forwarding with normal ssh connections. It is not enabled by default, but you can enable it for an ssh session with the -o ForwardAgent=yes option:

$ ssh -o ForwardAgent=yes primus

Since this is hard to type, there is a shortcut:

$ ssh -A primus

When you are logged in to the remote host with agent forwarding enabled, you can then ssh from there to another remote host (secundus) and it will try to use the keys from your local computer’s ssh-agent to authenticate.

$ ssh -A primus
Last login: ...
primus$ ssh secundus
Last login: ...
secundus$ 

This can be a useful strategy if direct access to the second remote host (other.mac.com) is blocked with firewalls.

You can also use this for the scp remote-to-remote copy. Unfortunately, scp does not have a convenient -A option, so you have to use the long parameter form:

$ scp -o ForwardAgent=yes mac.example.com:"~/Documents/ssh_test.txt" other.example.com:"~/Documents/"

When you need agent forwarding regularly for a specific host, you can enable it by default for this particular host in your ~/.ssh/config file. Add the following lines:

Host primus.example.com
    ForwardAgent yes

Note: There are some security concerns with agent forwarding. A user with root access on the intermediate system can gain access to the connection to your local ssh-agent, thus gaining the ability to encrypt with your private keys. Be aware of this in security sensitive environments.

SFTP

If you have many files in a complex file structure , scp can be a little cumbersome. There is a special interactive mode that you can invoke with the sftp command (secure file transfer program).

$ sftp mac.example.com

Note: sftp(according to its man page) is ‘similar’ to ftp but not identical. It also should not be confused with ftps which is ftp over SSL.

Obviously, if you have key authentication setup, sftp will use that.

There are many interactive commands to navigate the local and remote file system and upload (put) and download (get) files. You can look at the details in the sftp man page. However, if you need to use sftp frequently, then you should use a graphical sftp application. There are a large number of SFTP client for macOS and iOS. Here is a list of some popular clients: (AppStore links are affiliate links.)

• Transmit (Mac Version, also iOS AppStore): Aside from a great macOS application, Transmit also comes as an iOS App.
  Update 2018-01-10: Panic is going to cease development and sale of Transmit for iOS for the time being
• Fetch (Mac AppStore): Fetch offers free licenses for educational and charitable organisations
• CyberDuck (Mac AppStore): CyberDuck has an optional command line tool
• MountainDuck (Mac AppStore): Sibling to CyberDuck, but mounts remote servers in the Finder.
• FileZilla: free open source solution

All of these tools connect to many other server protocols other than sftp. However, the advantage of sftp is not just the built-in security, but that you don’t need other software than sshd running on the server.

Summary

Previous Post: Client Verification

• you can use scp [[user@]host:]source [[user@]host:]destination to copy files from or to a remote host over ssh
• you can use agent forwarding to simplify key management in triangle setups
• sftp help managing/transferring multiple files over ssh, there are many UI applications

Next Post: SSH Tunnels

So far in this series of posts on ssh on macOS:

• Quick Introduction to ssh for Mac Admins
• SSH Keys, Part 1: Host Verification
• SSH Keys, Part 2: Client Verification (this post)
• Transferring files with SSH
• SSH Tunnels

Please consider supporting Scripting OS X by buying one of my books!

In the previous post we discussed how ssh uses public and private keys to secure the data sent back and forth, between the remote computer (host) and your machine (client). The keys allow you to be sure that you are talking to the correct host and not an imposter.

However, how does the host know that you are who you claim to be?

Open Sesame…

The default way to authenticate yourself to the ssh host is to give it the correct username and password combination. This is and has been a common practice for computers, servers, websites for a long time and it works for ssh as well.

You will obviously need an account on the host for this to work. In larger organizations the computers will often be connected or bound to a directory service, to centralize the user and group management. If the computers are not bound to a central directory, as is often the case with mobile computers, you can use your management system to install one or more managed users on the system.

In the Sharing Preference Pane, you can control which users or groups of users can use ssh or ‘Remote Login’ to connect to this Mac. Behind the scenes the users and groups that can connect with ssh are in the com.apple.access_ssh group. You can find some scripts to manage the controls in this post.

Keys for the Client

Passwords, and user access control groups already provide a strong security, but have some disadvantages as well:

• you have to re-enter the password on every connection. This is especially tedious when you send individual commands with the ssh hostname command syntax
• automation is insecure because you have to provide the password in the script (in clear text)

With ssh we can use the same asymmetric keys to validate the client to the host, as we did to verify the host.

Sidenote: you will see a lot of actual keys and fingerprints in this post. While sharing the public key should not be a security issue, I deleted them all after finishing this post, just to be safe.

Creating a Key

First, you need to generate the key for your user. This is simple enough with ssh-keygen:

$ ssh-keygen -t rsa -b 2048 -C "Armin RSA 2017-07-06"
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/armin/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/armin/.ssh/id_rsa.
Your public key has been saved in /Users/armin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:jRydlFk3rRTpuKshFvnzon7UnzcYiHLQhKpQvLqPrl0 Armin RSA 2017-07-06
The key's randomart image is:
+---[RSA 2048]----+
|  .     . .+. ++ |
|   o   . +o. .o..|
|  . . . + o  + . |
| . . . o *  . o  |
|  o .   S o...   |
| . .   . =..o.   |
|  . E   =.+  oo. |
| o..   . .o+..o..|
|+oo.   .oo.o.  ..|
+----[SHA256]-----+

This command will create a new key pair of type ‘rsa’ (-t rsa) with a bit length of 2048 (-b 2048) and a comment of Armin RSA 2017-07-06.

RSA is the recommended type for best cross platform compatibility. Mac hosts and clients also support ECDSA, and ED25519. (DSA is in the list but considered insecure and obsolete.) When you connect to other, non-macOS, hosts, ask the administrators for the best key type to use.

The bit length of the key determines how complex the key is. Longer bit lengths are more secure, but require more CPU power for the encryption and decryption. Different key types can have different values for the bit length, read the ssh-keygen man page for details.

The comment is optional, but provides a way to add a label to a key which will help you identify it later. I usually add who created and when and if the key is for a specific purpose.

Next the ssh-keygen tool will ask where to store the key pair. Since this will be our ‘generic’ key, the default name and location ~/.ssh/id_rsa (or id_ecdsa, etc.) is exactly what we want. Sometimes you may have to create a key specifically for a single host, then you can choose a different name here, so you do not overwrite your other keys.

Next the tool will ask you for a passphrase. This will be used to encrypt (lock) the private key, so even if someone gets their hands on the private key file, they will not be able to use it. Since the private key represents your ssh identity it should be protected well. Use a password generator to create a long passphrase. Store the passphrase in a secure place (Password manager, you will need it again) and enter it here (twice).

Finally, the tool will show the files it generated (the private and public key), its fingerprint (with the comment) and a ascii art representation of the key, which is supposed to be more visually recognizable than the alphanumeric fingerprint.

Copying the Key to the Host

When we set up an ssh connection to a host we need to copy its public key into our ssh settings (the known_hosts file). Similarly we need to copy our public key to the host.

You have to copy the public key (i.e. id_rsa.pub) to the host and append its contents to the ~/.ssh/authorized_keys on the host in your user home directory. On some systems, the file may also be called authorized_keys2 and/or be in a different location.

On Mac OS X versions 10.11 and older you can use this command:

$ cat ~/.ssh/id_rsa.pub | ssh [user@]mac.example.com "cat >> ~/.ssh/authorized_keys"

This will pipe the contents of your public key file into ssh, which sends it to the remote host and appends (>>) it to the ~/.ssh/authorized_keys file. (This will fail if the ~/.ssh directory does not exist yet on the host.)

On many Linux systems and on macOS Sierra 10.12 there is a tool to help with this. You can use

$ ssh-copy-id [user@]mac.example.com

By default, the ssh-copy-id command will copy all available public keys to the host. You can tell it to use a specific key with the -i option. Either way, since the key is not yet in place, the command will need to provide you with the host’s password to authenticate to the host.

The ssh-copy-id command gives helpful additional information:

$ ssh-copy-id mac.example.com
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/armin/.ssh/id_ecdsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password:

Number of key(s) added:        1

Now try logging into the machine, with:   "ssh 'mac.example.com'"
and check to make sure that only the key(s) you wanted were added.

Now, when you attempt to connect to the remote host, you will be prompted to unlock your private key:

$ ssh mac.example.com
Enter passphrase for key '/Users/armin/.ssh/id_ecdsa': 
Last login: Thu Jul  6 09:00:28 2017 from [IP Address]
mac:~ armin$

and when you look at the contents of ~/.ssh/authorized_keys you will see your public key listed:

mac:~ armin$ more .ssh/authorized_keys 
ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAC3TR7ZYSjJO01JVNYMeXWNNgglNulQVuGGTnE2JiBULbs9In9uWrFWyqOXodi4XCs7lY92undONnN4wQKumsN/RgBnEDATKZZdh5OPI72ZGrmnPNXVUbU8mjzIJ4LpWfTu0x6nqR9zBu1LPNW9xPoQNwTogh7jIHxvEcYav1slKjeRkg== Armin ECDSA Key 2017-07-06

Note that the comment we added when creating the key is part of the public key (and its fingerprint). Considering that this file can contain multiple keys from different sources, the comment will help a lot in locating the proper key.

When a key is compromised or just not used any more, it should be removed from the host’s authorized_keys file. This file should be audited frequently.

Agent of Keys

When we connected to the remote host after copying the key, we did not have to enter the password for the host, because the key pair confirmed our identity. However, you still had to unlock the private key with the passphrase. You stored that somewhere secure, right? Without the passphrase the key is useless.

This does simplify passwords, because you can use the same key for multiple hosts, and you don’t have to memorize all those passwords. However, entering the passphrase over and over is tedious and error prone, too. Thankfully there is a solution to this. There is a process that can store the passphrase for the duration of your login session. The process is called ssh-agent. However you do not talk directly with this tool but use ssh-add to manage it instead. And on macOS you don’t even have to do that, because ssh-add and ssh-agent are integrated with the macOS Keychain.

The role of the ssh-agent is to hold on to your private key passphrases for the duration of your login session. On other systems (or with Keychain integration disabled) you have to add the passphrase once per login session with

$ ssh-add ~/.ssh/id_rsa

(this will prompt for the passphrase)

You can list the passphrases that ssh-agent is holding with

$ ssh-add -l

and clear all of them with

 $ ssh-add -D

Keychain Integration

Note: on macOS Sierra (10.12), Keychain integration is not enabled by default. To enable it, add these two lines at the top of ~/.ssh/config (create the file if necessary):

AddKeysToAgent yes
UseKeychain yes

With Keychain integration enabled, ssh-agent will also look for passphrases stored in the keychain and then remember them for the rest of the session. The passphrase is stored as an application password with the path to the private key in the name. So as long as the keychain is unlocked, it will find the passphrase in the keychain and use it to unlock the private key (and remember it for the rest of the login session).

On macOS Sierra, the passphrases are not stored in standard login keychain and not synced with iCloud.

Managing Keys with multiple clients

If you have many client Macs you use to connect to the same hosts, you have two choices:

• you can create a new key pair in each client and add the public key to the host’s authorized_keys file
• you can copy the key pair from one client to another

The first option can grow unwieldly quickly when you are managing several Macs and several users.

The second option might expose your private key when you copy them from one client to another. If you send the private key unencrypted over the network it can be intercepted by a man-in-the-middle attack. If you copy it on a USB stick to transfer, you should remember to securely delete any copies, after the transfer.

Your organization will (should) have rules on how to setup and manage keys.

Lockdown

Given proper key management, client verification with keys is safer than with username and passwords. Because of this some setups only allow ssh connections an authorized key.

In this case you cannot use ssh-copy-id to transfer and add your public key to the host. You will have to provide the public key to an administrator to add it for you. The authorized_keys file will probably be in a different, central and locked-down location.

Automation with ssh and keys

You can write scripts that use ssh to communicate with another host. This can yield some powerful workflows. However, if the script uses ssh repeatedly, you do not want to have to enter the password over and over. If the script runs within a user session, ssh-agent can provide the passphrase when necessary. But when a script has to run in background on a schedule without user interaction, ssh-agent will not be available.

In this case you can create a key with an empty passphrase. This key will be unlocked automatically. Obviously, such a key would need to be protected very well, and its use should be audited closely. A key with an empty passphrase is the security equivalent of leaving the house key under the doormat. However, for some automated workflows, it may be the best solution.

Summary

• previous post: ssh uses public/private keys to verify the host and encrypt the connection
• ssh can also use public/private keys to verify the user connecting to the host
• you generate a key with ssh-keygen
• the public key is added to the remote host’s ~/.ssh/authorized_keys
• the private key remains on the client and is locked with a passphrase
• ssh-agent and Keychain integration on macOS can simplify access to the private key’s passphrase

Next post: Transferring Files with ssh

So far in this series of posts on ssh on macOS:

• Quick Introduction to ssh for Mac Admins
• SSH Keys, Part 1: Host Verification (this post)
• SSH Keys, Part 2: Client Verification
• Transferring files with ssh
• SSH Tunnels

Please consider supporting Scripting OS X by buying one of my books!

In the earlier post, we talked about basic ssh setup and use.

We briefly mentioned ssh is securely encrypted. We encountered a cryptic prompt (pun intended) when connecting to the machine for the first time:

The authenticity of host 'mac.example.com (IP address)' can't be established.
ECDSA key fingerprint is SHA256:abcdefghhijklmonpqrstuvwxyz.
Are you sure you want to continue connecting (yes/no)? 

At that point we just wanted to connect and ignored this prompt. However, this is a crucial part of the ssh infrastructure and understanding it will help you use ssh securely and to its full potential.

Asymmetric Key Encryption

This is what ssh uses to encrypt traffic. It uses a two part encryption key to encode the data transmitted. The first part of the key is private and kept securely on the ssh host (the remote computer you want to log in to) and the second part is public and shared to the users who want to connect.

Data encrypted with one key can be decrypted with the other and vice versa. This way data can be kept secure between the client and the host.

When you connect for the first the remote host will send its public key to your computer. This is convenient, but opens the chance for malicious computer to impersonate the host you actually want to reach and set itself as a ‘man in the middle’, passing data back and forth and decrypting it in between.

OK, how does this work with SSH?

The key fingerprint the system displays on first connection is a ‘hash’ of the public key.

Using this fingerprint, you can verify that the DNS address or IP you are connecting to, is really the computer you want to connect to. To verify, you would need obtain the fingerprint from the host though some other means and compare.

On macOS the host keys are stored in /private/etc/ssh/ along with some other files required for ssh configuration. The key filenames have the format ssh_host_ABC_key where ABC is the key encryption type. On my Sierra Mac there are keys for dsa, ecdsa, rsa and ed25519. There two files for every type, the private key (no file extension) and the public key, with the .pub extension.

ECDSA (Elliptic Curve Digital Signature Algorithm – Wikipedia) is the default type of key ssh uses on macOS. The other keys are present for compatibility with other platforms and older versions of macOS/OS X. Usually the client and the server will negotiate which keys to use from the available options.

These keys are generated on every host, when the ssh server process starts for the first time. These keys are unique to this host.

When you look closely at the key files, you will see that the private keys can only be read by root, no other users. Private keys are like really important passwords and have to be kept safe. If someone can obtain both keys, they can impersonate this host.

The public keys are the part that are meant to be shared and can be read by any user.

Fingerprints

You can generate the fingerprint from a public key with the ssh-keygen command:

$ ssh-keygen -l -f /private/etc/ssh/ssh_host_ecdsa_key.pub 
256 SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo no comment (ECDSA)

ECDSA (Elliptic Curve Digital Signature Algorithm – Wikipedia) is the default type of key ssh uses on macOS. The other keys are present for compatibility with other platforms and older versions of macOS/OS X.

A security conscious admin would go to the remote machine in question (or ask another user/admin who has access to the server/machine) and generate the fingerprint. Then go back to the Mac from where he wants to remotely connect with ssh and compare the fingerprint shown on first connection:

$ ssh mac.example.com
The authenticity of host 'mac.example.com ([IP Address])' can't be established.
ECDSA key fingerprint is SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo.
Are you sure you want to continue connecting (yes/no)? 

Since the key and the fingerprint are unique, you can be sure you are talking to the correct remote machine and there is no other malicious computer impersonating the remote machine on the network.

Making a list, checking it twice…

Once you confirm the prompt to connect to a new host, its public key will be added to the file ~/.ssh/known_hosts. This file will contain one line per remote host. Each line will contain the hostname, the IP address, the key type and the public key data itself. You can view the file in a text editor, or you can use the ssh-keygen tool to search this file for a particular host:

$ ssh-keygen -F mac.example.com
# Host mac.example.com found: line 16 
mac.example.com,[IP Address] ecdsa-sha2-nistp256 AAAA1234...=

Once the key is stored in your known_hosts file, ssh can detect if the public key presented by a remote ssh host changes and warn you. The warning is not subtle:

$ ssh mac.example.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for mac.example.com has changed,
and the key for the corresponding IP address [IP Address]
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo.
Please contact your system administrator.
Add correct host key in /Users/armin/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/armin/.ssh/known_hosts:16
ECDSA host key for mac.example.com has changed and you have requested strict checking.
Host key verification failed.

In most cases this means that IP addresses have changed. In the worst case, someone is maliciously attempting to impersonate an ssh host in your network, probably in an attempt to gather login information.

As administrators, there is another common reason for this warning. When you re-install or re-image a client Mac, it will generate new ssh keys. So every time you re-image and re-connect to a test machine, your Mac will compare the old keys in known_hosts with a newly generated, different key on the freshly imaged Mac and bring up this warning.

You can remove the old keys from the ~/.ssh/known_hosts file directly in a text editor or use the ssh-keygen command to remove out of date keys:

$ ssh-keygen -R mac.example.com
# Host mac.example.com found: line 16
/Users/armin/.ssh/known_hosts updated.
Original contents retained as /Users/armin/.ssh/known_hosts.old

The hostname is used to identify hosts and keys in this file. If you connect to the client with different hostnames, DNS aliases, or you alternate between the ‘proper’ DNS name mac.example.com, partial DNS name mac, the Bonjour name (i.e. Mac.local), or the IP address, you may end up with multiple entries for the same host in known_hosts. Usually this is not a problem, but is useful to know when cleaning out stale keys.

If you have the public key for a host, you can pre-add it to your known_hosts file. The format you have to use is

mac.example.com,[IP Address] ecdsa-sha2-nistp256 AAAA1234...=

Don’t copy these keys

If you use ‘Golden Master’ imaging on macOS (you really shouldn’t, for several reasons, this being one of them), you might inadvertently copy the host ssh keys from your Golden Master to all the other clients. While this does not break how ssh works, it will undermine some of its security. You can simply delete the existing keys on the Golden Master and new keys will be created on the clients when needed.

Remember there are other files in /private/etc/ssh/ that are required for ssh to work which you should not delete.

Locking down even more

You can increase or decrease the host key security with the StrictHostKeyChecking setting in the ~/.ssh/config file for a particular user or the/private/etc/ssh/ssh_config` file for all users on a system.

The default value is ask which means that ssh will ask what to do with an unknown key from a host. This is the behavior described above.

You can increase security by changing this setting to yes which means ssh will refuse to connect to a host which is not listed in ~/.ssh/known_hosts:

$ ssh stranger.example.com
No ECDSA host key is known for stranger.example.com and you have requested strict checking.
Host key verification failed.

With this setting you will have to manually obtain a host’s public key and add it to your known_hosts file.

You can also set StrictHostKeyChecking to no. With this setting, unknown hosts will automatically be added to the known_hosts file and changed public keys will automatically be replaced. This could be useful in a very dynamic, low-security environment (i.e. a classroom where Macs are re-imaged frequently). However, it does undermine the security of ssh and you will not notice any attempts to insert malicious hosts. In general you should not use this setting.

Recap

• the ssh host (remote machine) has a set of encryption keys in /private/etc/ssh/
• the public key is sent to the client. A fingerprint of the public key is presented with a prompt wether you should connect. You should use the public key fingerprint to verify the host’s authenticity on first connect.
• generate the fingerprint from the public key on the host with ssh-keygen -l -f keyfile.pub
• the public keys of approved hosts are stored in ~/.ssh/known_hosts
• use ssh-keygen -R host to remove old or unused keys
• you can control this behavior with the StrictHostKeyChecking setting in the config file

When you verify the fingerprint of the host, you can be sure of its identity. The host however, cannot verify a client’s identity in this setup. This is why you still have to enter a password every time you log in with ssh remotely.

We will discuss how to improve on that in the next post : SSH Keys, Part 2: Client Verification

• Quick Introduction to ssh for Mac Admins (this post)
• SSH Keys, Part 1: Host Verification
• SSH Keys, Part 2: Client Verification
• Transferring files with ssh
• SSH Tunnels

Please consider supporting Scripting OS X by buying one of my books!

What is SSH?

SSH is short for ‘Secure SHell’. It is a protocol that allows you to open a shell (terminal session) on another remote computer over the network. On macOS you will usually initiate an SSH session with the Terminal application, though there are other third party applications for macOS and iOS that support the SSH protocol as well.

On macOS SSH access is sometimes called ‘Remote Login’.

As the name implies, connections over SSH are encrypted and secure. This may not seem unusual today, but it did replace many unsecured protocols such as telnet and rlogin.

SSH is supported on many platforms. You can use SSH to connect to Macs as well as any other computer which supports SSH and has it enabled. Of course, when you SSH to a different system, the environment you get through SSH might be very different (i.e. not bash, different commands, etc.), so be sure you understand the remote system.

Enable SSH access

By default SSH access is disabled on macOS. Before you can connect to a Mac with SSH you need to enable access. In the UI you can do this in System Preferences > Sharing, by enabling the ‘Remote Login’ option. Here you can also control whether all users on the Mac can get SSH access or just some of them.

The Sharing pane will also show the current hostname of this computer:

Computers on your local network can access your computer at:
client.example.com

You can also enable SSH access from the command line with

$ sudo systemsetup -setremotelogin on

and turn it off again with

$ sudo systemsetup -setremotelogin off

Turning SSH off with this command helpfully reminds you that if you are currently connected remotely, you will disconnect with this command and have to login locally to re-enable. To override this helpful notice, you can add the -f option:

$ sudo systemsetup -f -setremotelogin off

And if you want to know which hostname a Mac thinks it is, use the hostname command:

$ hostname
client.example.com

Note: If you want to control which users have access with a script, I have an example in this post. This post was written for Munki but should be fairly easy to adapt to other systems.

Connect with SSH

To connect to a machine from another Mac (the one you are working on) is simple enough. Open Terminal and enter:

$ ssh username@client.example.com

or alternatively (different syntax, same result, which one you prefer is a matter of taste)

$ ssh -l username client.example.com

If you do not give a username, ssh will use the short name you are currently logged in as. That is a useful shortcut if the username is the same on both systems.

When you connect to a remote machine for the first time you will get this prompt:

ssh client.example.com
The authenticity of host 'client.example.com (IP address)' can't be established.
ECDSA key fingerprint is SHA256:abcdefghhijklmonpqrstuvwxyz.
Are you sure you want to continue connecting (yes/no)? 

(Your actual fingerprint will look differently.) This prompt tells you that your ssh does not recognize this host and gives you a chance to not connect. For now, type yes to confirm. This will add the host to the list of known hosts, so the prompt will not return when you connect again. We will discuss keys and security in a later post.

Next ssh will prompt you for the password on the remote computer. Once you enter that you will get the prompt and have a secure shell to the remote computer. Any command you enter now, will be run on the remote computer.

You can connect with ssh when no user is logged in, i.e. the Mac is sitting at the login window. You can even connect with one user, while a different user is logged into the Mac with a UI session. In these cases, commands that interact with the UI, will fail, since the UI is either not running, or running as a different user. Most common examples are open and osascript (AppleScript).

$ open .
LSOpenURLsWithRole() failed with error -10810 for the file /Users/armin.

If you happen to be logged in on the same Mac with the same user, then the command will work, but open will open and display the Finder window on the remote Mac, not the Mac you are working on, which can be confusing.

Ending it

To end the remote session, just use the exit command.

$ exit
logout
Connection to client.example.com closed.

This will return you to the shell on your Mac where you executed the ssh command.

SSH Shortcut Files

If you frequently connect to certain hosts you may want to create an ssh short cut file, which you can then just double click or invoke with spotlight.

Execute Just One Command

Instead of opening a remote shell you can use ssh to just execute just a single command on the remote host:

$ ssh client.example.com sw_vers
Password:
ProductName:    Mac OS X
ProductVersion: 10.12.5
BuildVersion:   16F73

The ssh command will prompt for the password and the print the output of the command from the remote machine. This can be very useful. You can combine multiple commands with ;

$ ssh client.example.com hostname; sw_vers; system_profiler SPHardwareDataType

In some case the command you want to execute remotely can prompt for information, usually a password. You can add the -t option to make ssh use an interactive shell:

$ ssh -t client.example.com sudo installer -pkg ~/Downloads/myinstaller.pkg -tgt / -verbose

This will prompt twice for the password. The first time to establish the remote connection and the second time for sudo on the remote machine.

Sending single commands with ssh can be useful for automating workflows in scripts. However, the requirement to keep entering passwords will be very detrimental to automation. It will also be annoying when you frequently connect to specific remote Macs.

We will discuss how the key security works and how it can replace passwords in the following posts.