SSH Keys, Part 1: Host Verification

So far in this series of posts on ssh on macOS:

Quick Introduction to ssh for Mac Admins
SSH Keys, Part 1: Host Verification (this post)
SSH Keys, Part 2: Client Verification
Transferring files with ssh
SSH Tunnels

Please consider supporting Scripting OS X by buying one of my books!

In the earlier post, we talked about basic ssh setup and use.

We briefly mentioned ssh is securely encrypted. We encountered a cryptic prompt (pun intended) when connecting to the machine for the first time:

The authenticity of host 'mac.example.com (IP address)' can't be established.
ECDSA key fingerprint is SHA256:abcdefghhijklmonpqrstuvwxyz.
Are you sure you want to continue connecting (yes/no)?

At that point we just wanted to connect and ignored this prompt. However, this is a crucial part of the ssh infrastructure and understanding it will help you use ssh securely and to its full potential.

Asymmetric Key Encryption

This is what ssh uses to encrypt traffic. It uses a two part encryption key to encode the data transmitted. The first part of the key is private and kept securely on the ssh host (the remote computer you want to log in to) and the second part is public and shared to the users who want to connect.

Data encrypted with one key can be decrypted with the other and vice versa. This way data can be kept secure between the client and the host.

When you connect for the first the remote host will send its public key to your computer. This is convenient, but opens the chance for malicious computer to impersonate the host you actually want to reach and set itself as a ‘man in the middle’, passing data back and forth and decrypting it in between.

OK, how does this work with SSH?

The key fingerprint the system displays on first connection is a ‘hash’ of the public key.

Using this fingerprint, you can verify that the DNS address or IP you are connecting to, is really the computer you want to connect to. To verify, you would need obtain the fingerprint from the host though some other means and compare.

On macOS the host keys are stored in /private/etc/ssh/ along with some other files required for ssh configuration. The key filenames have the format ssh_host_ABC_key where ABC is the key encryption type. On my Sierra Mac there are keys for dsa, ecdsa, rsa and ed25519. There two files for every type, the private key (no file extension) and the public key, with the .pub extension.

ECDSA (Elliptic Curve Digital Signature Algorithm – Wikipedia) is the default type of key ssh uses on macOS. The other keys are present for compatibility with other platforms and older versions of macOS/OS X. Usually the client and the server will negotiate which keys to use from the available options.

These keys are generated on every host, when the ssh server process starts for the first time. These keys are unique to this host.

When you look closely at the key files, you will see that the private keys can only be read by root, no other users. Private keys are like really important passwords and have to be kept safe. If someone can obtain both keys, they can impersonate this host.

The public keys are the part that are meant to be shared and can be read by any user.

Fingerprints

You can generate the fingerprint from a public key with the ssh-keygen command:

$ ssh-keygen -l -f /private/etc/ssh/ssh_host_ecdsa_key.pub 
256 SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo no comment (ECDSA)

A security conscious admin would go to the remote machine in question (or ask another user/admin who has access to the server/machine) and generate the fingerprint. Then go back to the Mac from where he wants to remotely connect with ssh and compare the fingerprint shown on first connection:

$ ssh mac.example.com
The authenticity of host 'mac.example.com ([IP Address])' can't be established.
ECDSA key fingerprint is SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo.
Are you sure you want to continue connecting (yes/no)?

Since the key and the fingerprint are unique, you can be sure you are talking to the correct remote machine and there is no other malicious computer impersonating the remote machine on the network.

Making a list, checking it twice…

Once you confirm the prompt to connect to a new host, its public key will be added to the file ~/.ssh/known_hosts. This file will contain one line per remote host. Each line will contain the hostname, the IP address, the key type and the public key data itself. You can view the file in a text editor, or you can use the ssh-keygen tool to search this file for a particular host:

$ ssh-keygen -F mac.example.com
# Host mac.example.com found: line 16 
mac.example.com,[IP Address] ecdsa-sha2-nistp256 AAAA1234...=

Once the key is stored in your known_hosts file, ssh can detect if the public key presented by a remote ssh host changes and warn you. The warning is not subtle:

$ ssh mac.example.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for mac.example.com has changed,
and the key for the corresponding IP address [IP Address]
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:Izk7/TbLsH6aegfaGGs2hHQMAkDWkXR93gr1I/FKobo.
Please contact your system administrator.
Add correct host key in /Users/armin/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/armin/.ssh/known_hosts:16
ECDSA host key for mac.example.com has changed and you have requested strict checking.
Host key verification failed.

In most cases this means that IP addresses have changed. In the worst case, someone is maliciously attempting to impersonate an ssh host in your network, probably in an attempt to gather login information.

As administrators, there is another common reason for this warning. When you re-install or re-image a client Mac, it will generate new ssh keys. So every time you re-image and re-connect to a test machine, your Mac will compare the old keys in known_hosts with a newly generated, different key on the freshly imaged Mac and bring up this warning.

You can remove the old keys from the ~/.ssh/known_hosts file directly in a text editor or use the ssh-keygen command to remove out of date keys:

$ ssh-keygen -R mac.example.com
# Host mac.example.com found: line 16
/Users/armin/.ssh/known_hosts updated.
Original contents retained as /Users/armin/.ssh/known_hosts.old

The hostname is used to identify hosts and keys in this file. If you connect to the client with different hostnames, DNS aliases, or you alternate between the ‘proper’ DNS name mac.example.com, partial DNS name mac, the Bonjour name (i.e. Mac.local), or the IP address, you may end up with multiple entries for the same host in known_hosts. Usually this is not a problem, but is useful to know when cleaning out stale keys.

If you have the public key for a host, you can pre-add it to your known_hosts file. The format you have to use is

mac.example.com,[IP Address] ecdsa-sha2-nistp256 AAAA1234...=

Don’t copy these keys

If you use ‘Golden Master’ imaging on macOS (you really shouldn’t, for several reasons, this being one of them), you might inadvertently copy the host ssh keys from your Golden Master to all the other clients. While this does not break how ssh works, it will undermine some of its security. You can simply delete the existing keys on the Golden Master and new keys will be created on the clients when needed.

Remember there are other files in /private/etc/ssh/ that are required for ssh to work which you should not delete.

Locking down even more

You can increase or decrease the host key security with the StrictHostKeyChecking setting in the ~/.ssh/config file for a particular user or the/private/etc/ssh/ssh_config` file for all users on a system.

The default value is ask which means that ssh will ask what to do with an unknown key from a host. This is the behavior described above.

You can increase security by changing this setting to yes which means ssh will refuse to connect to a host which is not listed in ~/.ssh/known_hosts:

$ ssh stranger.example.com
No ECDSA host key is known for stranger.example.com and you have requested strict checking.
Host key verification failed.

With this setting you will have to manually obtain a host’s public key and add it to your known_hosts file.

You can also set StrictHostKeyChecking to no. With this setting, unknown hosts will automatically be added to the known_hosts file and changed public keys will automatically be replaced. This could be useful in a very dynamic, low-security environment (i.e. a classroom where Macs are re-imaged frequently). However, it does undermine the security of ssh and you will not notice any attempts to insert malicious hosts. In general you should not use this setting.

Recap

the ssh host (remote machine) has a set of encryption keys in /private/etc/ssh/
the public key is sent to the client. A fingerprint of the public key is presented with a prompt wether you should connect. You should use the public key fingerprint to verify the host’s authenticity on first connect.
generate the fingerprint from the public key on the host with ssh-keygen -l -f keyfile.pub
the public keys of approved hosts are stored in ~/.ssh/known_hosts
use ssh-keygen -R host to remove old or unused keys
you can control this behavior with the StrictHostKeyChecking setting in the config file

When you verify the fingerprint of the host, you can be sure of its identity. The host however, cannot verify a client’s identity in this setup. This is why you still have to enter a password every time you log in with ssh remotely.

We will discuss how to improve on that in the next post : SSH Keys, Part 2: Client Verification