Transferring Files with SSH

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

In the previous posts we looked how to connect with ssh to a remote computer (host) and how to setup the keys necessary for a secure connection.

Despite the name ssh does not actually provide a shell or command line interface to the host itself. it ‘merely’ provides a secure connection to connect to the default shell on the host itself.

Even this basic use of ssh is already very useful and powerful. It allows to open one or more full command line sessions to remote computers as if we sat at their keyboard. You can also send individual commands and receive and process the results.

However, ssh has a few more powerful tools available.

Copy Files Remotely

You can also use the ssh connection to copy files to and from a remote host.

The command you use for this is scp (secure copy) and it use the same basic syntax as the cp command

$ scp source destination

But, since scp can copy from the local computer to a remote host or vice versa, you usually add a bit more information:

$ scp [[user@]host1:]source [[user@]host2:]destination

(The examples will use a file name hello.txt. To create one quickly, simply type echo "Hello SSH" > hello.txt in Terminal.)
To simplify this, a few examples:

$ scp hello.txt mac.example.com:"~/Documents/"
hello.txt                                     100%   10    10.8KB/s   00:00

This will copy the local file sample.txt from the current working directory to the remote host’s ~/Documents/ directory. scp will show an ascii progress bar for every file copied. (though with these small files, you will not see much of it) You can suppress the progress display with the -q (quite) option.

For the destination, the colon : separates the hostname (DNS) from the file path.

This command will prompt for the user’s password on the remote host, unless you have added your public key to the remote host’s authorized_keys file.

Since no user name is given before the hostname (separated with an @) scp uses the username that you are logged in with on the local computer. If the remote user has a different name, use:

$ scp hello.txt user@mac.example.com:"~/Documents/"

We do not want the local shell to evaluate the ~ to the local home directory, but want the remote computer to evaluate ~ to the remote user’s home directory, so we have to quote the remote path.

Like cp, when the source is a file and the destination is a directory, then the file will be placed into the destination directory.

If the remote path does not exist, then scp will present an error:

$ scp hello.txt mac.example.com:"~/DoesNotExist/"
scp: /Users/armin/DoesNotExist/: No such file or directory

You can also rename the file while copying:

$ scp hello.txt mac.example.com:"~/Documents/ssh_test.txt"

You can copy files from the remote host to your local machine:

$ scp mac.example.com:"~/Documents/ssh_test.txt" .

In this case we passed . or the current working directory as the destination. You can also pass a local path:

$ scp mac.example.com:"~/Documents/ssh_test.txt" ~/Downloads

Use the -r option to copy all the contents of a directory:

$ scp -r ~/Projects/greatproject/ mac.example.com:"~/Documents/"

Remote-to-remote Copies

You can copy from one remote host to another.

There are two solutions for this. The first will copy the file to the local computer and then back up to the other remote host. You invoke this version of remote-to-remote with the -3 option.

$ scp -3 primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

(I am shortening the full domain names from primus.example.com and secundus.example.com to primus and secundus for simplicity.)

Under most circumstances copying a file down to your Mac and then back up to the other remote host is less than ideal. Imagine you are working from home with your laptop and want to copy a large file from one server at work to another.

The other option is to tell the source remote host to scp the to the other remote host. You could achieve this by sshing to the remote machine and running scp from there. Thankfully, scp is smart enough to attempt exactly that when you type

$ scp primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

This command will probably fail right now. It requires a few things to be set up to work:

  • either: client key authentication to be setup from primus to secundus
  • or: client key authentication from your local computer to primus and secundus with agent forwarding enabled

The first option is fairly easy to understand. scp will connect to primus and authenticate with your local client key. Then it will tell primus to connect to secundus, authenticating using primus’ client key and the copy the file. It basically works as if you sshed in to primus and ran scp without the extra typing.

There are drawbacks to this. If ssh-agent is not running on primus and does not have the passphrase stored yet, then primus cannot unlock its private key and authenticate to secundus.

Also you basically need to prepare all remote hosts to have keys exchanged between each other, which can be painful, if not impossible to manage.

Agent Forwarding

The second option, called ‘Agent Forwarding’, circumvents these problems. This will tell the first remote host (primus) to ask your local ssh-agent for a key to authenticate to secundus. The system does not actually transfer the private key, but asks ssh-agent on your local computer to encode the authentication challenge for primus.

That way you only need to manage the keys for all remote hosts on your local computer.

You can also use agent forwarding with normal ssh connections. It is not enabled by default, but you can enable it for an ssh session with the -o ForwardAgent=yes option:

$ ssh -o ForwardAgent=yes primus

Since this is hard to type, there is a shortcut:

$ ssh -A primus

When you are logged in to the remote host with agent forwarding enabled, you can then ssh from there to another remote host (secundus) and it will try to use the keys from your local computer’s ssh-agent to authenticate.

$ ssh -A primus
Last login: ...
primus$ ssh secundus
Last login: ...
secundus$

This can be a useful strategy if direct access to the second remote host (other.mac.com) is blocked with firewalls.

You can also use this for the scp remote-to-remote copy. Unfortunately, scp does not have a convenient -A option, so you have to use the long parameter form:

$ scp -o ForwardAgent=yes mac.example.com:"~/Documents/ssh_test.txt" other.example.com:"~/Documents/"

When you need agent forwarding regularly for a specific host, you can enable it by default for this particular host in your ~/.ssh/config file. Add the following lines:

Host primus.example.com
    ForwardAgent yes

Note: There are some security concerns with agent forwarding. A user with root access on the intermediate system can gain access to the connection to your local ssh-agent, thus gaining the ability to encrypt with your private keys. Be aware of this in security sensitive environments.

SFTP

If you have many files in a complex file structure , scp can be a little cumbersome. There is a special interactive mode that you can invoke with the sftp command (secure file transfer program).

$ sftp mac.example.com

Note: sftp(according to its man page) is ‘similar’ to ftp but not identical. It also should not be confused with ftps which is ftp over SSL.

Obviously, if you have key authentication setup, sftp will use that.

There are many interactive commands to navigate the local and remote file system and upload (put) and download (get) files. You can look at the details in the sftp man page. However, if you need to use sftp frequently, then you should use a graphical sftp application. There are a large number of SFTP client for macOS and iOS. Here is a list of some popular clients: (AppStore links are affiliate links.)

All of these tools connect to many other server protocols other than sftp. However, the advantage of sftp is not just the built-in security, but that you don’t need other software than sshd running on the server.

Summary

Previous Post: Client Verification

  • you can use scp [[user@]host:]source [[user@]host:]destination to copy files from or to a remote host over ssh
  • you can use agent forwarding to simplify key management in triangle setups
  • sftp help managing/transferring multiple files over ssh, there are many UI applications

Next Post: SSH Tunnels

Published by

ab

Mac Admin, Consultant, and Author