Terminal Primer – Part 1 – First Steps in Terminal

If you like this series and want to learn Terminal and the shell on macOS in more detail, get my book “macOS Terminal and Shell

Terminal Application

On macOS you can find the Terminal application in /Applications/Utilities/. There is a shortcut in Finder’s Go menu to get to the Utilities folder or use the keystroke ⌘⇧U. Terminal has a distinct black icon with a white prompt.

However, since you are reading this book, you are planning to use Terminal regularly. In this case, you really want to add the Terminal application to your Dock.

First Prompt

Once you have opened Terminal, it will show you a new window, which is white with black text by default. A short message will show the last login and then a line with the default prompt:

Mac:~ armin$

At first it shows the computer name (as set in the ‘Sharing’ pane in System Preferences. After the colon ‘:’ it shows a tilde ‘~’. Then your user short name, followed by a dollar sign ‘$’. Finally a dark gray block, the cursor, waiting for your input.

Note: if you have used Terminal before, you may have changed the default look of the Terminal and prompt. The functionality remains the same, no matter how it looks.

You can enter your first command. Type the letters p, w and d and then hit the return key ↩︎

Mac:~ armin$ pwd
/Users/armin

You have typed the pwd command and executed it with the return key ↩. The return key confirms a command and runs it.

When you mistype a command you can delete the last character with the delete or backspace key ←. You can also move the cursor through the characters you entered with the left and right arrow keys.

When you hit return with a wrong command or a command with a typo the shell will complain that it cannot find the command:

Mac:~ armin$ pws
-bash: pws: command not found

Terminal Terminology

We have already encountered a few different terms which may be very confusing in the beginning. The bad news is there are more terms, the good news is that it will all make sense eventually.

All of these terms date back to the days when computing started with big mainframe computers. Since computers were scarce, complex and expensive, many users had to share access. They did this by typing and reading on a ‘terminal’ or ‘console’ — a dedicated device talking to the main frame. Terminals and consoles may have looked somewhat like like modern workstations and computers, but did not have their own CPU. They would just allow to enter and send commands and receive and display the results.

Very early in the history of computing, terminals used mechanical printers or typewriters to show the results, these were called tele-typewriters or tty. The protocol that the mainframe used to communicate with the typewriters, was named tty as well. The protocol and its name have remained, long after the mechanical typewriters are gone. Later the mechanical interfaces where replaced by terminals with electric keyboards and cathode ray screens.

As computers got smaller, more powerful and cheaper it became possible to have one on everybody’s desk. Even so, it was (and is) still convenient to run a shell, either locally or to connect to larger mainframes (servers). To do that you would open a program (or application) that ran a ‘virtual terminal.’

The Terminal application on macOS is such a virtual terminal program. To confuse things a little, macOS also has an application called Console. The Console application on macOS is not a virtual terminal or virtual console, but used to display and filter log files.

The terminal (whether virtual or real) only provides a means (virtual or mechanical) to enter and display text. It will display a ‘prompt’ which tells the user, that the system is ready and the user can enter a command, and a ‘cursor’ which shows where typed text will be displayed or inserted.

Shells

There is another program which interprets the text, executes code and sends the output generated back to the terminal.

This program (or class of programs) is called a ‘shell.’ A shell protects the user from the dangerous, complicated parts of the system and abstracts differences from one system to the other. Another way of looking at it, is that a shell protects the vulnerable, fragile parts of the the system from the user.

There are many different shell programs. One of the earliest shell programs was call just ‘shell’ or sh for short. Surprisingly, sh is still around after nearly fifty years.

On macOS the default shell is called bash for ‘bourne again shell.’ This comes from the fact that it was developed as a replacement for bsh or ‘Bourne shell,’ which was named after its main developer Stephen Bourne.

You will notice that unix shells and commands often exhibit a particular style of pun humor. ‘bash’ is merely the beginning.

Today, there are many different shells. Different shells have different ways of interpreting commands. The choice of shell is a personal preference and can be (as many things in computing) the cause of passionate argument.

Note: As system administrators, the choice of shell is not just determined by preference, but also practicality. There are a set of pre-installed shells on macOS and while it is possible to install additional shells, it increases complexity and management effort. Also when you are sharing commands and scripts with fellow administrators, bash is the commonly agreed upon shell.

bash is available for most platforms and operating systems and is the default shell on many systems. Because of this prevalence bash is a good choice for your first shell.
bash has been the default shell for Mac since Mac OS X 10.3 (Panther). It is also the default shell on most Linux distributions and was the choice for the Unix command shell on Windows 10.

Note: the bash that ships with macOS is version 3.2.x. There is a newer major version: bash4. However, since bash4 is licensed as GPLv3, Apple still only includes the older bash 3.2. You can download and install bash4 if you want to, but many system administrators stick with the pre-installed version. We will be covering bash 3.2 here.

If you are curious, you can list all available shells on macOS with the following command in Terminal:

$ cat /etc/shells
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/bash
/bin/csh
/bin/ksh
/bin/sh
/bin/tcsh
/bin/zsh

Warning: When entering commands, you have to watch that you type the command exactly as given, including spaces and other special characters. Terminal and shells are a ‘pro’ tool and assume that you know what you are doing. They are not forgiving to errors.

There are a few mechanisms that will usually protect you from ruining your system and data (and thus your day) entirely, but you need to always be careful and check.

Note: As we saw earlier the default prompt shows more information. However, this information (computer name, user name) is different for every Mac and user.
To keep things simple and short, when displaying terminal input and output, we will not show the entire prompt, but only the final character ‘$’. This designates the commands that you should enter in your terminal, without the ‘$’.
Subsequent lines without the leading ‘$’ show output that you should expect from this command. Sometimes the output in this book will be abbreviated to make it fit the layout.

Technically, ‘graphical user interfaces’ (GUI) which display files, folders and programs as icons in windows such as macOS, Windows, KDE, Gnome and even iOS and Android are also shells, since they shield the system internals from the user and vice versa.

However, usually the term ‘shell’ is used for interfaces where the user has to type commands, or ‘command line interfaces’ (CLI).

Shell Scripts

Command Line shells commonly have two major roles. The first is to interpret and execute commands entered in an interactive prompt and deliver the results back to the user. The second role is to interpret and process list of commands, called scripts.

While scripts basically use the same set of commands as the interactive shell, scripts can also use control statements, loops, and variables which makes them a related task, but much more complex.
In this book we will focus on the interactive part of bash.

Next Post: Navigating the File System

Weekly News Summary for Admins — 2017-07-28

This weekly news summary will be the last until the end of August. I will keep track of interesting posts, and provide a big summary at the end of the month, though I will also be offline a lot and probably miss something interesting. If you find any post or link you think is worthy, you can send to it me on Twitter or the MacAdmins forum (also scriptingosx) and I will make sure to include it.

I have scheduled a bunch of articles to be posted on the weblog over the next few weeks. You can find details in yesterday’s post: Vacation Time, Travel Time, Sneak Preview Time

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

♻️Updates and Releases

📺To Watch

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Thoughts on the iPod (2001–2017)

“Dead!?” Mine still works!
I started working for Apple Germany in January 2001.

It was a few years after the release of the iconic Bondi Blue iMac and the deal with Microsoft. The worst of the dark ages were over. But still, many of my friends were astounded, when Microsoft and Linux would obviously split the market in the future, and Apple had no future. My main reason for joining was that I was that I was excited for Mac OS X, which was (in my opinion) the only OS that successfully combined a decent UI with a Unix core. I got the job then because I knew both Mac and Unix.

Even though Mac OS X 10.0 (later known as Cheetah) had been released earlier that year, the uptake with customers was still minuscule as major third party vendors were slow to adopt the new platform. (You’re thinking Adobe, but the main laggard was actually Quark. Adobe would gain market share from adopting Mac OS X quickly, but I am leaping ahead here.) Apple’s main “bread’n butter” business was selling PowerMac G4 and the new Titanium PowerBook G4 to print and design professionals.

Because of time zones Keynotes were shown in the Apple Germany Office after hours. For the Keynote in October 2001 there were only three people who stayed to watch. The rumor mill had predicted some ‘iTunes announcement.’ Most co-workers dismissed the iPod as a toy that wouldn’t help us sell Macs.

I had been eyeing other MP3 players and really liked the idea of a ‘1000 Songs in Your Pocket.’ So I took advantage of the employee deal to buy an iPod for half price. When I finally got my iPod it quickly became a “must carry” device along with my cell phone.

It is really hard to remember that we used to have at best one or two handful of music CDs or cassettes with us when we were on the road.

Over the next few months and years, I would count the increasing number of people wearing the iconic white headphones on my commute on the Munich S-Bahn (metro). At first there’d only be a few and we would exchange knowing nods, as if we were the members of some secret club. From that anecdotal count and from the sales numbers it was obvious that Apple was on to something. Interestingly, the Mac sales numbers rose together with iPod sales. The press and Apple executives called this the ‘halo effect.’

My main job during the early first decade of the millennium was consulting IT departments, mainly on Xserves, Xserve RAIDs and Xsan (which were introduced just a few months later). Many customers were, however, much more interested in the latest iPod. Mac and Xserve was work, but iPods were fun! I remember doing a workshop at a university on Mac/Unix/Servers and Storage when the news dropped that the iTunes Store was now (finally) available in Germany. This announcement got cheers and applause from an IT crowd.

The iPod certainly turned out to be a successful ambassador for Apple. It gave Apple good press and helped raise the image of Apple out of permanent ‘beleaguered’ status. It also showed that design and user experience could be successful against mere feature lists and price. The combination of operating system, interface and hardware from the same designer mattered and made the difference. This emboldened Apple to stick to this same philosophy with the Mac.

Once cell phones started storing and playing music, the demise of the iPod was obvious, though it was a slow recline. Apple did the ‘courageous’ act of cannibalizing their own product with the iPhone.

Yesterday, Apple removed the last dedicated iPods from the Store and the webpage. The iPod lives on in the iPod touch and the Music app on iPhones and iPads. It still marks the end of an era.

The iMac demonstrated that Apple could and would keep doing what Apple was good at: building great personal computers. The iPod showed that Apple could be more than that.

The iPod transformed Apple from a one-product company (the Mac), to a consumer electronics company with multiple product lines and platforms. Apple had attempted this before with the Newton, but had not been successful. Ironically, they probably were not even planning this with the iPod. The iPod also lead to the iTunes Store, which was the platform the AppStore is based on.

Not bad for a “lame” product.

Vacation Time, Travel Time, Sneak Preview Time

It is vacation time here in the Scripting OS X headquarters and I will be enjoying time off with family for most of August.

Tomorrow’s weekly news summary will be the last until the end of August. I will keep track of interesting posts, and provide a big summary at the end of the month, though I will also be offline and lot probably miss something interesting. If you find any post or link you think is worthy, you can send to it me on Twitter or the MacAdmins forum (also scriptingosx) and I will make sure to include it.

However, I did not want to go entirely dark during the month. To keep you interested, I will post a few sections from the book I am currently working on over the next week. They are still a bit rough and unfinished.

These sections are from a chapter called ‘Terminal Primer’. They are much more basic than the series of Terminal tips and tricks I ran a few weeks ago. They are targeted to Mac Admins who are new to Terminal and bash. The book this chapter is from, will contain this primer, but also sections on how to use Terminal and bash effectively as a Mac Admin.

Yes, I know, I have promised a book on autopkg. Which I am also still working on. However, while writing the autopkg book I realized that Terminal and bash skills are fundamental for Mac Admins and I feel I need to get it out of the way first.

Now you may ask, “Aren’t there other skills you need to use autopkg effectively, such as property lists, Python, git and Github?” and I will answer, “I know, right!?”

The Terminal Primer sections will be posted (automatically scheduled) on this blog over the next few weeks. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

Terminal Primer

  • First Steps in Terminal
  • Navigating the File System
  • Special Characters
  • Commands
  • Managing Files

Have a great August!

Terminal and SSH Apps for macOS and iOS

This is an addendum to my series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

So far we have used ssh entirely from macOS’ built-in Terminal application. In most cases Terminal is entirely useful and sufficient. However, there are some really useful terminal/ssh applications from third parties. There are also applications for iOS you can use to connect to other computers with ssh.

(iOS and Mac AppStore links are affiliate links.)

Panic’s Prompt 2

Prompt 2 (vendor page) is my favorite iOS application to connect to a shell. (Together with Edovia’s Screens, it forms the backbone of my admin toolkit on iOS.)

Prompt supports external keyboards and keyboard shortcuts. It also supports key based authentication and agent forwarding. It can optionally sync server bookmarks, clips and credentials through the Panic Sync cloud server. Prompt can also detect SSH hosts in the local network that advertise via Bonjour.

Prompt 2 is a one time-purchase for iPhone and iPad, available at the AppStore.

Termius

Termius is a cross-platform SSH solution. It has apps for iOS, macOS, Android, Linux and Windows, so if you need to move across many of these platforms it may be very useful. (I have only tested iOS and macOS.) It is free for basic use, but you can unlock ‘premium’ features for a subscription fee. The premium features include a bookmark and key sync service, clippings, sftp and agent-forwarding.

It has some interesting features such as port forwarding and a the ability to send a public key to a server (like ssh-copy-id) on iOS.

iTerm2

iTerm2 is an alternative terminal application for macOS. It has many features that Terminal.app is lacking or added much later than iTerm2.

Cathode

Cathode is an alternative Terminal application for macOS and SSH client for iOS. Its main claim to fame is to emulate the appearance of old-style cathode ray displays, including 8-bit fonts, distortion and flicker. Cathode is a one-time purchase for macOS and iOS.

Mosh and Blink

mosh (mobile shell) is a protocol for remote shells that is designed for modern mobile connections. mosh can maintain connection to a server even when the client’s ip address and/or connection method changes (i.e. a device switches from Wi-Fi to cell data or vice versa). It uses ssh to establish the connection and security, but then uses its own connection. So you can use ssh keys to authenticate. However mosh requires a server component to be installed on the host.

The iOS mosh client is called Blink and commercially available on the AppStore. Though, since the project is open source, you could also compile the client yourself.

SSH Tunnels

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

We have learned so far that ssh is a really useful and flexible protocol. It can be used to connect securely to a remote shell, or to transfer files securely.

Rather than providing the shell itself, ssh provides a secure way to transmit data to and from the remote shell. In a similar way, ssh can be used to provide access to other remote services as well.

SSH Tunnels with Two Computers

Access to important services are usually blocked behind a firewall or router. Since ssh, when setup correctly, is quite secure, you can usually get access to a server with ssh even when other protocols are blocked. (Though some administrators move ssh access to a different port than the default 22.)

You can use ssh port forwarding or ‘tunneling’ to gain access to other services through ssh.

Imagine you want to use Screen Sharing to connect to a remote Mac (remote.example.com). Screen Sharing on macOS uses the VNC port 5900 to connect to a remote Mac. Since VNC itself is inherently insecure, (mac Screen Sharing adds a few things to make it more secure) this port is blocked by many firewalls.

However, I do have ssh access to remote.example.com. So, how do I tell both systems to ‘tunnel’ the screen sharing traffic through ssh?

(When you test this, remember to enable either ‘Screen Sharing’ or ‘Remote Management’ (i.e Apple Remote Desktop) access in the ‘Sharing’ pane in System Preferences on the remote Mac.)

The tunnel starts on my local machine and ends on remote.example.com at port 5900 (where the screen sharing service is listening on the remote Mac.)

The starting point also needs a port number, and I can basically choose freely. Port numbers under 1000 and over 49000 are reserved for the system and require root privileges. There are also many numbers that are commonly used by certain services (such as 5900 for VNC/Screen Sharing) and may already be in use. I will choose 5901 for the local port.

To connect the local port 5901 to port 5900 on the remote Mac use the following command:

$ ssh -N -L localhost:5901:localhost:5900 remote.example.com

(You can just try this with a second Mac or virtual machine in your network, even without a firewall.)

The syntax of this command is less than obvious. Let’s break it into pieces:

The -N option tells ssh that we do not want to invoke a remote shell or run a remote command.

The -L option creates a local port forwarding setup. This option takes a parameter with three or four parts, separated by colons :. The first pair (localhost:5901) are the tunnel start point. The second pair (localhost:5900) are the remote end point of the tunnel.

The second localhost is resolved on the remote host, so this means port 5900 on the remote host.

The last parameter states the remote host, to connect to, remote.example.com.

This commands tell ssh to connect to remote.example.com and establish a tunnel that transfers traffic from port 5901 on my computer to port 5900 on the remote computer.

Since the origin of my tunnel is usually on my local computer, the first localhost can be omitted, so you only see the origin port.

$ ssh -N -L 5901:localhost:5900 remote.example.com

When you execute the command nothing will happen. You will not even get a new prompt, because the ssh process is running until you cancel it with ctrl-C. Don’t cancel it yet, however, since it needs to run to provide the tunnel.

So, when you open the Screen Sharing application (from /System/Library/CoreServices/Applications/) and connect to localhost:5901 all traffic will be forwarded by sshto port 5900 on the remote Mac.

You can also use the open command to connect with Screen Sharing:

$ open vnc://localhost:5901

You should be able connect with Screen Sharing, even when port 5900 is blocked by a Firewall.

When you are done with the Screen Sharing session, you can end the ssh tunnel process in Terminal with ctrl-C.

SSH triangle

You can also use ssh port to use the remote host as a gateway or ‘jump host’ to a third computer. Imagine you want to use Screen Sharing to connect to secundus.example.com behind a firewall and you only have ssh connection to primus.example.com available. You can tell primus to point the endpoint of an ssh tunnel at secundus with:

$ ssh -N -L 5902:secundus.example.com:5900 primus.example.com

Note: secundus.example.com or whatever host or IP address you enter there will be resolved on the remote host. So you can use NAT IP addresses or .local host names here, even if they do not make sense in the network your work Mac is in. (They do have to make sense on the remote host, though, otherwise you will get an error.)

In the following examples the local IP address 192.168.1.200 or the Bonjour hostname Secundus.local will be resolved on the remote host, even if they don’t work on my local computer:

$ ssh -N -L 5902:192.168.1.200:5900 primus.example.com
$ ssh -N -L 5902:Secundus.local:5900 primus.example.com

Either way, you can then point Screen Sharing at localhost:5902 and it will connect through primus to Screen Sharing on secundus.

Keep in mind, that while the connection from the start point (on your Mac) to the host primus is secured by ssh the connection from primus to secundus is not.

Stumbling over HTTP hosts

In general you can use ssh port forwarding (or tunnels) for any service. Some services however, may introduce extra pitfalls.

For example, I wanted to use ssh port forwarding to gain access to my home router’s web interface. I can use ‘Back to My Mac’ to ssh into one of the iMacs at home, and thought it should be easy to connect to the router with an ssh tunnel:

$ ssh -N -L 8080:192.168.1.1:80 mac.example.com

This seemed to work, but whenever I tried to point a browser to localhost:8080 it couldn’t connect to the web page. The problem here is not the ssh tunnel but the the web server on the router. As part of the http request, the browser sends the name of the domain requested to the web server. This allows web servers to host different pages for different domains. With this request, the browser told the router it wanted the web page for localhost and the router replied with “I don’t serve pages for that host”… (Your router might behave differently.)

With curl I could convince the router to serve me the page with:

curl -H "Host: 192.168.1.1" http://localhost:8080

However, since navigating the web interface of the router with curl is out of the question I had to find a different solution.

Tunnel All the Things!

What if I could send all traffic through the iMac at home?

With the command

$ ssh -N -D 9001 remote.example.com 

I can create a tunnel from my computer (on port 9001) to the remote Mac that acts as a SOCKS proxy. Then I can set the Socks proxy to localhost:9001 in the proxy tab in the Network pane in System Preferences. You probably want to create a new network location for this setup. Then all network traffic will be securely routed through the ssh tunnel to my Mac at home where it can connect to the router.

This can also serve as a temporary VPN solution.

However it is somewhat painful to set up and maintain, so if you start using this more frequently, you probably need to look into a proper VPN service solution (some routers, ironically, provide one…).

Summary

  • you can bypass firewalls and other network blocks by tunnel traffic for any service through a ssh tunnel
  • the command describes which local port to connect to which port on the remote host
  • you can even tell the remote host to connect the end point to a third computer, behind the firewall
  • you can also create a SOCKS proxy with ssh to tunnel all traffic

Previous Post: Transferring Files with ssh

Weekly News Summary for Admins — 2017-07-21

Another busy summer week. mac OS Sierra 10.12.6 and iOS 10.3.3 dropped (along with watchOS and tvOS updates). 10.12.6 unifies the build for the new iMacs and MacBooks Pro. These are expected to be the final updates for Sierra and iOS 10.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

PSU Mac Admin Conference Presentation Slides

I re-included last week’s links as well.

♻️Updates and Releases

📚To Read

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Transmit 5 for Mac released

Panic has updated their awesome file transfer application Transmit for Mac to version 5.

(Transmit iOS is still at version 3. Available on the iOS AppStore.)

I just mentioned Transmit in my post on Transferring files with SSH earlier this week (I just updated the links in that post).

Also, this version of Transmit is no longer available in the Mac AppStore, adding to the ever longer list of applications that are leaving the AppStore. Panic’s reason is that they want to give users a free trial option.

Transferring Files with SSH

So far in this series of posts on ssh on macOS:

Please consider supporting Scripting OS X by buying one of my books!

In the previous posts we looked how to connect with ssh to a remote computer (host) and how to setup the keys necessary for a secure connection.

Despite the name ssh does not actually provide a shell or command line interface to the host itself. it ‘merely’ provides a secure connection to connect to the default shell on the host itself.

Even this basic use of ssh is already very useful and powerful. It allows to open one or more full command line sessions to remote computers as if we sat at their keyboard. You can also send individual commands and receive and process the results.

However, ssh has a few more powerful tools available.

Copy Files Remotely

You can also use the ssh connection to copy files to and from a remote host.

The command you use for this is scp (secure copy) and it use the same basic syntax as the cp command

$ scp source destination

But, since scp can copy from the local computer to a remote host or vice versa, you usually add a bit more information:

$ scp [[user@]host1:]source [[user@]host2:]destination

(The examples will use a file name hello.txt. To create one quickly, simply type echo "Hello SSH" > hello.txt in Terminal.)
To simplify this, a few examples:

$ scp hello.txt mac.example.com:"~/Documents/"
hello.txt                                     100%   10    10.8KB/s   00:00    

This will copy the local file sample.txt from the current working directory to the remote host’s ~/Documents/ directory. scp will show an ascii progress bar for every file copied. (though with these small files, you will not see much of it) You can suppress the progress display with the -q (quite) option.

For the destination, the colon : separates the hostname (DNS) from the file path.

This command will prompt for the user’s password on the remote host, unless you have added your public key to the remote host’s authorized_keys file.

Since no user name is given before the hostname (separated with an @) scp uses the username that you are logged in with on the local computer. If the remote user has a different name, use:

$ scp hello.txt user@mac.example.com:"~/Documents/"

We do not want the local shell to evaluate the ~ to the local home directory, but want the remote computer to evaluate ~ to the remote user’s home directory, so we have to quote the remote path.

Like cp, when the source is a file and the destination is a directory, then the file will be placed into the destination directory.

If the remote path does not exist, then scp will present an error:

$ scp hello.txt mac.example.com:"~/DoesNotExist/"
scp: /Users/armin/DoesNotExist/: No such file or directory

You can also rename the file while copying:

$ scp hello.txt mac.example.com:"~/Documents/ssh_test.txt"

You can copy files from the remote host to your local machine:

$ scp mac.example.com:"~/Documents/ssh_test.txt" .

In this case we passed . or the current working directory as the destination. You can also pass a local path:

$ scp mac.example.com:"~/Documents/ssh_test.txt" ~/Downloads

Use the -r option to copy all the contents of a directory:

$ scp -r ~/Projects/greatproject/ mac.example.com:"~/Documents/"

Remote-to-remote Copies

You can copy from one remote host to another.

There are two solutions for this. The first will copy the file to the local computer and then back up to the other remote host. You invoke this version of remote-to-remote with the -3 option.

$ scp -3 primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

(I am shortening the full domain names from primus.example.com and secundus.example.com to primus and secundus for simplicity.)

Under most circumstances copying a file down to your Mac and then back up to the other remote host is less than ideal. Imagine you are working from home with your laptop and want to copy a large file from one server at work to another.

The other option is to tell the source remote host to scp the to the other remote host. You could achieve this by sshing to the remote machine and running scp from there. Thankfully, scp is smart enough to attempt exactly that when you type

$ scp primus:"~/Documents/ssh_test.txt" secundus:"~/Documents/"

This command will probably fail right now. It requires a few things to be set up to work:

  • either: client key authentication to be setup from primus to secundus
  • or: client key authentication from your local computer to primus and secundus with agent forwarding enabled

The first option is fairly easy to understand. scp will connect to primus and authenticate with your local client key. Then it will tell primus to connect to secundus, authenticating using primus’ client key and the copy the file. It basically works as if you sshed in to primus and ran scp without the extra typing.

There are drawbacks to this. If ssh-agent is not running on primus and does not have the passphrase stored yet, then primus cannot unlock its private key and authenticate to secundus.

Also you basically need to prepare all remote hosts to have keys exchanged between each other, which can be painful, if not impossible to manage.

Agent Forwarding

The second option, called ‘Agent Forwarding’, circumvents these problems. This will tell the first remote host (primus) to ask your local ssh-agent for a key to authenticate to secundus. The system does not actually transfer the private key, but asks ssh-agent on your local computer to encode the authentication challenge for primus.

That way you only need to manage the keys for all remote hosts on your local computer.

You can also use agent forwarding with normal ssh connections. It is not enabled by default, but you can enable it for an ssh session with the -o ForwardAgent=yes option:

$ ssh -o ForwardAgent=yes primus

Since this is hard to type, there is a shortcut:

$ ssh -A primus

When you are logged in to the remote host with agent forwarding enabled, you can then ssh from there to another remote host (secundus) and it will try to use the keys from your local computer’s ssh-agent to authenticate.

$ ssh -A primus
Last login: ...
primus$ ssh secundus
Last login: ...
secundus$ 

This can be a useful strategy if direct access to the second remote host (other.mac.com) is blocked with firewalls.

You can also use this for the scp remote-to-remote copy. Unfortunately, scp does not have a convenient -A option, so you have to use the long parameter form:

$ scp -o ForwardAgent=yes mac.example.com:"~/Documents/ssh_test.txt" other.example.com:"~/Documents/"

When you need agent forwarding regularly for a specific host, you can enable it by default for this particular host in your ~/.ssh/config file. Add the following lines:

Host primus.example.com
    ForwardAgent yes

Note: There are some security concerns with agent forwarding. A user with root access on the intermediate system can gain access to the connection to your local ssh-agent, thus gaining the ability to encrypt with your private keys. Be aware of this in security sensitive environments.

SFTP

If you have many files in a complex file structure , scp can be a little cumbersome. There is a special interactive mode that you can invoke with the sftp command (secure file transfer program).

$ sftp mac.example.com

Note: sftp(according to its man page) is ‘similar’ to ftp but not identical. It also should not be confused with ftps which is ftp over SSL.

Obviously, if you have key authentication setup, sftp will use that.

There are many interactive commands to navigate the local and remote file system and upload (put) and download (get) files. You can look at the details in the sftp man page. However, if you need to use sftp frequently, then you should use a graphical sftp application. There are a large number of SFTP client for macOS and iOS. Here is a list of some popular clients: (AppStore links are affiliate links.)

All of these tools connect to many other server protocols other than sftp. However, the advantage of sftp is not just the built-in security, but that you don’t need other software than sshd running on the server.

Summary

Previous Post: Client Verification

  • you can use scp [[user@]host:]source [[user@]host:]destination to copy files from or to a remote host over ssh
  • you can use agent forwarding to simplify key management in triangle setups
  • sftp help managing/transferring multiple files over ssh, there are many UI applications

Next Post: SSH Tunnels

Weekly News Summary for Admins — 2017-07-14

Much busier this week. Some interesting releases and news. I have also gathered all the links to notes and slides from PSU presentations. If I missed any, let me know and I will include the link next week.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

PSU Mac Admin Conference Presentation Slides

♻️Updates and Releases

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!