Book Updates for High Sierra

Happy Update Day!

macOS High Sierra (10.13) will be released some time tonight. There have already been many articles on many of the new features (or issues) in High Sierra, especially in my Weekly Newsletter. But how does High Sierra affect my books and information therein?

The good news is: surprisingly little. There were many rumors and concerns in the build-up to WWDC this year, but the worst did not happen. I posted about my reaction to the news in WWDC here.

Nevertheless, the tutorials in the books needed to be tested on High Sierra and there were quite a few changes that had accumulated over time so I threw those in as well. The advantage of digital books is if you have already purchased the books (Thank you!) you will get these updates for free in the iBooks Store (you might have to check in the ‘Updates’ tab).

If you don’t have the books yet, you can go and buy them now and get future updates to these books as well!

PR3 is still in review limbo, but should be through soon. I’ll post an update, when it clears. ‘Packaging 1.5’ is available on the iBooks store already!

If you have already purchased books, all I ask in return for the free update of new information, is to go to the iBooks Store and leave a review. iBooks Store segregates reviews by territory, so every single one of them will be very important for other users to find and evaluate the books.

Thank you!

Some notes on each of the books in particular:

Packaging for Apple Administrators

The basic tools and methods for packaging in High Sierra have not changed. But since I had to go through the book to test all the examples again, I made quite a few minor corrections and clarifications.

Note: The current version of Whitebox Packages, does not run on High Sierra. I believe there will be an update soon, so I did not change the section in the book to reflect that right now.

I also added two entirely new sections: (not dependant on High Sierra)

  • a simple example on how to build Un-Installer scripts, something macOS does not automatically provide.
  • based on this blog post: how to extract a component from a distribution package.

Other than that, Packaging remains very relevant to a Mac Administrator’s skill set with High Sierra, so go and get the book! (and please leave a review)

Property Lists, Preferences and Profile for Apple Adminstrators (PR3)

Note: as mentioned before, PR3 is still in Apple review limbo. I will post as soon as it clears. If you haven’t bought it yet, you can buy the current version now and will get the update pushed in iBooks, as soon as it clears review!

First, with High Sierra comes Swift 4, which brings a new Property List serialization API. I added new sample code to the Swift section for Swift 4.

Second, the profiles tool comes in a new version in High Sierra, with new syntax and some new functionality. You can see the new command syntax in the man page of the profiles command in High Sierra. (You can also still get the old syntax on High Sierra by calling the man page for profiles.old.)

However, while the old syntax is considered deprecated the new version on High Sierra still supports it. So there is no reason (yet) to run out and change your scripts. Nevertheless, both versions are documented in the relevant section in the book.

There is some new functionality in the new syntax (startup type profiles) and I assume that new features will only added to the new syntax going forward. As long as you still need to support Sierra Macs and older, you will have to use the old syntax or maintain both versions.

And, like the Packaging book, while I was working through the examples in the books, there are many corrections, additions and small clarifications added.

With many interesting new features in MDM, profiles will increase in relevance for adminstrators. Go get the book! (and please leave a review)

Weekly News Summary for Admins — 2017-09-22

Happy Equinox, everyone! ☀️

This was the big week for the iOS family of Apple OSes (iOS, watchOS and tvOS) and hardware. While macOS HighSierra will not ship until next Tuesday (Sep 25) we already got a lot of new relevant information from the iOS release.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

🍏Apple Support

♻️Updates and Releases

macOS

iOS

📺To Watch

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

On Distribution Packages

Distribution packages are a special format of installer packages. Distribution packages can contain one or more normal or component packages. They can also contain extra resources to customize and control the user interface in the Installer application.

In most cases administrators prefer component packages since they are easier to create and maintain. However, there are a few cases where distribution packages are necessary:

  • add a package to a custom installation in NetInstall, AutoDMG or a system installer created with createOSXinstallPkg
  • combine multiple component pkgs into a single installer
  • restrict hardware and system requirements
  • modify the interface presented in Installer.app
  • push the package with MDM’s InstallApplication command

Building Distribtution Packages

You can easily convert an existing component package, built with pkgbuild to a distribution package with the productbuild command:

$ productbuild --package component.pkg dist.pkg

You can also combine multiple components into a single distribution package:

$ productbuild --package A.pkg --package B.pkg combined.pkg

You can add the --sign option to the productbuild command when the distribution package needs to be signed:

$ productbuild --sign "Installer: Armin" --package component.pkg dist.pkg

You can find valid identities with

$ security find-identity -p basic -v

The string you pass with the --sign parameter can be a partial match to the full identity name.

Note: munkipkg has a flag to build a distribitution package instead of a component package.

Extracting Component Installers from Distribution Packages

Sometimes you may want to extract a component installer pkg from a distribution package.

First you need to expand the distribution pkg with pkgutil:

$ pkgutil --expand dist.pkg dist_expanded

When you use the --expand option on a distribution package, components will also be expanded into subfolders that end in .pkg. Because of this Finder will erroneously display them as installer bundle files. This is misleading, since the components are not functional in this form.

When you want to use the component package without any modifications, you can quickly recompress or ‘flatten’ the expanded component:

$ pkgutil --flatten dist_expanded/component.pkg component.pkg

The process of expanding and flattening a component will of course remove any signature the original pkg might have had. You can re-sign the flattened package with productsign:

$ productsign --sign "Installer: Armin" component.pkg component_signed.pkg

Note: Obviously, when you are tearing a distribution package apart you need to know what you are doing. Components in a distribution package may depend on other components or on scripts and tools in other components. As always: test, test, test.

Packaging Book

You can learn more on building installer packages in my book: “Packaging for Apple Administrators”

Weekly News Summary for Admins — 2017-09-15

High Sierra Golden Master candidate release today! Time to get that last minute testing in! Release date will be September 25.

Also Apple had their big event, showing the new Steve Jobs Theater at the Apple Park and the new iPhones and Apple Watches. The fact that macOS did not get a mention is not that unusual. The September event has always been for iPod/Music/iPhone. The iPads did not get a mention, either.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

🍏Apple High Sierra

♻️Updates and Releases

📺To Watch

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

So Long, Cassini, and Thanks for all the Pics!

(Please indulge me while I stray from my usual topics.)

Today the Cassini orbiter will burn up in Saturn’s atmosphere, nearly twenty years after it was launched from Earth and thirteen years after it entered orbit around Saturn.

It makes me terribly sad to see the (physical) end of its mission. But I believe it is better to celebrate what an amazing and mind-boggling mission it has been.

Not only did the Huygens probe land on another planet’s moon, but every time the controllers pointed Cassini’s camera and sensors at something new, it discovered a surprise.

Cassini saw the hexagonal storm on Saturn’s pole. A moon that looks like a dryed out sponge, moons with ridges, hydrocarbon lakes on Titan, water volcanoes on Enceladus, gravity waves in the rings

The list goes on and on.

And the pictures sent back to Earth kept being utterly beautiful, showing a planetscape and moonscapes vastly different from our terrestrial experience. We could never have imagined the views and features revealed to us by the orbiter’s camera.

But now we can.

This is of course, the point of science and exploration. Where ever humanity starts to explore, we always discover the unexpected.

Cassini, Rosetta, Juno, NewHorizon and all the other probes in space, prove over and over again, that the universe is stranger than we can imagine. But once someone has measured or seen our universe has expanded. We have learned.

After today there will be one spacecraft left in the outer solar system: Juno is orbiting Jupiter and scheduled to be scuttled into the Jovian atmosphere in 2018. NewHorizons has zipped through the Plutonian system and is preparing to meet a small icy body in the Kuiper belt on its way out of the system.

As of now, none of the space agencies are preparing another mission to the outer solar system. There are plans. However, the preparation and travel necessary for these missions requires several years.

Once Juno is scuttled, we will not see new pictures from the outer solar system for at least a decade or more.

This makes me much more sad than seeing the end of this glorious, astounding and wonderful mission.

Thank you, Cassini, and everyone who made this happen!

Weekly News Summary for Admins — 2017-09-08

Big Apple Event next week! Tuesday will certainly be interesting.

Note: Byword – the app I use to write my blog posts, including this weekly – is on sale this week, both for Mac and iOS!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

♻️Updates and Releases

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Terminal Primer – Part 6 – Symbolic Links

I will be posting drafts of parts of my next book on macOS Terminal and bash. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

Symbolic Links

When you get a detailed list of the file system root / you will see a few interesting entries: (output abbreviated)

$ ls -l /
lrwxr-xr-x@    1 root  wheel    etc -> private/etc
lrwxr-xr-x@    1 root  wheel    tmp -> private/tmp
lrwxr-xr-x@    1 root  wheel    var -> private/var

These are symbolic link files. Symbolic links are identified by the first character l on a long ls. Symbolic links are redirections to other files or directories. /etc, /tmp and /var are standard directories for certain kinds of files in Unix systems, but in this case these directories are actually located in /private.

Note: The reason for this is historical and goes back to NeXTStep. This setup could allow the /private folder to be mounted from another disk or file share separate from the rest of the system. This is not used in macOS anymore, but the directory structure remains.

Aside from the long ls you can use the readlink command to determine where a symbolic links points to:

$ readlink /etc
private/etc

or the stat -l command:

$ stat -l /etc
lrwxr-xr-x 1 root wheel 11 Nov 17 07:50:53 2016 /etc -> private/etc

A symbolic link contains a path pointing to the original file or directory. Most operations such as reading or changing are directed to the original, rather than the symbolic link. ls /tmp and ls /private/tmp will show you both the contents of the original /private/tmp.

An example for a symbolic link to a file is the file /usr/share/dict/words (a list of english words, which can be surprisingly useful to have around) which points to a file web2 in the same directory. Symbolic links can be used as a means to ‘switch’ between files without having to change around filenames and configuration files.

When you read the file /usr/share/dict/words the read command will be redirected to /usr/share/dict/web2:

$ cat /usr/share/dict/words | wc 
  235886  235886 2493109
$ cat /usr/share/dict/web2 | wc 
  235886  235886 2493109

Note: the wc command counts words, lines and bytes in a file.

Symbolic links can be relative or absolute. However, most of the time they are relative, since you do not want them pointing to different files, depending on which volumes the system is booted from. Relative paths are resolved relative to the link itself, not the current working directory. The above link for /etc points to the relative path private/etc so to the sub-directory etc in the directory private in the same location as the symbolic link itself.

To create a symbolic link use the ln -s command.

$ mkdir directory
$ touch directory/file
$ ln -s directory/file
$ readlink file
directory/file

This series of commands will create a directory, and en empty file in directory and then a link to that file in the current working directory. When you want the symbolic link to have a different name, you can give that as a second argument:

$ ln -s directory/file second_link
$ readlink second_link 
directory/file

The first argument is not really the path to a file or directory, but the path the symbolic link points to. When this path is relative it will be resolved relative to the location of the symbolic link. So if you wrote:

$ ln -s directory/file directory/link_to_file

A symbolic link named link_to_file will be created in directory but it will point to directory/directory/file instead of the file next to it. When you try to read from the symbolic link pointing to nowhere, you will get an error:

$ cat directory/link_to_file 
cat: directory/link_to_file: No such file or directory

The reference you pass when creating the symbolic link has to be relative to the where the symbolic link is created. The correct command would have been:

$ ln -s file directory/link_to_file

When the second argument is a directory itself, a link named the same as the file it refers to will be created:

$ mkdir another_dir
$ ln -s ../directory/file another_dir

will create a link named file pointing to ../directory/file. Since you have to give the path to the target relative to the where the link is created you have to add the ../ to ‘go up a level’ out of the another_dir directory and then back into directory.

Note: you can create a symbolic link that points to non-exiting path. Also when the original file or directory gets deleted, a ‘dangling’ symbolic link will remain.

When you use rm or mv on a symbolic link, only the link will be affected, not the original item.

When you run cp on a symbolic link to a file, the contents of the original will be copied:

$ cp another_dir/file filecopy
 $ stat -l filecopy
-rw-r--r-- 1 armin staff 0 Sep  4 14:51:44 2017 filecopy

However, when you recursively copy a directory tree containing symbolic links, they will be copied as symbolic links:

$ cp -R another_dir/ copy_dir
$ ls -l copy_dir/
total 8
lrw-r--r--  1 armin  staff  17 Sep  4 14:52 file -> ../directory/file

Usually the destination of a copy will be in a vastly different location and this will break the links:

$ cp -R another_dir/ /Users/Shared/copy_dir
$ ls -l /Users/Shared/copy_dir
total 8
lrwxr-xr-x  1 armin  wheel  17 Sep  4 15:19 file -> ../directory/file
$ cat /Users/Shared/copy_dir/file 
cat: /Users/Shared/copy_dir/file: No such file or directory

You can use cp‘s -L option to make the copy process resolve symbolic links and copy their contents instead:

$ cp -RL another_dir/ /Users/Shared/another_dir
$ ls -l /Users/Shared/another_dir
total 8
-rwxr-xr-x  1 armin  wheel  17 Sep  4 15:19 file

Other commands will have similar options to control the behavior when encountering symbolic links. However, their names are not standardized in anyway and you will have to consult the man page to find out the details.

Symbolic Links vs Finder Aliases

In macOS Finder, you can create aliases with the menu item ‘Make Alias’ from the ‘File’ or context menu. Finder Aliases have much the same role as symbolic links, but a few significant differences in behavior:

  • the shell and most command line tools cannot resolve Finder Aliases and treat them as files
  • Finder Aliases will ‘follow’ the original when it is moved or renamed.
  • However, when the original is deleted and replaced by an item of the same name, a Finder Alias will resolve to the new item.
  • When a Finder alias points to an item on a file share, double clicking the alias in Finder will attempt to connect to the file share, if it not already connected.

Finder will display Aliases and symbolic links with a small arrow in the corner of the icon. Both symbolic links and Finder Aliases have a ‘Show Original’ menu item in the ‘File’ menu or context menu.

There are no commands to create or resolve Finder aliases in Terminal, but you can use AppleScript with the osascript command:

$ osascript -e 'tell app "Finder" to make new alias to posix file "/Users/armin/Documents" at posix file "/Users/armin/Desktop"'

This will create a new Finder alias to ‘Documents’ in the ‘Desktop’ folder. The user who is running this command has to be logged in to the Mac, so that osascript can connect to the Finder to run this command.

To find out the original of a Finder alias, you can use:

$ osascript -e 'tell application "Finder" to get POSIX path of ( (original item of (POSIX file "/Users/armin/Desktop/Documents" as alias) ) as alias)'

Note: there is a lot of type casting (as alias) in this command. To further confuse matters the data type alias in AppleScript is not the same as a Finder alias. It is a special data type in AppleScript that references a file or folder.

Weekly News Summary for Admins — 2017-09-01

Another week, another beta. We also have a date (September 12) for the iPhone Keynote. Everything looks to be ramping up for a September or early October release for macOS High Sierra and iOS 11. Fun times ahead!

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

#! On Scripting OS X

📰News and Opinion

🔨Support and HowTos

♻️Updates and Releases

🎧To Listen

📚Support

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Parse Binary Property Lists in Finder Metadata

For more info on plutil and everything property list related read my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’

macOS and Finder use extended attributes to store plenty of extra information about files. For example, when you download a file in Safari, it stores when a file was downloaded and which website and download URL was used.

As an example, I downloaded the latest Firefox disk image. When you look at the downloaded file in the Terminal, you see an @ after the file mode which tells us this file has additional extended attributes:

$ ls -l ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg

We can further inspect the extended attributes with the -@ option:

$ ls -l@ ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg
    com.apple.metadata:kMDItemDownloadedDate          53 
    com.apple.metadata:kMDItemWhereFroms         203 
    com.apple.quarantine          57 

This shows the three extended attributes attached to this file, their names and their sizes in bytes.

When you double-click the dmg file to mount it, the system will store the checksum and file system check result in more extended attributes:

$ ls -l@ ~/Downloads/Firefox\ 55.0.3.dmg 
-rw-r--r--@ 1 armin  staff  51137761 Aug 30 15:11 /Users/armin/Downloads/Firefox 55.0.3.dmg
    com.apple.diskimages.fsck         20 
    com.apple.diskimages.recentcksum          81 
    com.apple.metadata:kMDItemDownloadedDate          53 
    com.apple.metadata:kMDItemWhereFroms         203 
    com.apple.quarantine          57 

To inspect the contents of the extended attributes in further detail, we have to use the xattr command: xattr -l filename will show all extended attributes, or you can use xattr -pl attributename filename to get just a particular one:

$ xattr -pl com.apple.quarantine ~/Downloads/Firefox\ 55.0.3.dmg 
com.apple.quarantine: 0083;59a6b982;Safari;E56EFC36-29AB-4F77-89E6-F4264336060F

The contents of the quarantine attribute is a string with some numbers (presumably a hash) the application that downloaded it (Safari) and a UUID.

The downloaded date however, looks a lot different:

$ xattr -pl com.apple.metadata:kMDItemDownloadedDate ~/Downloads/Firefox\ 55.0.3.dmg 
com.apple.metadata:kMDItemDownloadedDate:
00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 BF 56 F1 02  |bplist00..3A.V..|
00000010  53 AF D9 08 0A 00 00 00 00 00 00 01 01 00 00 00  |S...............|
00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 13                                   |.....|
00000035

This attribute is some binary data (xattr automatically uses binary representation when it detects a nil byte. The first few bytes in the binary data are bplist which tells us we are working with binary plist format. Unfortunately the rest of the binary plist data is quite unreadable for humans.

We can use xxd to convert the binary representation into actual data and plutil to print the plist:

$ xattr -px com.apple.metadata:kMDItemDownloadedDate ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -p -
[
  0 => 2017-08-30 13:11:30 +0000
]

Note that the options for xattr changed from -pl to -px which forces the output to be binary data only.

And the same command for the ‘WhereFroms’:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -p -
[
  0 => "https://download-installer.cdn.mozilla.net/pub/firefox/releases/55.0.3/mac/en-US/Firefox%2055.0.3.dmg"
  1 => "https://www.mozilla.org/en-US/firefox/new/?scene=2"
]

This uses plutil’s -p option to just print the data in a human readable form. You can also have plutil convert the plist data into XML:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Downloads/Firefox\ 55.0.3.dmg | xxd -r -p | plutil -convert xml1 -o - -
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
    <string>https://download-installer.cdn.mozilla.net/pub/firefox/releases/55.0.3/mac/en-US/Firefox%2055.0.3.dmg</string>
    <string>https://www.mozilla.org/en-US/firefox/new/?scene=2</string>
</array>
</plist>

or JSON:

$ xattr -px com.apple.metadata:kMDItemWhereFroms ~/Download.0.3.dmg | xxd -r -p | plutil -convert json -r -o - -
[
  "https:\/\/download-installer.cdn.mozilla.net\/pub\/firefox\/releases\/55.0.3\/mac\/en-US\/Firefox%2055.0.3.dmg",
  "https:\/\/www.mozilla.org\/en-US\/firefox\/new\/?scene=2"
]

For more info on plutil and everything property list related read my book: ‘Property Lists, Preferences and Profiles for Apple Administrators’

Terminal Primer – Part 5 – Managing Files

It is vacation time here in the Scripting OS X headquarters and I will be traveling with family for most of August. To compensate for that I will be posting a draft of parts of my next book on macOS Terminal and bash. Let me know how you like them or if you think something important is wrong or missing. You can give feedback in the comments, over Twitter or in the MacAdmins forum (also scriptingosx). Thank you for your interest and feedback!

Managing Files

We already know how to navigate and read the file system with cd and ls.
Now we want to actually do something to the files.

Create an Empty File

Sometimes it can be useful to quickly create an empty file. You can use the touch command to do this.

$ touch emptyfile

If you use the touch command on a file that already exists it will update the file’s modification date and change nothing else.
macOS (and other Unix-like operation systems) sometimes uses the existence of a file in a certain directory as a flag for configuration.
For example, when a file named .hushlogin exists at the root of a user’s home directory, the ‘Last Login: …’ message when you open a new shell (terminal window) is suppressed.

$ touch ~/.hushlogin

will create this file and subsequent new Terminal windows will not show this message. To return to showing the message, you will have to delete the file.

Deleting Files

To delete a file use the rm (remove) command:

$ rm document.txt

You have to use special care with the rm command.

In the Finder, deleted files are moved to the Trash, which is actually the invisible directory ~/.Trash. There the file will remain until the user chooses ‘Empty Trash’. Only then is the file removed from disk.
The command line has no such safety net.

When you delete a file (or directory) with the rm command it is gone. You have to be especially careful with files with special characters. If a filename has a space in it, and you run the rm command without escaping or quotes, then you will get an error or even worse, might delete the wrong file.

For example:

$ rm My Important Document.txt

Will delete the three files My, Important, and Document.txt, if they exist. If they do not exist it will show errors.

Use escape sequences or quotation marks to protect from spaces and other special characters in and directory names:

$ rm 'My Important Document.txt'

Tab-completion will also protect from improperly typed or escaped file names. If the tab-completion will not work, even though you believe you have the right file or path then something went awry and you have to step back and verify your working directory and paths.

To delete the .hushlogin file we created above, you use rm

$ rm ~/.hushlogin

Once the file is removed, new terminal windows will show the ’Last Login: …" message again.

You can add the -i option to the rm command which will ask for confirmation before actually deleting a file.

$ rm -i ~/.hushlogin 
remove /Users/armin/.hushlogin? y

Creating Directories

To create a new empty directory (or folder) you use the mkdir command.

$ mkdir scratchspace

you can give the mkdir command multiple arguments when you want or need to create multiple directories at once

$ mkdir folder1 folder2 folder3

When you create a nested directories, all the directories in between already have to exist:

$ mkdir LevelA
$ mkdir LevelA/LevelB/LevelC
mkdir: LevelA/LevelB: No such file or directory 

When you need to create nested directory hierarchies like this, you can use mkdir’s -p option:

$ mkdir -p LevelA/LevelB/LevelC

This will create all three folders at once, if they do not already exist.

Moving and Renaming

You can move a file or directory using the mv command. The mv command needs two arguments: the source and the destination.

$ touch testfile
$ mkdir testdir
$ mv testfile testdir
$ ls testdir
testfile

This mv command reads as ‘move the file testfile to the directory testdir’. To move it back to the current working directory you can use the ‘.’ short cut.

$ mv testdir/testfile .

The mv command can also rename a file:

$ mv testfile samplefile

Moving and renaming is considered the same in the shell.

Warning: When a file already exists in the destination, mv will mercilessly overwrite the destination file. There is no way to retrieve an overwritten file.

You have to take care to type the proper paths in the shell. It is a very unforgiving environment.
To make mv a bit safer add the -i option which will prompt to confirm when it will overwrite a file:

Warning: when you use mv to move between volumes, the source file will be removed after it is moved to the destination. This is different from the behavior in Finder, where the default drag action between volumes is copy.

Filename Extensions

On macOS and other operating systems it is common to denote the file type with an extension. The extension is a standard alphanumeric code separated from the rest of the file’s name by a dot or period. E.g. .txt, .pdf or .mobileconfig.

In bash, the filename extension is part of the filename. There is no special treatment for the extension.
On macOS, however, Finder usually hides the file extension from the user. You can control the display of the file extension in the ‘Advanced’ tab of Finder Preferences. You can also control this setting for each individual file in its Info panel.

Finder will also warn when you attempt to change the file extension, since it might change which application is used to open a file. (You can also disable this in the Finder Preferences.) bash has no such warning mechanism.

$ mv hello.txt hello.md

Note: A feature specific to macOS is that some directories will have filename extensions and Finder will display them as if they were files, not folders. These folders are called ‘packages’ or ‘bundles.’ The most common example are applications with the .app extension.
Packages and bundles are used to hide complex file and data structures from users. Another example is the ‘Photos Library’ (or ‘iPhotos Library’ on older systems) which hides a big and complex folder structure.
You can choose ‘Show Package Contents’ from the context menu in Finder to drill down further into the internal structure of a package or bundle.
bash and other shells are not really aware of packages or bundles and treat them like normal directories.
You can read more detail on Bundles and Packages on the Apple Developer Page.
Note to the Note: the name ‘packages’ is also used for package installer files (with the .pkg extension). These are different uses of the same word.

Copying

The command to copy is cp. It follows similar syntax as the mv command:

$ cp source destination

So you can copy the samplefile we created earlier:

$ cp samplefile newsamplefile

You can also copy to a directory:

$ cp samplefile testdir

Warning: the cp command will mercilessly overwrite existing files with the same name!

When you run cp again it will overwrite the existing copy:

$ cp samplefile testdir

As with the rm command overwritten files are lost. There is no way to retrieve overwritten files.

The -i option shows a prompt to confirm whenever a file will be overwritten:

$ cp -i samplefile testdir
overwrite testdir/samplefile? (y/n [n]) n
not overwritten

When you try to copy a directory, you will get an error message:

$ cp testdir newdir
cp: testdir is a directory (not copied).

Since the command to copy a file or directory would look exactly the same, cp expects an extra option to be certain you know what you are doing. The -R option (for recursive) will tell cp to recursively copy all files and sub-directories (and their contents) of a folder.

$ cp -R testdir newdir

This will create a copy of testdir and all its contents with the name newdir.

Warning: the option for recursive copying is -R (uppercase R). There is a legacy option -r (lowercase r) which seems to do the same thing. However, there is a difference in behavior mentioned in the cp man page:

Historic versions of the `cp` utility had a `-r` option.  This implementation supports that option; however, its use is strongly discouraged, as it does not correctly copy special files, symbolic links, or fifo's.
If the destination directory already exists the way you write the path of the source directory will influence the behavior of cp.

When the path to the source does not end with a /, cp will create a copy of the directory in the destination directory:

$ mkdir dirA
$ cp -R testdir dirA
$ ls dirA
testdir

When the path of the source directory ends with a /, cp will copy all the contents of the source directory to the destination folder:

$ ls testdir
samplefile
$ mkdir dirB
$ cp -R testdir dirB
$ ls dirB
samplefile

Warning: when you use tab-completion to complete paths to directories the / is always appended! You will need to consider whether you want to keep the trailing / or not.

You can add more source arguments to a cp command, the last argument will be the destination:

$ cp -R samplefile otherfile hello.txt bigfolder

Wildcards (Globbing)

Note: In early versions of Unix wildcard substitution was the responsibility of a program called glob, short for ‘global command.’ Because of this the action of replacing wildcards with actual paths and filenames was and still is called globbing.

When you have to address or manage many files at once, it can be slow, tedious and ineffective to address each file individually. bash provides wildcard characters to make that easier.

There are two commonly used wildcard characters: * and ?

The asterisk or star * will match zero or more characters. It can be placed anywhere in a path.

The question mark ? will match any character, but there has to be a character.

By default filenames that start with a period ‘.’ are not matched, unless you specifically start the string with a dot .*

It is important to keep in mind that bash will build a list of filenames that match the wildcards and substitute this list in place of the name with the wildcard(s) before executing the command.

When you enter

$ cd ~
$ ls D*

The D* will be replaced with the list of filenames that match (Desktop Documents Downloads) and then executed:

$ ls Desktop Documents Downloads

This can lead to some unforeseen consequences. For example say you are in a folder with some files and directories:

$ ls -F
dirA/  dirB/  dirC/  file1  file2  file3

And you run

$ cp file? dir?

The wildcards will be expanded to

$ cp file1 file2 file3 dirA dirB dirC

Which means that the three files as well as dirA and dirB will be copied into dirC, since treats the last argument as the destination and all previous arguments will be copied.

You can use wildcards in paths, so /Users/*/ will expand into all directories in the /Users folders.

However, /Users/*/Desktop will expand into a list of all users’ Desktop folders. Note that the first list contains /Users/Shared while the second does not contain /Users/Shared/Desktop, because that directory does not exist!

Warning: Wildcards can be extremely useful, but also very dangerous. They have to be handled with utmost caution, especially with potentially destructive commands such as mv, cp, and rm.

You can always test the result of wildcard expansion with the echo command:

$ echo /Users/*/
/Users/Guest/ /Users/Shared/ /Users/armin/
$ echo /Users/*/Desktop
/Users/Guest/Desktop /Users/armin/Desktop

You can also hit the escape key twice and bash will show the expansion, if there are any:

$ echo /Users/*<esc><esc>
Guest/  Shared/ armin/

Finally, bash has a third globbing or wildcard character, but it is a bit more complex. You can provide a list of possible characters between square brackets.

[bcr]at

will match

bat, cat or rat, but not Bat, Cat or Rat

Since shell commands are case-sensitive, you may have to provide both cases, if you want to match:

[bBcCrR]at

No matter how many characters are in the square brackets, they will match to exactly one character:

[bB][aei]t matches bat, Bat, bet, Bet, bit, or Bit

Deleting Directories

We have been creating and copying a lot of files. It is time to clean up. We already know the rm command to remove files. However, when you try to use rm to delete a directory you get:

$ rm newdir
rm: newdir: is a directory

There is a command rmdir which is the destructive equivalent of mkdir. However, rmdir can only remove empty directories:

$ rmdir newdir
rmdir: newdir: Directory not empty

You can use the * wildcard to delete all files in newdir:

$ rm newdir/*

Note: the * wildcard will not expand to filenames starting with a period. You may have to explicitly delete dot files as well:

$ touch newdir/.dotfile
$ rm newdir/*
$ rmdir newdir
rmdir: newdir: Directory not empty
$ rm newdir/.*
rm: "." and ".." may not be removed
$ rmdir newdir

This will work as long there are only files or empty directories in newdir. When the directory you want to delete contains an entire hierarchy of files and directories, then this approach will be cumbersome.

For this the rm command has the -R option which will recursively delete all contents and subdirectories.

$ rm -R testdir

Since there is no way to recover a file deleted by rm you should always use this command with care, especially when using the -R option.

You can add the -i option when using -R as well, but then you will be prompted for every single file and subdirectory, which can be very tedious and counter-productive.

Note: unlike the cp command, the -r and -R option for the rm command are synonyms. However, for consistency’s sake and to build muscle memory. I would recommend making a habit of using the -R syntax for both commands.