On hidden Files, especially Library

I published a book: “Packaging for Apple Administrators

While writing on the next book “Automated Packaging for Apple Administrators”, I will keep publishing small side notes and excerpts. There is a nice gem for macOS Sierra in the last section, so keep reading.;)

Mac OS X has always hidden certain folders and files from the user. The more ‘UNIXey’ folders like /usr, /bin, and /etc were considered too confusing or even dangerous for most users and hidden away. Most users noticed this in OS X Lion when Apple started hiding the user’s Library. Messing with files in the Library can cause damage or data loss if a user does not know exactly what they are doing. Here is the summary on hidden and invisible files.

Dot Files

In UNIX, files or directories with a name beginning with ‘.‘ (period or dot) are considered hidden and will not be shown in a normal file list with ls. You can however easily list them with the option ls -a. Usually dot files are configuration files or folders.

When does Finder consider a File hidden?

Like the ls command Finder will not show files beginning with a ‘.‘ (period or dot). However, there is also an extra hidden flag that Finder will check to see wether it should hide a file. You can see this hidden flag in Terminal with the -O (capital o) option for ls

$ ls -lO 
drwx------+ user  staff  -        Downloads
drwx------@ user  staff  hidden   Library
drwx------+ user  staff  -        Movies

(I removed lines and columns to make the output more legible.)

You can also use the find command to show all files with the hidden flag:

$ find ~ -flags +hidden -print

Use the chflags command to set or unset the hiddenflag:

$ chflags nohidden ~/Library
$ chflags hidden ~/Library

Finder will show or hide the file or folder immediately.

Navigating to your hidden Library

When you click on Finder’s ‘Go’ Menu with the option key, Library will appear as an option.

You can also use Finder’s ‘Go to Folder…’ menu and enter ~/Library as the target. This is especially useful since you usually want to go to a subfolder of Library anyway. This panel supports tab-autocompletion like the shell. OS X 10.11 and earlier would autocomplete to the alphabetically first match so ~/Library/Pref would complete to ~/Library/PreferencePanes rather than ~/Library/Preferences. macOS Sierra will show a popup list if the completion is ambiguous. The keyboard shortcut for ‘Go to Folder…’ command-shift-G will also work in open and save panels.

If you are already in a Terminal window you can use the open command:

$ open ~/Library/

Show all hidden Files and Folders

macOS Sierra has added a great Finder keyboard shortcut to quickly show hidden files and folders. Command-Shift-. (dot or period) will quickly show all hidden files and a second time will re-hide them.

This keyboard shortcut has worked in open and save dialogs for a while already.

In older versions of OS X you have to open Terminal and run:

$ defaults write com.apple.Finder AppleShowAllFiles true
$ killall Finder

Change the true to false to switch it back.

Packaging Book on Sale and other Deals for Apple Admins

Remember that my book “Packaging for Apple Administrators” is on sale until Cybermonday! 20% off!

I have gathered a few other interesting Black Friday Sales for Admins:

  • HumbleBundle Books Unix is an amazing bundle of 15 O’Reilly books for Unix. Many of these books are also extremely useful for macOS administrators
  • VMware sale: VMware products, including VMware Fusion Pro are 40% off
  • Parallels Black Friday Bundle: Parallels is offering an interesting bundle of applications along with a license of their virtualisation software
  • Edovia Screens VNC for iOS and for Mac are 50% off
  • Deliveries Package Tracker for iOS and for Mac has reduced price as well

Thanksgiving-Black-Friday-Cybermonday Sale!

“Packaging for Apple Administrators” is on Sale until Cyber Monday!

Get 20% off!

Here in Europe we call the upcoming week-end the… uh… last week-end in November.

However, Americans have this wonderful week-end of commercial frenzy ahead. So I decided to give everyone who has been considering buying the book, but is still reluctant, a friendly nudge. And even though Thanksgiving-Black-Friday-Cybermonday is a US thing, my sale is in all the countries where my book is available!

Go and get the book on the iBook Store!

 

Editing Property Lists with plutil

I stumbled over these option this morning. I do not know when they were introduced, but I can see the options in 10.11 and 10.12. You can see them yourself with plutil -help. (The options are not listed in the man page.)

Note: Managing and editing Property List files and preferences is covered in much more detail and depth in my book “Property Lists, Preferences and Profiles for Apple Administrators

Quick recap: plutil manipulates property list files. Its main use up to now was to convert between property list formats (mainly from binary plists to something readable)

$ plutil -convert xml1 /path/to/propertylist.plist

and to check wether the syntax is valid

$plutil -lint /path/to/propertylist.plist

On Sierra, when you run plutil -help you see some new options. These allow you to directly manipulate keys and values in a property list. This may be useful to replace PListBuddy and defaults to manipulate property lists.

When testing this I noticed one downside of plutil immediately: it cannot be used to create a new property list file. Copy this to create an empty plist file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict/>
</plist>

Inserting a new key/value pair

$ plutil -insert somekey -string somevalue test.plist
$ plutil -insert flag -bool YES test.plist
$ plutil -insert bestNumber -integer 1 test.plist
$ plutil -insert pi -float 3.141592 test.plist

This is pretty straightforward.

Inspecting a property list

You can use the -p option to check our progress:

$ plutil -p test.plist
{
  "newkey" => "newvalue"
  "pi" => 3.141592
  "bestNumber" => 1
  "flag" => 1
}

This uses a non-standard output format, and the help text warns to use this parse plists. But it will do to look at the content.

Note: you can use plutil -p to read the content of binary plists without converting!

Modifying values

You can modify values with the -replace option.

$ plutil -replace flag -bool NO test.plist

Note that you can create new entries with -replace:

$ plutil -replace otherkey -string othervalue test.plist

But you cannot overwrite an existing value with -insert.

Deleting values

Very straightforward:

$ plutil -remove otherkey test.plist

Arrays

You can insert an empty:

$ plutil -insert list -xml '<array/>' test.plist

or

$ plutil -insert list -json '[]' test.plist

and add items to the array

$ plutil -insert list.0 -string 'list item 1' test.plist
$ plutil -insert list.1 -string 'list item 2' test.plist
$ plutil -insert list.2 -string 'list item 3' test.plist

or do it all at once

$ plutil -replace list -json '[ "yes", "no", "maybe" ]' test.plist

Dictionaries

$ plutil -insert dictionary -xml '<dict/>' test.plist
$ plutil -replace dictionary -json '{}' test.plist
$ plutil -insert dictionary.key1 -string value1 test.plist
$ plutil -replace dictionary -json '{ "otherkey" : "othervalue" }' test.plist

Getting Values

It looks like -extract is meant to get values from a property list, but there is caveat. -extract will not merely get the value of a key in the property list but will write it to a new file! And by default if you do not provide an new output file path with the -o option it will overwrite the current file with the extracted data.

The proper, non-destructive syntax to use -extract is:

$ plutil -extract list xml1 -o - test.plist 
$ plutil -extract list json -o - test.plist 

This will print a full property list file to stdout. The -o - option tells plutil to print to stdout. You can give a filename instead of the -.

Since the output is encumbered with the json or xml syntax, it will be hard to use this to get to property list values in shell scripts. However, it still may be useful to, well, extract property list data from a complex plist file.

Conclusion

Keep in mind that there now is an alternative to defaults and PlistBuddy. Not having to convert a plist before changing data might be helpful, as well as the possibility to manipulate arrays and dictionaries with key paths. (You still should always use defaults when working with preference plist files, since defaults will go through the preferences system and possibility notify a process to update data.)

If you are using python or a similar high level scripting language it will still be more effective to use the libraries for property lists.

Prepare for autopkg Recipe auditing

Note: I am working on my next book ‘Automated Packaging for Apple Adminstrators’ and will cover this and other new features of autopkg v1.0 more in depth there. To pass the time until that book is published, get prepared with ‘Packaging for Apple Administrators’

A release candidate for autopkg v1.0 was released yesterday. As the version number implies, this is a big and important one.

Among a few other features, this release adds a new verb audit which checks a recipe and its parents for certain features which may have security implications. From the release notes:

New audit verb, used to output helpful information about any recipes that:

  • Are missing a CodeSignatureVerifier step
  • Use non-HTTP URLs for downloads
  • Supply their own processors and thus will run code not provided by AutoPkg itself
  • Use processors that may potentially be modifying the original software downloaded from the vendor

If you are hosting and sharing recipes, then there are a few steps you need to do to prepare for the release.

Once you have downloaded and installed the release candidate on your test machine, you can audit a recipe:

$ autopkg audit VMwareHorizonClient.download
    File path:        ./VMwareHorizonClient/VMwareHorizonClient.download.recipe
    Missing CodeSignatureVerifier

You can run audit against your entire repository with the find command:

$ cd ~/Library/AutoPkg/RecipeRepos/com.github.autopkg.scriptingosx-recipes/
$ find . -name '*.recipe' -exec autopkg audit {} ';' | open -f

This command pipes the output into TextEdit so you can review it better. You can of course pipe it into a file ( > audit.txt ) or your preferred text editor.

Then you have to work your way through the warnings.

Before you start working on fixes, you want to branch your repository, because some of the updated recipes may not work with older versions. You do not want to break your recipes until the final version of autopkg 1.0 is released. Remember to update the MinimumVersion value in your recipes.

There may be good reasons that you cannot fix all warnings. For example, there are a few products in my repository that aren’t signed by the developer, so I cannot add a CodeSignatureVerifier step.

I am not yet finished, but you can check out my branched recipe repository with the changes.

Here are a few notes as to what you may need to do:

Code Signature Verifier

Missing CodeSignatureVerifier

If the product you download is signed, you need to add a CodeSignatureVerifier Process to the download recipe. Read about this here: Using Code Signature Verification

Modifying Processors

The following processors make modifications and their use in this recipe should be more closely inspected:
        PkgCreator
        Copier

This warns of recipe processor that can change the content of what is downloaded. Of course in most cases this is intentional by the recipe author. However, the audit is merely warning you as a recipe user that you need to verify what is happening here.

As an author, you need to check if you can replace the common sequence of PkgRootCreator, Copier, PkgCreator with the new AppPkgCreator processor. This will not always be possible, but if you can this audit warning will go away (because AppPkgCreator does not change the content).

Insecure http URLs

    The following http URLs were found in the recipe:
        Input:
            DOWNLOAD_URL: http://download.ap.bittorrent.com/track/stable/endpoint/utmac/os/osx

Check if the software provider has secure https URLs instead.

Non standard Processors

    The following processors are non-core and can execute arbitrary code, performing any action.
    Be sure you understand what the processor does and/or you trust its source:
        Python3URLProvider

You will get this warning every time a recipe uses a Processor that is not part of the core processors provided by autopkg. If you use a custom processor to parse an URL and the version out of a website, you should check wether you can use URLTextSearcher instead.

Visual Studio on the Mac

At its heart, Visual Studio for Mac is a macOS counterpart of the Windows version of Visual Studio. If you enjoy the Visual Studio development experience, but need or want to use macOS, you should feel right at home. Its UX is inspired by Visual Studio, yet designed to look and feel like a native citizen of macOS. And like Visual Studio for Windows, it’s complemented by Visual Studio Code for times when you don’t need a full IDE, but want a lightweight yet rich standalone source editor.

Read more at MSDN