Things 3 is out!

My favorite to-do-list manager that I keep returning to has gotten the long awaited update!
Things 3 from Cultured Code is out.

I have been using the beta for a while now and just love the application. Things 3 has plenty of features, but also a gorgeous and clean interface.

You can download a trial for the Mac version from their website. You can purchase the Mac, iPhone and iPad versions from the respective AppStores. Currently the app is on 20% introductory sale until May 25.

Tab Completion for autopkg

Tony Williams aka ‘honestpuck’ has built a script to enable tab-completion for autopkg in bash.

This means that you can type

$ autopkg s⇥

(where ⇥ is the tab key) and it will autocomplete to

$ autopkg search 

This will also work for recipe names:

$ autopkg run BBEdit⇥⇥
BBEdit.download  BBEdit.jss       BBEdit.pkg       
BBEdit.install   BBEdit.munki     

This is really useful. Auto-completion not only saves on typing, but helps to avoid errors.

Installing autocompletion in your profile

Tony has provided instructions on how to install the script with brew. However, it not hard to install this manually in your .bash_profile or .bashrc. First, clone the github repository on to your system (I keep all projects like this in an un-creatively named ‘Projects’ folder):

$ cd ~/Projects
$ git clone https://github.com/Honestpuck/autopkg_complete.git

This will download the project to autopkg_create. The file we need is the autopkg file inside that folder.

Then add the following lines to your .bash_profile or .bashrc:

if [[ -r "$HOME/Projects/autopkg_complete/autopkg" ]]; then
    source "$HOME/Projects/autopkg_complete/autopkg"
fi

You will need to adjust the path if you are using a different location. Basically these lines say: if this file exists and is readable, then read and interpret it as bash source. Since you need to define functions in the context of the shell, you need to `source` the file, rather execute it as script. (When you run the the file as a script, the functions will be defined in the context of the script, and then ‘forgotten’ when the script ends.)

Save your new profile and open a new Terminal window or type

$ source ~/.bash_profile

to update an existing shell.

Thanks again to Tony Williams, this is very useful!

Weekly News Summary for Admins – 2017-05-12

On Scripting OS X

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

Updates and Releases

Posts and Opinion

Support and HowTos

To Listen

Where PATHs come from

In an earlier post we talked about how to append to the PATH variable, so you can add your own directories to bash’s search path.

In macOS the default PATH on a ‘clean’ installation is:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:

However, if you have installed some tools (such as the macOS Server.app, Xquartz or Munki) you will see those in the PATH as well:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Server.app/Contents/ServerRoot/usr/bin:/Applications/Server.app/Contents/ServerRoot/usr/sbin:/usr/local/munki:

Where does this pre-set PATH come from?

Since the PATH is pre-set on a clean new account without a .bashrc or .bash_profile, we have to look in a central location which applies to all users. In another earlier post, we saw that /etc/profile is run for every user shell, even before a .bash_profile is executed. When you look into this file, you see that the very first set of commands look like this:

if [ -x /usr/libexec/path_helper ]; then
    eval `/usr/libexec/path_helper -s`
fi

This looks very promising. The path_helper tool has a man page. This tool does a few things to assemble the PATH. First it reads the file /etc/paths which on macOS looks like this:

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

So this is where the default ‘clean’ macOS PATH comes from. Then path_helper will read every file from /etc/paths.d and append each line of each file in that directory to the PATH as well. This is where optional and third party applications and tools, like Xquartz or Munki, can install their own additions to the PATH for all users.

(Files in this folder will be read in alpha-numerical order of the filename. Some tools, like Xquartz, attempt to influence the order by preprending a number, e.g. 40-XQuartz.)

Finally, if path_helper runs in an environment where PATH is already set, it will append that PATH value to what it built from the files and then remove duplicates.

path_helper does not change the environment variable directly, but it generates the commands necessary to set the PATH correctly. It will generate the right commands wether it is called from a bourne type shell (on macOS: sh, bash, ksh and zsh) or a csh type shell (on macOS: csh and tcsh). You can see the output of the two styles by running path_helper with the -s or -c options:

$ /usr/libexec/path_helper -s
PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"; export PATH;
$ /usr/libexec/path_helper -c
setenv PATH "/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin";

(You have to type the full path to path_helper because, ironically, but intentionally, /usr/libexec is not in the standard PATH.)

To actually execute the commands generated by path_helper you can use the eval command, like the /etc/profile does:

$ eval $(/usr/libexec/path_helper)

Don’t touch my Profile!

Some command line tool installers understandably feel the need to add their tools to the default PATH. Since there is no unified approach among different flavors of UNIX and Linux on how to do this, you will find several different approaches. Some tools will edit /etc/profile and others will look for the various profile files in a user’s home directory and edit those. Usually the installation process will append a line that appends their tools directory to the PATH.

One example for this is the Python 3 installer. It contains a compnent package that will attempt to determine which profile file you are using and appends a line to append to the PATH.

However, this is not only highly intrusive but also quite fragile. Changes to /etc/profile might be overwritten by a future macOS update. Changes to a user’s profile file, might be overwritten by the user. Also the installer will only append their setting to the current user, not other users that may be present or created in the system.

Sample paths.d installer package

On the other hand, dropping a file into /etc/paths.d with a package installer will affect all users on a system. The file in paths.d can be updated for future updates if necessary and is also easily identified and removed by an admin. It will work wether it is installed to the startup volume or another volume. It can be pushed with management tools.

Building an installer for a file in /etc/paths.d is very simple:

$ mkdir CustomToolPathInstaller
$ cd CustomToolPathInstaller
$ mkdir payload
$ echo "/usr/local/customtool" >> payload/customtool
$ pkgbuild --root payload --install-location /private/etc/paths.d --version 1.0  --identifier com.example.customtool.path CustomToolPath.pkg
pkgbuild: Inferring bundle components from contents of payload
pkgbuild: Wrote package to CustomToolPath.pkg

Only five commands, three of which create the folder structure. You can find this sample project (which is slightly more elaborate) on my GitHub.

If you want to learn more about building installer packages for macOS, please read my book “Packaging for Apple Administrators”.

What about MANPATH?

This is usually not used on macOS since the the default settings for the man tool are quite flexibel. (Look at the man page for man and the file /etc/man.conf for details.) However, if a MANPATH environment variable is set when path_helper runs, it will also assemble the command to set the MANPATH built in a similar way to the PATH from the files /etc/manpaths and the directory /etc/manpaths.d.

Usually the MANPATH is not set on macOS so you will not see this. But if you want to manage your MANPATH and want to leverage path_helper all you have to do is set the MANPATH.

$ export MANPATH=/usr/share/man
$ /usr/libexec/path_helper
PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"; export PATH;
MANPATH="/usr/share/man:/usr/local/share/man"; export MANPATH;

Re-order the PATH

We have seen path_helper is extremely useful. There is one caveat, however. path_helper may reorder your PATH. Imagine you are pre-pending ~/bin to your PATH because you want to override some standard tools with your own. (Dangerous, but let’s assume you know what you are doing.) Then some process launches a subshell which can call path_helper again. path_helper will ‘find’ your additions to the defined PATH, but it will append to the list of default paths from /etc/paths and /etc/paths.d, changing your order and thus which tools will be used.

$ export PATH=~/bin:$PATH
$ echo $PATH
/Users/armin/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
$ /usr/libexec/path_helper 
PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/armin/bin"; export PATH;

You can see behavior like this when you use Xterm (The X11 based terminal in Xquartz) which does not execute .bash_profile but still picks up the PATHenvironment variable from somewhere…

# in Xquartz Terminal:
bash-3.2$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Server.app/Contents/ServerRoot/usr/bin:/Applications/Server.app/Contents/ServerRoot/usr/sbin:/usr/local/munki:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Users/armin/bin:/opt/X11/bin

A better way to override built-in commands which is not affected by path_helper would be to use bash aliases or functions in your profile.

Typefaces for Coding and Terminal

Since the previous posts were about customizing the shell for shell I thought I’d update an older post and look at some monospaced fonts suitable for Terminal and text editors to get a change from Menlo.

Not so serious, but fun…

C64 TrueType is a fun addition at the end. As the name implies this font recreates the 8-pixel characters from the C64. Together with some extra settings in Terminal and your bash_profile you can take your terminal back to the 80s.

If you have been following along my lose series on Terminal in macOS, this serves as a nice example of some more exotic Terminal customization.

Weekly News Summary for Admins – 2017-05-06

We had vacation last week, so summary is a bit late!

On Scripting OS X

To support Scripting OS X, consider buying one (or both) of my books. Thank you!

If you have already bought and read the books, please leave a review on the iBooks Store. Reviews are important to help new potential readers make the purchase decision. Thank you (again)!

News and Opinion

Conferences

Support and HowTos

To Listen

MacSysAdmin 2017 Conference Open for Registration

You can now register for the MacSysAdmin Conference in Göteborg (Sweden) which will go from October 3-6.

It is a great conference and I have wanted to go for years, but never managed to make it. The list of speaker has been and is really very impressive.

So I am very proud to announce that not only will I be attending but also presenting a session on macOS bash scripting this year! (so literally a session on ‘Scripting OS X… macOS’)

Looking forward to seeing you all there!

Configuring bash with aliases and functions

In the previous posts we talked about which files you could use to customize your bash environment.

There are (mainly) three customizations you can perform:

This post will look at some useful aliases and functions.

bash aliases

Note: bash aliases are something completely different from Finder aliases. The closest shell equivalent for Finder alias files are symbolic links.

bash aliases are basically text substitutions. For example a common alias is to define:

alias ll="ls -l"

You can type this alias definition directly into bash but then the definition will only exist for that particular shell. When you open a new terminal window it will know nothing of this alias. If you want an alias to exist in all your shells (and usually you do), you need to add the above line to your .bash_profile or .bashrc. (Learn about the difference here.)

Whenever you modify the .bash_profile or .bashrc it will not automatically be loaded into any shell that is already open, you either have to close the Terminal window and open a new one or write

$ source ~/.bash_profile

(or ~/.bashrc) to load the new settings.

Once you have set the alias, anytime you type ll at the start of a command, bash will replace it with ls -l before executing. Since subsequent arguments are left alone they will just be picked up by the substituted ls command, so if you type ll -a bash will substitute that to ls -l -a and it will work just your would expect.

You can make the alias more complex if you want:

alias lll="ls -alhTOe@"

If you always want to use the long format of ls, you can alias the ls command itself:

alias ls="ls -l"

Then, when ever you type ls it will be replaced with ls -l. Note the lack of spaces around the ‘=’, as usual when assigning values in bash.

Uncovering the alias

You can unset or delete an alias with the unalias command:

$ unalias ls

This will return to the default ls command (for this bash instance only).

Since alias substitution only happens at the beginning of the command, you can also bypass it once by starting the command line with a different character:

$ "ls" -S
$ \ls -S

Either way of typing this will use the original ls command, avoiding substitution, just this once.

If you are unsure if a command has an alias or want to know what gets substituted, then you can have the alias definition printed with

$ alias ls
alias ls='ls -l'
$ alias lll
alias lll='ls -alhTOe@'

You can also list all defined aliases by running alias without any arguments:

$ alias
alias ll='ls -l'
alias lll='ls -alhTOe@'
alias ls='ls -l'

Some Alias examples

Some users like to alias the potentially dangerous commands rm, mv and cp with the -i option, which forces the user to confirm when a file is going to be deleted or overwritten:

alias rm="rm -i"
alias mv="mv -i"
alias cp="cp -i"

Then if you do something that could destroy an existing file you will be prompted to confirm:

$ rm my_important_file 
remove my_important_file? 

You can still use \rm to bypass the alias, if you believe you know what you are doing and want to avoid being prompted several times.

You can add short cuts to cd to parent folders:

alias ..="cd .."
alias ...="cd ../.."
alias cd..="cd .."

Since alias substitution only takes place at the beginning of the command, you can still use .. normally as an argument.

Note that the last alias cd.. is a substitution of a common typo. Since the output of the alias is not checked for further alias substitutions you cannot use alias cd..=.. to define it using the previous alias. Each alias must stand for itself.

You can also use aliases to make complex or hard to memorize commands easier to remember:

alias badge="tput bel"

The first command will play a system alert and set a ‘badge’ on the terminal application if it is in the background. This is useful to get notified of long running commands:

$ hdiutil convert image.sparseimage -format UDZO image.dmg; badge;

The substitution can be long and multiple commands can be separated by ;

alias dockspace="defaults write com.apple.dock persistent-apps -array-add '{\"tile-type\"=\"spacer-tile\";}'; killall Dock;"

The dockspace alias uses this hack to add a spacer item to the dock.

We have seen earlier with the ll alias that any text or arguments after the alias will just be added to the substituted command. We can use this to our advantage:

alias reveal='open -R'
alias xcode='open -a Xcode'
alias pacifist='open -a Pacifist'

All of these commands will require a file path as an ‘argument’ to work properly.

Functions: beyond the alias

While bash aliases are useful and easy to define, they have limitations. Aliases will only be replaced at the beginning of the command prompt. Also additional arguments will be appended after the replacement. If you need to insert additional arguments somewhere in the replacement, you cannot achieve that with aliases.

For example, to display the man page for ls in the Preview application you need the following commands:

$ man -t ls | open -f -a "Preview"

If you wanted to alias this you would have to insert the arguments after man -t and before the pipe. Since aliases, only substitute the beginning of the command, this will not work. we need to define a bash function:

function preman() {
    man -t $@ | open -f -a "Preview"
}

Sometimes you will see bash functions defined without the function keyword, but I prefer to use it since it makes the code more readable.

The $@ will be replaced with all the arguments passed into the function, so when use the function

$ preman ls

the argument ls will be substituted where the $@ is.

You can also use the positional argument variables $1, $2, etc:

function recipe-open() {
    open "$(autopkg info '$1' | grep 'Recipe file path' | cut -c 22-)"
}

Within functions the full power of bash scripting is at your service. You can even call other scripting languages.

# prints the path of the front Finder window. Desktop if no window open
function pwdf () {
    osascript <<EOS
        tell application "Finder"
            if (count of Finder windows) is 0 then
                set dir to (desktop as alias)
            else
                set dir to ((target of Finder window 1) as alias)
            end if
            return POSIX path of dir
        end tell
EOS
}

# changes directory to frontmost 
alias cdf='pwdf; cd "$(pwdf)"'

The last alias cdf has to be defined as an alias. Since a function or script could not change the directory for the current shell, you have to use alias substitution to get the cd.

In general functions (and scripts) are more powerful and versatile than aliases, but aliases provide and easy and comparatively safe way to customize the your shell environment.

Putting it all together

Here is a sample .bash_profile with many of the examples given here and in the previous article. You can use this as a starting point for your own bash configuration.