On env Shebangs

There was a comment to my previous post about using the /usr/bin/env shebang to gain system portability.

Regarding shebang, a more system independent way would be to use e.g. ‘#!/usr/bin/env bash’. With this the script would also be usable on FreeBSD, where bash would be installed from FreeBSD Ports and then be available as ‘/usr/local/bin/bash’ or some Linux systems (e.g. Debian) where it is ‘/usr/bin/bash’. Unfortunately there are some other unixode systems around, where ‘env’ is not in ‘/usr/bin/’ and so the shebang needs to be adjusted.

(I replied to the comment there, but then realized this deserves its own post.)

The /usr/bin/env shebang provides a means for system portability. It has many valid use cases. However, you don’t just magically gain portability by switching to an env shebang. There are many trade-offs to consider.

The note on how the env binary may not be in /usr/bin on all platforms, hints at some of these trade-offs, but there are more.

The trade-offs are a loss of predictability and reliability, or functionality, or increased maintenance and management.

Let me elaborate…

How the /usr/bin/env shebang works

When used in the shebang, the /usr/bin/env binary will use the current environment’s PATH to lookup the interpreter binary for the script, in the same way the shell looks up commands.

As an example, let us imagine you have installed bash v5 on your Mac. Either manually or using brew or some other package management system.

This will (usually) put the bash v5 binary at /usr/local/bin/bash. The /usr/local/bin directory is a common choice for custom command line tools, because it is part of the default PATH for interactive shells on macOS and not protected by SIP/SSV. The default PATH on macOS for interactive shells is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Some installations will put the binary in a different location in the file system. Then you you have to pre-pend the directory containing the binary to your PATH variable in your shell configuration. The order of the directories in the PATH is important, because the shell and env will stop the search when they find the first match. If /usr/local/bin came after /bin in the PATH the new binary would not be ‘seen’ since the pre-installed, old /bin/bash binary is found first.

Some installations solve this by placing a symbolic link to the binary in /usr/local/bin.

• Where PATHs come from

When you run a script from the interactive shell with a shebang of #!/usr/bin/env bash, then env would find the bash v5 binary first in /usr/local/bin, so your script is interpreted with bash v5. This is probably what you were hoping for, when you installed bash v5 in the first place.

When you run the same script on a different Mac (same macOS version, but it doesn’t have bash v5 installed) env will pick up /bin/bash. Your script will work even though that other Mac doesn’t have /usr/local/bin/bash, so you gained portability.

However, /bin/bash is bash v3.2, so your script may behave differently. If the script uses bash v5 features that are not available in the 15-year-old bash v3.2, it will generate errors. Since you actively chose to install bash v5 on the first Mac, it is likely you needed some of these bash v5 features, so it is likely your script will fail on other Macs, which don’t have bash v5 installed.

You lost either predictability and reliability (which version and features are available? Does my script run successfully?), or you lose functionality (the features added to bash v5 since v3.2). To retain reliability, you can restrict the script to features that work in both bash versions. But then using the env shebang gives you no advantage, and you might as well use /bin/bash as the shebang.

Some solutions

One alternative is to use a /usr/local/bin/bash shebang for scripts which use bash v5 functionality and continue to use /bin/bash for scripts that need to run across multiple Macs, where you pay attention to using only features available in bash v3.2. You gain predictability and reliability, but your bash v5 scripts aren’t portable to other Macs. They may even fail on other Macs with bash v5 installed, if the bash v5 binary is installed in a different location.

When you use /usr/bin/env bash for a bash v5 script, it will run fine on all Macs which have bash v5 installed and the PATH configured properly to find it. (Configuring and maintaining the PATH does not happen on its own.) But the script will still fail on Macs without any bash v5. You can (and probably should) add a version check to the script, but now you are increasing code maintenance.

When you are managing a fleet of Macs, you also have the option (or in this case, I would say, the duty) to install bash v5 in a consistent location and version across all Macs in your fleet and pre-configure the proper PATH in the contexts the script will run in. Then you get predictability and functionality, but it requires extra effort in deployment and maintenance.

This requires a decently experienced MacAdmin and the proper tooling, neither of which comes for free.

Note: There are great open source solutions for macOS in this area, but I consider them ‘free, as in puppy,’ so they come with higher skill requirements and/or maintenance effort for the admin. And this isn’t supposed to imply that all commercial solutions are ‘easy to use,’ either. It’s trade-offs all the way down.

Context changes the PATH

Notice that so far I kept talking about “the default PATH for the interactive shell.”

The PATH variable may be different depending on the context, even on the same Mac with the same user. For example, when you run your script with the AppleScript do shell script command, the PATH in that context is not the same as the PATH in your interactive shell. It will be:

/usr/bin:/bin:/usr/sbin:/sbin

You can verify this by opening Script Editor and running the do shell script "echo $PATH". Other context, like scripts in installation packages will see other PATH values.

• Setting the PATH in Scripts

Most importantly, the PATH in these other contexts, does not contain /usr/local/bin, or any other addition you made to your PATH in the shell configuration files. An /usr/bin/env shebang will not ‘see’ a bash 5 binary you installed on the system. The same script with the same user on the same computer, will behave differently when run in a different context.

These different PATH values are an intentional choice. In these contexts, especially installation package scripts, reliability and predictability are extremely important. You do not want user and third-party installed custom binaries to interfere with the command lookup.

Sidenote on Python

With python and python3 and other run time interpreters, it gets even more difficult. There may multiple different versions installed and the behavior and functionality between versions varies more. My Mac currently has four different Python 3 binaries, each with a different version, and I not even remotely a full-time Python developer. When you call python3 on a non-developer Mac it will trigger the ‘You have to install Developer Command Line Tools’ dialog when Xcode is not installed. (Developers seem to have a hard time considering that there are Macs without Xcode installed.)

With the demise of python 2 in macOS 12.3, some developers reacted by changing the shebang in their python scripts from /usr/bin/python to /usr/bin/env python which solves nothing, when the binary goes away without replacement. Some switched to /usr/bin/env python3 which can makes things worse, by triggering the Developer Tools installation or picking a random python3 binary of the wrong version.

• macOS Monterey 12.3 removes Python 2 – Link collection

The only reliable solution for the mess that is python is to deploy a consistent version of Python 3 in a consistent location. You can do this by either bundling the python framework and module your tool needs together with the tool, or by deploying and maintaining the Python frameworks and modules with a management system.

MacAdmin perspective

As a MacAdmin, my scripts don’t need to be portable to systems other than macOS. They usually use tools and access files and folders that only exist on macOS. Predictability and reliability, however, are paramount. Configuration and installation scripts need to run reliably on thousands of Macs across multiple versions of macOS (even future versions) regardless of what else is installed.

As MacAdmins, we also (should) have the tools and experience to deploy and maintain binaries in predictable locations. But then, like everyone, we only have limited time, and often need to prioritize business critical software over our own tooling. So, the pre-installed interpreter binaries have little ‘friction’ to use, even if they may have a reduced functionality when compared to the latest version available elsewhere.

This is the reason bash v3.2 is still present on macOS 12.3 and it will never be easy when Apple ultimately decides to remove it. So many tools and scripts rely on /bin/bash.

(I don’t expect the removal to be any time soon, but there is a limit to how long Apple will or can keep this interpreter from 2007 on the system. We got the first warning when Apple switched the default interactive shell to zsh in Catalina. There will be more warnings …I hope. With the removal of the Python 2 binary we saw that Apple can move quickly when they feel the need. They did not even wait for a major macOS release.)

In this context, there is no gain in using /usr/bin/env. The trade-offs favor the absolute shebang very strongly.

Cross-platform portability

After this rant, you may think that I recommend against using /usr/bin/env shebangs always. But there are very good use cases. Using /usr/bin/env shebangs is the solution for workflows where cross-platform portability is required.

When your scripts need to run across multiple platforms, installing the binaries in the same location in the file system may not be possible, or require an unreasonable effort. For example, the /bin and /usr/bin are protected by SIP and the Sealed System Volume on macOS, so you cannot add your tooling there without significantly impacting the integrity and security of the entire system.

In these cases, using a /usr/bin/env shebang provides the required flexibility, so your scripts can work across platforms. But the portability does not come magically from just switching the shebang.

The target platforms need the binary to be installed and the versions should match. The installation location of the binary has to be present in the PATH in the context the script runs in. To get reliable behavior, the systems you are porting between need to be well managed, with a predictable setup and configuration of the interpreter binary and environment.

When your scripts work ‘effortlessly’ across systems with the env shebang, it is thanks to the work of the platform developers and your sysadmins/devops team for creating and maintaining this consistency. Even if you are the sole developer and devops admin, maintaining all the systems, you have to put in this work. Also the platform developers put in a lot of effort to achieve much of this consistency out of the box. As the commenter noted, some platforms don’t even agree where the env binary should be.

You gain portability at the price of increased maintenance.

Trade-offs all the way down

Alternatively, you can keep the scripts simple – restricted to a subset of features common across platforms and versions – so that the differences have no impact. Then you trade for reliability and portability at the price of functionality.

This is often the trade-off with Python (restrict the use of python features to those common among versions) and one of the reasons Python 2 was kept around for so long on macOS.

Using POSIX sh, instead of bash or zsh, is another option to gain portability, but that has its own trade-offs. Most of these trade-offs will be in functionality, but also consider not all sh emulations are equal, and supporting multiple different emulators or the real common subset of functionality, requires extra effort.

Conclusion

Shebangs with absolute paths have their strengths and weaknesses, as do shebangs with /usr/bin/env. Each has their use case and you have to understand the trade-offs when using either. Neither env shebangs nor absolute path shebangs are generally ‘better.’ Either one may get you in trouble when used in the wrong context.

When someone says ‘you should be using,’ or ‘a better way would be,’ you always need to consider their context, use case, and which trade-offs they are accepting. You need to understand the tools to use them efficiently.