Wrangling Pythons

As I noted in my last Weekly News Summary, several open source projects for MacAdmins have completed their transition to Python 3. AutoPkg, JSSImport and outset announced Python 3 compatible versions last week and Munki already had the first Python 3 version last December.

Why?

Apple has included a version of Python 2 with Mac OS X since 10.2 (Jaguar). Python 3.0 was released in 2008 and it was not fully backwards compatible with Python 2. For this reason, Python 2 was maintained and updated alongside Python 3 for a long time. Python 2 was finally sunset on January 1, 2020. Nevertheless, presumably because of the compatibility issues, Apple has always pre-installed Python 2 with macOS and still does so in macOS 10.15 Catalina. With the announcement of Catalina, Apple also announced that in a “future version of macOS” there will be no pre-installed Python of any version.

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. (macOS 10.15 Catalina Release Notes)

This also applies to Perl and Ruby runtimes and other libraries. I will be focussing on Python because it is used more commonly for MacAdmin tools, but most of this post will apply equally to Perl and Ruby. Just mentally replace “Python” for your preferred language.

The final recommendation is what AutoPkg and Munki are following: they are bundling their own Python runtime.

How to get Python

There is a second bullet in the Catalina release notes, though:

Use of Python 2.7 isn’t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS won’t include Python 2.7. Instead, it’s recommended that you run python3 from within Terminal. (51097165)

This is great, right? Apple says there is a built-in Python 3! And it’s pre-installed? Just move all your scripts to Python 3 and you’ll be fine!

Unfortunately, not quite. The python3 binary does exist on a ‘clean’ macOS, but it is only a stub tool, that will prompt a user to download and install the Command Line Developer Tools (aka “Developer Command Line Tools” or “Command Line Tools for Xcode”). This is common for many tools that Apple considers to be of little interest to ‘normal,’ non-developer users. Another common example is git.

Dialog prompting to install the Command Line Tools
Dialog prompting to install the Command Line Tools

When you install Xcode, you will also get all the Command Line Developer Tools, including python3 and git. This is useful for developers, who may want to use Python scripts for build operation, or for individuals who just want to ‘play around’ or experiment with Python locally. For MacAdmins, it adds the extra burden of installing and maintaining either the Command Line Developer Tools or the full Xcode install.

Python Versions, a multitude of Snakes

After installing Xcode or the Command Line Developer Tools, you can check the version of python installed: (versions on macOS 10.15.3 with Xcode 11.3.1)

> python --version    
Python 2.7.16
> python3 --version    
Python 3.7.3

When you go on the download page for Python.org, you will get Python 3.8.1 (as of this writing). But, on that download page, you will also find download links for “specific versions” which include (as of this writing) versions 3.8.1, 3.7.6, 3.6.10, 3.5.9, and the deprecated 2.7.17.

The thing is, that Python isn’t merely split into two major release versions, which aren’t fully compatible with each other, but there are several minor versions of Python 3, which aren’t fully compatible with each other, but are still being maintained in parallel.

Developers (individuals, teams, and organisations) that use Python will often hold on to a specific minor (and sometimes even patch) version for a project to avoid issues and bugs that might appear when changing the run-time.

When you install the latest version of Munki, it will install a copy of the Python framework in /usr/local/munki/ and create a symbolic link to that python binary at /usr/local/munki/python. You can check its version as well:

 % /usr/local/munki/python --version
Python 3.7.4

All the Python code files for Munki will have a shebang (the first line in the code file) of

#!/usr/local/munki/python

This ensures that Munki code files use this particular instance of Python and no other copy of Python that may have been installed on the system.

The latest version of AutoPkg has a similar approach:

> /usr/local/autopkg/python --version    
Python 3.7.5

In both cases the python binary is a symbolic link. This allows the developer to change the symbolic link to point to a different Python framework. The shebangs in the all the code files point to the symbolic link, which can be changed to point to a different Python framework.

This is useful for testing and debugging. Could MacAdmins use this to point both tools to the same Python framework? Should they?

The Bridge to macOS

On top of all these different versions of Python itself, many scripts, apps, and tools written in Python rely on ‘Python modules.’ These are libraries (or frameworks) of code for a certain task, that can be downloaded and included with a Python installation to extend the functionality of Python.

The most relevant of these modules for MacAdmins is the “Python Objective-C Bridge.” This module allows Python code to access and use the native macOS Cocoa and CoreFoundation Frameworks. This not only allows for macOS native GUI applications to be written in Python (e.g. AutoDMG and Munki’s Managed Software Center [update: MSC was re-written in Swift last year]), but also allows short scripts to access system functions. This is sometimes necessary to get a data that matches what macOS applications “see” rather than what the raw unix tools see.

For example, the defaults tool can be used to read the value of property lists on disk. But those might not necessarily reflect the actual preference value an application sees, because that value might be controlled by a different plist file or configuration profile.

(Shameless self-promotion) Learn more about Property lists, Preferences and Profiles

You could build a tool with Swift or Objective-C that uses the proper frameworks to get the “real” preference value. Or you can use Python with the Objective-C bridge:

#!/usr/bin/python
from Foundation import CFPreferencesCopyAppValue
print CFPreferencesCopyAppValue("idleTime", "com.apple.screensaver")

Three simple lines of Python code. This will work with the pre-installed Python 2.7, because Apple also pre-installs the Python Objective-C bridge with that. When you try this with the Developer Tools python3 you get an error:

ModuleNotFoundError: No module named 'Foundation'

This is because the Developer Tools do not include the Objective-C bridge in the installation. You could easily add it with:

> sudo python3 -m pip install pyobjc

But again, while this command is “easy” enough for a single user on a single Mac, it is just the beginning of a Minoan labyrinth of management troubles.

Developers and MacAdmins, have to care about the version of the Python they install, as well as the list of modules and their versions, for each Python version.

It is as if the Medusa head kept growing more smaller snakes for every snake you cut off.

(Ok, I will ease off with Greek mythology metaphors.)

You can get a list of modules included with the AutoPkg and the Munki project with:

> /usr/local/munki/python -m pip list
> /usr/local/autopkg/python -m pip list

You will see that not only do Munki and AutoPkg include different versions of Python, but also a different list of modules. While Munki and AutoPkg share many modules, their versions might still differ.

Snake Herding Solutions

Apple’s advice in the Catalina Release Notes is good advice:

It’s recommended that you bundle the runtime within the app.

Rather than the MacAdmin managing a single version of Python and all the modules for every possible solution, each tool or application should provide its own copy of Python and its required modules.

If you want to build your own Python bundle installer, you can use this script from Greg Neagle.

This might seem wasteful. A full Python 3 Framework uses about 80MB of disk space, plus some extra for the modules. But it is the safest way to ensure that the tool or application gets the correct version of Python and all the modules. Anything else will quickly turn into a management nightmare.

This is the approach that Munki and AutoPkg have chosen. But what about smaller, single script solutions? For example simple Python scripts like quickpkg or prefs-tool?

Should I bundle my own Python framework with quickpkg or prefs-tool? I think that would be overkill and I am not planning to do that. I think the solution that Joseph Chilcote chose for the outset tool is a better approach for less complex Python scripts.

In this case, the project is written to run with Python 3 and generic enough to not require a specific version or extra modules. An admin who wants to use this script or tool, can change the shebang (the first line in the script) to point to either the Developer Tool python3, the python3 from the standard Python 3 installer or a custom Python version, such as the Munki python. A MacAdmin would have to ensure that the python binary in the shebang is present on the Mac when the tool runs.

You can also choose to provide your organization’s own copy Python with your chosen set of modules for all your management Python scripts and automations. You could build this with the relocatable Python tool and place it in a well-known location the clients. When updates for the Python run-time or modules are required, you can build and push them with your management system. (Thanks to Nathaniel Strauss for pointing out this needed clarifying.)

When you build such scripts and tools, it is important to document which Python versions (and module versions) you have tested the tool with.

(I still have to do that for my Python tools.)

What about /usr/bin/env python?

The env command will determine the path to the python binary in the current environment. (i.e. using the current PATH) This is useful when the script has to run in various environments where the location of the python binary is unknown.

This is useful when developers want to use the same script in different environments across different computers, user accounts, and platforms. However, this renders the actual version of python that will interpret the script completely unpredictable.

Not only is it impossible to predict which version of Python will interpret a script, but you cannot depend on any modules being installed (or their versions) either.

For MacAdmin management scripts and tools, a tighter control is necessary. You should use fixed, absolute paths in the shebang.

Conclusion

Managing Python runtimes might seem like a hopeless sisyphean task. I believe Apple made the right choice to not pre-install Python any more. Whatever version and pre-selection of module versions Apple would have chosen, it would only have been the correct combination for a few Python solutions and developers.

While it may seem wasteful to have a multitude of copies of the Python frameworks distributed through out the system, it is the easiest and most manageable solution to ensure that each tool or application works with the expected combination of run-time and modules.

Weekly News Summary for Admins — 2020-02-07

We got a remarkable series of open source updates this week. AutoPkg, JSSImporter, and Outset all published updates that make the project independent of the pre-installed Python 2.7 on macOS. Since Python 2.7 will be deprecated really soon, Apple had announced that future macOS versions will not include a pre-installed Python anymore. These projects join Munki which had the update with “Python” independency with version 4.0 in December. Thanks to all contributors who put their efforts into these open source projects!

The next beta cycle for the Apple operating systems started this week. Once again, it looks as it if the “Spring Updates” for macOS 10.15.4 and iOS 13.4 will add features and changes relevant for users and administrators. Check out the details in the AppleSeed for IT portal.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

MacAdmins on Twitter

  • Martin Lang: “The number of SAP colleagues who chose a Mac as their primary computer has doubled over the last 15 months, it’s at almost 26K now with 82% of them running the latest macOS Catalina”
  • Laura Rösler: “We‘ve finally more than 26,000 Macs at SAP ….and crossed the 85% for macOS Catalina.”
  • Andrew Laurence: “For those in the ITlife who might not be familiar: MS plans to flip configurations on DCs, requiring either packet signing on LDAP connections, or using TLS with LDAP. This has been recommended for years.” (Thread)
  • Parker Higgins: “I used to wonder why the interfaces on Star Trek are so clunky given that it’s centuries in the future, but I guess that’s just Enterprise software for you”

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Watch

To Listen

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Getting the current user in macOS – Update

I recently posted an article on how to get the current user in macOS. If you read through that entire post, it’ll be obvious that it was actually an excuse to write about the deprecation of python 2 and /bin/bash and the situation with sh ‘compatible’ code.

The ‘tl;dr’ of that post was to stop using the python one-liner and use this scutil based one-liner, based on Erik Berglund’s post, instead.

zsh and bash scripts

loggedInUser=$( scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

sh scripts

loggedInUser=$( echo "show State:/Users/ConsoleUser" | scutil | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

From now on, I will call these the ‘SystemConfiguration’ solution (this also includes the python one-liner). Both the python and the scutil based one-liners go through the SystemConfiguration framework to get at the desired information: which user is using the ‘Console?’

The ‘console’ in this case is an abstract definition, which encompasses the screen output and most inputs. I could not find a clear definition of what the ‘Console’ is exactly on macOS. (If you can point me to one, I’d be grateful.) But it serves to think of the console as all the processes, inputs and outputs attached to a user’s login session.

The other one-liner

There is another solution that keeps popping up in MacAdmins Slack and blog posts.

loggedInUser=$(stat -f %Su /dev/console)

(The stat solution.) This certainly looks much shorter and simpler. But does it return the same in all situations? Is it actually the same?

Testing the Edge

I have done some testing with Fast User Switching and a second remote Screen Sharing session. These are the weird edge cases for which the original python one-liner was devised.

I had a remote ssh session open from my main Mac while I logged two test users in and out of a second test Mac. I ran the commands for the SystemConfiguration and stat solutions in the ssh session, and compared the return values.

In (nearly) all multiple user sessions combinations that I tested, both the SystemConfiguration and the stat solution return the same value.

It is worth pointing out, that when you have two users with graphical sessions at the same time, neither of the two solutions will actually return the “currently” active user session, but the user owning the sessions that was last logged in to. This is especially relevant when someone use Screen Sharing to create a remote graphical session on an Mac. In this case both methods will return the last user that logged in, even though both users currently have active sessions.

How the stat solution works

The stat command shows the user owning the /dev/console file. When we look at that ‘file’ with the ls command:

> ls -al /dev/console  
crw-------  1 armin  staff    /dev/console

We get a file type (the first character of the listing) of ‘c’ which represents a ‘character special file.’ All the ‘files’ in /dev are of this file type. When you look closely you will ‘files’ in here representing the disks connected to your Mac (disk*), the Bluetooth devices (cu.*), and some other weird ‘files.’

This stems back to the Unix underpinnings of macOS where everything has a representation in the file system hierarchy, whether it is actual data on a disk or not. The ‘files’ we are seeing here are an abstraction of devices and other objects in the system.

With that background in mind, it should be clear that we are looking at the same data from the system, abstracted in different ways. Either through the SystemConfiguration framework or the Unix file system. So, we get the same data either way, right?

The difference at login window

Not quite. I found one difference. When the Mac is at the login window, the SystemConfiguration solution returns an empty string and the stat solution returns root.

This is not the entire truth, though. When you look closely at the SystemConfiguration solution, you will that there is a clause, in both the python and scutil versions, that catches a return value of loginwindow and returns and empty string instead.

When you look at the raw output of the scutil command used, we actually get quite a lot of information:

> scutil <<< "show State:/Users/ConsoleUser"
<dictionary> {
  GID : 20
  Name : armin
  SessionInfo : <array> {
    0 : <dictionary> {
      kCGSSessionAuditIDKey : 100009
      kCGSSessionGroupIDKey : 20
      kCGSSessionIDKey : 257
      kCGSSessionLoginwindowSafeLogin : FALSE
      kCGSSessionOnConsoleKey : TRUE
      kCGSSessionSystemSafeBoot : FALSE
      kCGSSessionUserIDKey : 501
      kCGSSessionUserNameKey : armin
      kCGSessionLoginDoneKey : TRUE
      kCGSessionLongUserNameKey : Armin Briegel
      kSCSecuritySessionID : 100009
    }
  }
  UID : 501
}

The awk statement after the scutil command filters out the information we need:

awk '/Name :/ && ! /loginwindow/ { print $3 }' 

This will return lines containing Name : and not containing loginwindow. In other words, if the scutil command returns loginwindow in the line, the entire scutil ... | awk ... construct will return nothing or an empty string.

When you leave out the extra awk clause to filter the loginwindow return value, you will get loginwindow while the login window is shown:

scutil <<< "show State:/Users/ConsoleUser" | awk '/Name :/  { print $3 }'

The edge of the edge case

So, the SystemConfiguration solution returns loginwindow or nothing when the Mac is sitting at a login window and the stat solution returns root. Not a big deal right? I just have to consider this difference when I test whether a user is logged in?

I found one edge case (to the edge case) where this difference matters: the stat solution will return root when at the login window and when the root account or System Administrator is logged in.

Now, enabling the root account and logging in and working as root is most definitely not recommended on macOS. So, there is an argument that this edge case should not occur.

But users have a way of ignoring these kind of recommendations. The SystemConfiguration solution will allow your scripts to actually distinguish between a Mac sitting at the login window and the (admittedly rare but extremely dangerous) situation where a user is logged in as root.

Conclusions

So, in most cases the stat solution is identical to the SystemConfiguration solution. It has the advantage of being shorter and easier to read.

When you anticipate that you might need to distinguish between the computer being at the login window and a root user being logged in, or when you are just paranoid about catching all edge cases, the SystemConfiguration solution is more complete.

If in doubt, use the SystemConfiguration solution.

Weekly News Summary for Admins — 2020-01-31

Update week! macOS Catalina 10.15.3 and the iOS 13.3.1 and other related updates dropped. And while the Catalina update seems to fix some things it also breaks others. Thanks to the all the contributors for the extra information that Apple does not provide.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

📰News and Opinion

🐦MacAdmins on Twitter

  • Sarah: “Well, it’s racked. Looks kind of funny next to an Xserve though. https://t.co/BnKwR499VI” (Images)
  • Rosyna Keller: “Don’t use PackageMaker to make installer packages. It’s a long dead tool (2012 or so) and developers should switch to productbuild/pkgbuild*, which incorporate the command line utilization. It also can’t create signed packages. *Or is a third-party toool that scripts these. Installer packages are required to be signed to be notarized (so PackageMaker packages are out) and, as PackageMaker was a 32-bit app, it won’t run on Catalina. It’s that old.”

🔨Support and HowTos

🤖Scripting and Automation

🍏Apple Support

🏝macOS Catalina 10.15.3 and iOS 13.3.1

♻️Updates and Releases

🎧To Listen

🎈Just for Fun

📚 Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Weekly News Summary for Admins — 2020-01-24

We got the third group beta releases for the next updates this week. Can I just gripe for a moment that macOS 10.15.2 was released along side iOS 13.3 and watchOS 6.1.1, while 10.15.3 will (probably) be released along side iOS 13.3.1 and watchOS 6.1.2? Apple versioning makes no sense at all, any more.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

News and Opinion

MacAdmins on Twitter

  • Tim Perfitt: “Reboot to recovery on next reboot: sudo nvram "recovery-boot-mode=unused" Subsequent reboots go back to macOS.”
  • Roben Kleene: “Some of the built-in scripting language binaries (only the interactive ones?) now print a warning message about not being included by default in a future version of macOS. Just in case you were wondering how serious they are about this: They’re serious.”
  • Jeff Johnson: “I’ve learned that Apple engineers have internal tools which allow them to delete macl xattr as well as to bypass other Catalina privacy and sandbox protections without rebooting and disabling SIP. Inside Apple they don’t suffer the same problems as external users and developers.”

Support and HowTos

Scripting and Automation

Updates and Releases

To Listen

Just for Fun

  • Jenny on Twitter: “No one asked for this. But here it is: every macOS wallpaper from Mac OS X 10.0 Cheetah to macOS 10.15 Catalina combined.” (Follow link for image)

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Weekly News Summary for Admins — 2020-01-17

Somewhat slow week for MacAdmin news. The new year is still ramping up.

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

On Scripting OS X

News and Opinion

On Twitter

  • Steve Hayman: “Happy 40th Birthday to one of my favourite Unix features, the Shebang! Dennis Ritchie introduced the two-character #! sequence – which tells Unix how to execute a script file – 40 years ago today.”
  • Carl Ashley: “Interrogating a preinstall script for a package. It’s doing naughty things. This is just a sample. What package sins keep you awake at night, macadmins?… ” (Image)

Bugs and Security

Support and HowTos

Scripting and Automation

To Listen

Updates and Releases

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Weekly News Summary for Admins — 2020-01-10

Happy New Year!

New decade, even. Well, the last decade has certainly been… interesting.

What ever the new year (and decade) brings from the Apple Admin side of things, I will link to it!

I will be presenting at MacAD.UK (March 24–25) this year again. The topic of my session will be “Terminal Wizardry and and Witchcraft 2020” which will pick up on the presentation from two years ago. Much has changed since then and it is time for an update. I am looking forward to meeting you there!

Quite a bunch of links this time, because we were off for two weeks and this is the catch up summary. You can tell that most MacAdmins were able to take some time off as well, except for Howard Oakley (Eclectic Light) who just kept publishing like he always does. (Thank you for all your work, Howard!)

If you would rather get the weekly newsletter by email, you can subscribe to the Scripting OS X Weekly Newsletter here!! (Same content, delivered to your Inbox once a week.)

Headlines

“Apple Device Management: A Unified Theory of Managing Macs, iPads, iPhones, and AppleTVs” by Rich Trouton and Charles Edge

On Scripting OS X

News and Opinion

MacAdmins on Twitter

  • Fiora: “looks like dropbox doesn’t like me using dropbox as an automatic backup service… ” (Thread)
  • Jason Broccardo: “It’s 2020 and you know what that means, right? Adobe Flash Player dies at the end of this year!”
  • Roy van Rijn: “One of our office chairs turns off monitors… we couldn’t believe it, but we have it on tape. Surprisingly, there even is a known issue for it”

Bugs and Security

Support and HowTos

Scripting and Automation

Apple Support

Updates and Releases

To Listen

Just for Fun

Support

There are no ads on my webpage or this newsletter. If you are enjoying what you are reading here, please spread the word and recommend it to another Mac Admin!

If you want to support me and this website even further, then consider buying one (or all) of my books. It’s like a subscription fee, but you also get a useful book or two extra!

Command-Control-Power Podcast featuring Scripting OS X!

I have the honor of being a guest on the latest Command-Control-Power podcast!

The wonderfully gracious hosts Joe Saponare and Sam Valencia talk with me about life as admin/consultant, my new book “Moving to zsh,” what the new shell means for Mac users and also the EraseInstall app we built at ProWarehouse and how workflows are important for admins and consultants.

I had a great time while we were recording this podcast, even though we had to schedule it “way past my usual bedtime.” I hope you enjoy listening, as well!

Book Update – Moving to zsh v3

I have pushed an update for the “Moving to zsh” book.

Just a few changes and fixes that have accumulated over the past two weeks. Much of this has been from feedback of readers. Thanks to everyone who sent in their notes.

The update is free if you have already purchased the book. You should get a notification from the Books application to update. (On macOS, I have seen that it can help to delete the local download of the book to force the update.)

If you are enjoying the book, please rate it on the Books store, or (even better) leave a review. These really help, thank you!

Also, please recommend the book to friends, co-workers, and anyone else (not just MacAdmins) who might be facing the zsh transition as they upgrade to Catalina.

The changes in v3 are listed here. This list is also in the ‘Version History’ section in the book. There, you will get links to the relevant section of the book, so you can find the changes quickly.

  • Added a section explaining how to work with upper- or lower-case strings in zsh scripts
  • Added a section explaining the differences in the read built-in command
  • Clarified the section on Connected Variables
  • Fixed file names in the table for Configuration Files and added a note for how to use configuration files with python environments
  • As usual, several typos and clarifications (Thanks to many readers)