This is the third partof a twelve-day series on my new year’s resolutions to become a better Mac admin. During these twelve days my book “Packaging for Apple Administrators” is on sale! (Previous Post: “Learn to Code (again)”)
Building and maintaining a testing environment is crucial for good system administration. Every bug, error and problem that you, your team and your testers catch during the testing and beta phase is one less problem that you may have to “urgently” fix right now for some user.
As admin you want to test every configuration, and every script or package that configures something.
A good testing environment helps you catch and identify problems early and efficiently.
I believe a good testing environment should have three tiers.
Tier 1: Virtual Machines
Virtual Machines are great. They can be quickly created, duplicated and destroyed. You can have VMs with different configurations available and only start those that you actually need.
There are currently three major VM solutions that allow you to virtualize macOS:
All of these provide snapshots, a feature that let’s you ‘freeze’ a system in a given state and quickly revert to it, discarding changes you made in between.
Your first level of testing should be against a ‘clean’ macOS system. I.e. a system which has a little configuration applied to it as you can get away with. Usually you will have to create a user or two, configure the network, enable remote access with ssh and/or screen sharing, apply updates and security patches and any tools or drivers the vm solution may require.
The next level of testing should be against a virtual machine that is connected to your management machine and setup in way that is as close to your production machines as possible. This way you can test all the interactions between software and configurations. This may include testing the imaging/system installation process over NetBoot et al on the VM.
You need a clean and a managed virtual machines for every major macOS/OS X version you support. (and possibly a few of the recent minor update versions as well)
Tier 2: Real Hardware
Virtual machines are great and flexible and will usually allow you to home in on most software realated issues quickly. However, sometimes you will get problems that come from interaction with hardware, that you cannot re-create in a VM. You should always verify everything on “real” hardware that is representative of what is being used in your deployment. With Apple’s habit of delivering new hardware with new ‘special’ versions of macOS, you need to have new hardware for testing as soon as it is going to be deployed (or before).
Also test everything on different network environments. Today that can be on wired network, (corporate) Wi-Fi, public (guest) Wifi, and from outside the network (home/hotel/coffee shop network). And once again you need to test this for every major macOS (and some minor) macOS version you support. This is harder to achive on real hardware, but large hard drives and multiple partitions are obviously a good place to start.
Tier 3: Beta Users or Machines
Depending on your deployment identify a group of users and/or a group of machines that can be used for early release/beta testing. For users, you want to look for users that can identify and report problems in a way that helps you with the fix. You do not, on the other hand wnat users that will fix the problem on thier own and never tell you. If you have a machine based deployment (labs, classroom, kiosks) you either want machines that are physically close to you, or managed supervised by a lab tech with good error reporting skills.
In both cases you want machines that are not critical for day to day business and can easily or quickly be replaced. (For example we did not install the early release on podium/lecture Macs, unless we had a spare available.) However, you do want to have early release/beta machines in all use scenarios.
You could have multiple groups of early release users, or alpha/beta groups. This really depends on how much testing you need or want to do.
Tier 4: Production
Not really a testing tier any more. You can stagger rollout into production, to give you another chance to catch any issues. Depending on the size of your deployment you may do this to lessen the load on your servers anyway. You should always have a rollback plan (and test the rollback plan!)
Servers
And now that you have a good testing workflow for your clients, you also need to build a plan a testing strategy for updating your management servers.