AutoPkg recipe writing: things to look out for

AutoPkg is a cool project for Mac admins (in theory, Windows admins could use it, too, and there are even a few Windows recipes). Although it's a flexible framework that can be applied in many different ways, what it's most useful for is automating the tedious process of going to a website, downloading a new version of the software, and then importing that download into whatever you're using to push updates out to your Mac clients.

For a while, I was using existing recipes (there are many, so this is a totally valid approach), but eventually there was software I didn't see recipes for, so I started writing my own recipes. At first, I just started by copying existing templates and just modifying certain parts (the download URL, or the regular expressions to search for within the search URL).

Here are some things I noticed, in case you ever want to write your own recipes and run into these issues.

Arguments need to be separate

I ran into this issue where I was trying to purge the destination before unarchiving a .zip file, but it didn't seem to be working. Even though the archive_path and destination_path seemed to work fine without being in the Arguments dictionary, the purge_destination key wasn't registering until I put them all into the Arguments dictionary, as I should have from the start... so, remember to always put all arguments in an actual Arguments dictionary. Example:

<dict>
<key>Processor</key>
<string>Unarchiver</string>
<key>Arguments</key>
<dict>
<key>purge_destination</key>
<true/>
<key>archive_path</key>
<string>%RECIPE_CACHE_DIR%/downloads/%NAME%.zip</string>
<key>destination_path</key>
<string>%RECIPE_CACHE_DIR%/%NAME%/</string>
</dict>
</dict>

Code signature verification within disk images

When you're doing code signature verification on a disk image, you don't have to explicitly use the DmgMounter processor to mount the disk image. Instead, you can just treat the .dmg as a folder that includes the bundle to be verified. Here's an example (where %pathname% refers to the downloaded .dmg):

<dict>
<key>Processor</key>
<string>CodeSignatureVerifier</string>
<key>Arguments</key>
<dict>
<key>input_path</key>
<string>%pathname%/DiskMaker*.app</string>
<key>requirement</key>
<string>identifier "net.gete.diskmakerx" and anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = "2U4ZFMT67D"</string>
</dict>
</dict>

Dealing with regular expressions

If you're not a regex expert, some of the regular expression searches for the URLTextSearcher processor may look like gibberish to you.

A few tips to help with that, apart from (or maybe in addition to?) reading up on all the details of the Python regex documentation:

  • Before you put the regex into your recipe, you can test out your regex using Regex101 (select the Python one).
  • Generally speaking, the most useful thing I've found is creating a capture group with
    (?P<nameofcapturegroup>bunchofregex)
  • Just as you're about to put the regex into your recipe, make sure to substitute &lt; for < and &gt; for >

Script AutoPkg trust verification and trust update process

Starting with version 1, AutoPkg began evaluating trust info for recipes, so you could see what changes were made to a recipe (if changes were made) and then accept the changes if you wanted to. Here is what the typical trust verification workflow looks like.

Whether running a list of recipes via script or via AutoPkgr schedule, I'd occasionally get error'ed recipes when trust was broken, have to manually run

autopkg verify-trust-info -vv NAMEOFRECIPE
and then, after review, run
autopkg update-trust-info NAMEOFRECIPE
and then run the recipe after updating the trust info:
autopkg run -v NAMEOFRECIPE
So I thought I'd take a stab at scripting the whole process. Basically my script updates all the repos (to see if there are changes), verifies trust info on each of the recipes in the recipe list, and then prompts the user to review changes and approve them or not, before running all the approved or unchanged recipes.

It's still in the early testing phase, but it seems to work so far....

Automating an AutoPkg Munki import when vendors don’t package installers properly

You may have, when using (or creating) a .munki AutoPkg recipe, come across a situation in which you run it:

autopkg run -v NAMEOFITEM.munki
and then get something back like this:
Item NAMEOFITEM already exists in the munki repo as OLDNAMEOFITEM.
even though you're sure the item is newer than the one in the Munki repo.

That has to do with the find_matching_item_in_repo() function the MunkiImporter processor uses to determine whether the item exists already or not.

It compares a number of things between the to-be-imported item and what's already in the Munki repo—installer item hash, installs, receipts, files and paths, etc. If any of those matches up, MunkiImporter considers it a match.

So, for example, if you have BADLYPACKAGEDBYVENDOR 3.7.3, which is an update for BADLYPACKAGEDBYVENDOR 3.7.2, but the receipts for both are just 1 (yes, 1 and not 3.7.2 or 3.7.3), the MunkiImporter processor will see the two as the same and not do "another" import of the same item. Likewise, if the version in the app bundle is 3.7 and not 3.7.2 or 3.7.3, the MunkiImporter processor will see them as the same. I've even run into situations in which a vendor artificially ups the number but the "new" package or .app bundle is exactly the same. In that case, the installer hash will be the same, and the MunkiImporter processor will see them as the same.

So what do you, apart from complain to the vendor and pray it fixes the problem?

There may not be anything you can do apart from force an import. You may find a convoluted workaround, though. For LockDown Browser, I had to create an installs array based on the executable and also essentially override the useless receipts array. You might have to do something similar, depending on how bad the vendor package is.

When an AutoPkg recipe fails to import a .dmg

If you ever have an AutoPkg recipe that seems to be working fine for weeks or even months and then suddenly fails with a message like this one:

Error in local.munki.FileZilla: Processor: MunkiImporter: Error: creating pkginfo for
/Users/USERNAME/Library/AutoPkg/Cache/local.munki.FileZilla/FileZilla.dmg failed: Could not mount
/Users/USERNAME/Library/AutoPkg/Cache/local.munki.FileZilla/FileZilla.dmg!
(doesn't have to be FileZilla—could be anything), you may not see the .dmg is mounted in Disk Utility (or even diskutil list), but you can check to see if it's a phantom mount by seeing if it shows up in the output of
hdiutil info
If it does show up there, then run
hdiutil detach /dev/diskFILLINLOCATION
and then re-run the recipe. Should be fine after that.

Acknowledgements: Thanks to Eric Holtam for the tip—just documenting it here for anyone else who may benefit from it.

AutoPkg and Munki with UnRarX

AutoPkg and Munki are a great combination to keep client machines updated with the latest versions of software. UnRarX (at least as of this writing) has kind of a funky situation with AutoPkg, though. The AutoPkg download recipe is currently downloading the beta of 2.2 instead of the final release of 2.2.

What makes this complicated are a few things:

  • When you launch up UnRarX 2.2B, the app itself checks for an update and then prompts you to upgrade to UnRarX 2.2.
  • The binary for 2.2B and 2.2 is exactly the same (if you do an MD5 check on /Applications/UnRarX.app/Contents/MacOS/UnRarX for both version, they turn up the same hash).
  • If you do a
    makepkginfo -f /Applications/UnRarX.app/Contents/Info.plist
    an installs array with the CFBundleVersion will come up instead of an MD5 checksum on the file itself.
  • As far as Munki's concerned, 2.2B is a higher version number than 2., which makes sense.

It's a bit of a mess, because 2.2B appears to be higher than 2.2, but it's essentially the same binary as 2.2... and it prompts you to upgrade to 2.2 when you launch it.

So how do you get your 2.2B clients "upgraded" to 2.2? I had to do a few things.

First of all, I got the output of

makepkginfo -f /Applications/UnRarX.app/Contents/MacOS/UnRarX
just to get the basic format for the installs array.

Then I ran

md5 /Applications/UnRarX.app/Contents/Info.plist
on each version to get the unique checksum, and I modified the installs array to point to that file and to use that hash instead of the one for the executable.

Finally, I changed the version number in Munki of the 2.2B to 1.0 (or some other arbitrarily lower number than 2.2).

Once I did that, my clients says 2.2 as an upgrade to the existing 2.2B and then upgraded. I've also had to temporarily turn off the AutoPkg recipe for UnRarX, because the Sparkle feed is still showing 2.2B.

AutoPkg Adobe Flash Player recipes, licenses, and Munki

13 October, 2016 update: Adobe just broke this by making the distribution page require an Adobe ID login before you can view it.

Even though Flash is slowly being deprecated, people still use it. And while people still use it, Mac admins keep having to make sure their users' Adobe Flash Player plugins are up to date so the plugin is the smallest security hole that it can be.

Adobe Flash Player is free to download, but Adobe would like you to apply for a license to distribute Adobe Flash Player. Once you do that, they send you a unique link to download a distributable Flash Player (which is not materially different from the normal one you would download without the license).

One AutoPkg recipe maintainer wrote up a series of recipes to deal with this type of download. Here's one example.

For some reason, even though that recipe works to download Flash Player, I wasn't able to get it to work to download Flash Player and then have Munki import the approved download into the Munki repo.

So, as a workaround, I made a hodge-podge recipe combining the general Adobe Flash Player download AutoPkg recipe with the proper one, and then modifying it to include my institution's unique download link.

If you can get DLA to work with your Munki recipe, ignore the suggested code below (and comment to let me know how you did it). Otherwise, you may want to make an override for the standard Flash download recipe, and then paste in the following, and modify the link to be your institution's unique download link:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Description</key>
<string>Downloads Adobe Flash Player installer.</string>
<key>Identifier</key>
<string>com.github.autopkg.download.FlashPlayer</string>
<key>Input</key>
<dict>
<key>NAME</key>
<string>AdobeFlashPlayer</string>
</dict>
<key>MinimumVersion</key>
<string>0.6.0</string>
<key>Process</key>
<array>
<dict>
<key>Processor</key>
<string>URLTextSearcher</string>
<key>Arguments</key>
<dict>
<key>url</key>
<string>https://YOURORGANIZATIONSUNIQUEURL</string>
<key>re_pattern</key>
<string>Flash Player (?P&lt;version&gt;.*?) \(Win .*?Mac\)&lt;\/span&gt;</string>
</dict>
</dict>
<dict>
<key>Processor</key>
<string>URLTextSearcher</string>
<key>Arguments</key>
<dict>
<key>url</key>
<string>https://YOURORGANIZATIONSUNIQUEURL</string>
<key>re_pattern</key>
<string>(?P&lt;downloadurl&gt;https.*?osx_pkg.dmg)</string>
</dict>
</dict>
<dict>
<key>Processor</key>
<string>URLDownloader</string>
<key>Arguments</key>
<dict>
<key>url</key>
<string>%downloadurl%</string>
<key>filename</key>
<string>%NAME%.dmg</string>
</dict>
</dict>
<dict>
<key>Processor</key>
<string>EndOfCheckPhase</string>
</dict>
<dict>
<key>Processor</key>
<string>CodeSignatureVerifier</string>
<key>Arguments</key>
<dict>
<key>input_path</key>
<string>%pathname%/Install Adobe Flash Player.pkg</string>
<key>expected_authority_names</key>
<array>
<string>Developer ID Installer: Adobe Systems, Inc.</string>
<string>Developer ID Certification Authority</string>
<string>Apple Root CA</string>
</array>
</dict>
</dict>
</array>
</dict>
</plist>

I paired that with this Munki recipe override:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Identifier</key>
<string>com.github.autopkg.munki.FlashPlayer</string>
<key>Input</key>
<dict>
<key>MUNKI_REPO_SUBDIR</key>
<string>internet_plugins</string>
<key>NAME</key>
<string>AdobeFlashPlayer</string>
<key>pkginfo</key>
<dict>
<key>catalogs</key>
<array>
<string>testing</string>
</array>
<key>description</key>
<string>Adobe® Flash® Player is a cross-platform browser-based application runtime that delivers uncompromised viewing of expressive applications, content, and videos across screens and browsers.</string>
<key>display_name</key>
<string>Adobe Flash Player</string>
<key>name</key>
<string>%NAME%</string>
<key>unattended_install</key>
<true/>
</dict>
</dict>
<key>ParentRecipe</key>
<string>com.github.autopkg.download.FlashPlayer</string>
<key>Process</key>
<array>
<dict>
<key>Processor</key>
<string>MunkiPkginfoMerger</string>
<key>Arguments</key>
<dict>
<key>additional_pkginfo</key>
<dict>
<key>version</key>
<string>%version%</string>
</dict>
</dict>
</dict>
<dict>
<key>Arguments</key>
<dict>
<key>pkg_path</key>
<string>%RECIPE_CACHE_DIR%/downloads/%NAME%.dmg</string>
<key>repo_subdirectory</key>
<string>%MUNKI_REPO_SUBDIR%</string>
</dict>
<key>Processor</key>
<string>MunkiImporter</string>
</dict>
</array>
</dict>
</plist>

Disable auto-updates for Firefox when deploying using Munki and AutoPkg

Note before you begin
Acknowledgements
Why you should do this
The actual procedure
Caveat
Thunderbird
The CCK2 method

Note before you begin

If you know nothing about Munki or AutoPkg, then this page will be of little use to you. If you're interested in a basic setup tutorial, check out the Absolute beginner’s guide to setting up Munki (not monkey).

Acknowledgements

Special thanks to Nick McSpadden for Deploying and Managing Firefox Part 2: Working with Munki, which lays out the disable-update concept with a slightly different procedure (using CCK and symlinks), and to Mozilla for documenting directory changes in Deploying Firefox in an enterprise environment.

Why you should do this

For a while, I was deploying Firefox using Munki but then not disabling updates, because it actually wasn't causing any problems.

firefoxupdateerror
Recently, though, users have been getting this kind of error, which is odd, because Firefox should, in theory, still be able to update itself, even if Munki has been pushing updates, because all Munki does is essentially automate the regular installation process (dragging the Firefox.app bundle to the /Applications folder).

The actual procedure

Create a text file with these contents:

pref("general.config.filename", "mozilla.cfg");
pref("general.config.obscure_value", 0);
Save that text file as autoconfig.js

Create a second text file with these contents:

// Disable updater
lockPref("app.update.enabled", false);
// make absolutely sure it is really off
lockPref("app.update.auto", false);
lockPref("app.update.mode", 0);
lockPref("app.update.service.enabled", false);
Save that text file as mozilla.cfg.

Create a DisableFirefoxUpdates package (if you prefer a GUI, I'd recommend Packages) that distributes a payload of two files. I decided to put the files in /Library/Application Support/Mozilla/DisableAutoUpdates, but you can put them somewhere else, as long as you can find them later (if you do decide on a different location, make sure to tweak the scripts below accordingly).

Import the DisableFirefoxUpdates package into Munki (using munkiimport) and then make it a requirement for the Firefox package.

You may also want to add this as a post_install script for DisableFirefoxUpdates:

#!/bin/bash

# Distribute the script if Firefox is already installed
if [ -d "/Applications/Firefox.app/Contents/Resources" ]; then

# Copy the autoconfig.js
cp /Library/Application\ Support/Mozilla/DisableAutoUpdates/autoconfig.js /Applications/Firefox.app/Contents/Resources/defaults/pref

# Copy the mozilla.cfg file
cp /Library/Application\ Support/Mozilla/DisableAutoUpdates/mozilla.cfg /Applications/Firefox.app/Contents/Resources/

fi
That's handy if you, like me, are starting to distribute the DisableFirefoxUpdates package after having already deployed Firefox.

There is one more final step, which is super important. If you're using AutoPkg (or its graphical frontend, AutoPkgr) to automatically get the latest Firefox and import it into Munki, you want to create an override recipe (if you have one already, tweak the existing override), and then include the following as part of the pkginfo dict:

<key>postinstall_script</key>
<string>#!/bin/bash

# Copy the autoconfig.js
cp /Library/Application\ Support/Mozilla/DisableAutoUpdates/autoconfig.js /Applications/Firefox.app/Contents/Resources/defaults/pref

# Copy the mozilla.cfg file
cp /Library/Application\ Support/Mozilla/DisableAutoUpdates/mozilla.cfg /Applications/Firefox.app/Contents/Resources/
</string>
What this does (minus the check to see if Firefox is installed) is just make sure that every time you install a new version of Firefox, you copy the DisableFirefoxUpdate files over to the newly-installed version.

Another way to approach it would be to have the Munki item disabling auto-updates also have an installs array in its pkginfo, so that if the new Firefox overwrites it, Munki will reinstall (here's an example):

<key>installs</key>
<array>
<dict>
<key>md5checksum</key>
<string>326b871936cca29e198fdcaf50371e90</string>
<key>path</key>
<string>/Applications/Firefox.app/Contents/Resources/mozilla.cfg</string>
<key>type</key>
<string>file</string>
</dict>
<dict>
<key>md5checksum</key>
<string>0377ceecfff0b937c590f3ae1770e8f6</string>
<key>path</key>
<string>/Applications/Firefox.app/Contents/Resources/defaults/pref/autoconfig.js</string>
<key>type</key>
<string>file</string>
</dict>
</array>
That installs array is a combination of the results of these commands:
/usr/local/munki/makepkginfo -f /Applications/Firefox.app/Contents/Resources/defaults/pref/autoconfig.js
/usr/local/munki/makepkginfo -f /Applications/Firefox.app/Contents/Resources/mozilla.cfg

Caveat

I've done some testing, and this works. The only thing to keep in mind is that if you deploy the DisableFirefoxUpdates after having already deployed Firefox to users, users will still likely have auto-updates enabled... until they restart Firefox.

So if a user says "I'm getting an error about Firefox not updating," have the user quit out of Firefox and launch up Managed Software Center, and it should be good from that point forward.

Thunderbird

Someone asked about this, so I did testing, and this exact same config works for Thunderbird as well. You would just have to add in the appropriate script to copy the files over (from the same payload):

#!/bin/bash

# Distribute the script if Thunderbird is already installed
if [ -d "/Applications/Thunderbird.app/Contents/Resources" ]; then

# Copy the autoconfig.js
cp /Library/Application\ Support/Mozilla/DisableAutoUpdates/autoconfig.js /Applications/Thunderbird.app/Contents/Resources/defaults/pref

# Copy the mozilla.cfg file
cp /Library/Application\ Support/Mozilla/DisableAutoUpdates/mozilla.cfg /Applications/Thunderbird.app/Contents/Resources/

fi

The CCK2 method

If you would like to use CCK for some more major customizations instead of just disabling auto-update, here is a recent blog post on how to do so:
Firefox, CCK2 and an AutoPKG Recipe.

Here are a few fun screenshots of my own to complement the ones on that page:

And this is the Munki recipe you'd use with the resulting autoconfig.zip file.

Getting Started with Munki (not monkey)

5 April, 2018 update: This page used to have a tutorial that walked you through setting up a bunch of GUI tools to manage and keep up-to-date a Munki server. I've reconsidered, and I don't think that's the best approach for new Munki administrators to take to learning Munki.

First Steps

Here are a series of links you should read and follow, in this order, to explore Munki on your own:

  1. Demonstration Setup: Walks you through a very basic setup, using a Mac as a Munki server and another Mac (or even the server itself) as a test Munki client. Even though you can do so, I'd highly recommend against setting up a Mac running Server.app to be your Munki server. If you use a Mac, just use regular macOS with the built-in Apache.
  2. Overview: If you're absolutely new to Munki, you should really understand the basic mechanics of it and what catalogs and manifests are. If you read the overview and are still confused, ask for clarification from other Munki administrators (see links in Getting Help).
  3. How Munki Decides What Needs To Be Installed: This is one of the most frequently asked questions from new Munki administrators, so it's very important you understand why you may have a Munki item that's in an endless install loop (and how to fix it).
  4. An opinionated guide to Munki manifests : Some opinions on how you should structure your manifests in Munki.
  5. Another opinionated guide to Munki manifests: Some more (related) opinions on how you should structure your manifests in Munki.

Getting Help

And here are some great places to go for help, if you have Munki-related questions:

  1. MacAdmins Slack #munki channel
  2. Munki-Discuss Google Group
  3. MacEnterprise mailing list

Server Setup

Want to secure your Munki repo and/or move it to Linux? You may find these links handy:

  1. Using https / self-signed certificates and basic authentication with Munki: If you want to stay with macOS and have basic authentication on an internal-only server.
  2. Certbot Apache on macOS: If you're going for basic authentication on a public-facing server, and you want a proper (not self-signed) SSL certificate.
  3. Setup a Munki repo on Ubuntu 14.04 - Part 1: Yes, I know it says 14.04, but the basic instructions still work for 16.04, and they'll probably work for 18.04, too.
  4. Certbot Nginx on Ubuntu 16.04 (xenial): Proper SSL certificate for Ubuntu (again, if your server is public-facing).
  5. Using Munki With SSL Client Certificates: Basic authentication not enough security for you? Set up revokable client certificates instead.

Munki-related Helper Tools

Need other helper tools?

  1. AutoPkg: Allows you to automate downloading and installing new software into your Munki repo. Get to know the command-line tool well first if you choose to also install the (no longer maintained) GUI management for it called AutoPkgr.
  2. MunkiAdmin: A great graphical frontend for managing your Munki repo (after you've already understood how the pieces work together... and, frankly, even with MunkiAdmin, I'd still recommend using munkiimport on the command-line to manually import new items that don't come through AutoPkg). As an alternative, you may want to check out mwa2, which is web-based.
  3. MunkiReport-PHP or sal: If you want to add in a reporting piece to see what your Munki clients are up to.