Critical Update Hero

Last week Joe Chilcote discovered an interesting message when imaging a Late 2016 MacBook Pro TouchBar (from here on out referred to as TBP):

A critical software update is required for your Mac. To install this update you need to connect to a network. Select a Wi-Fi network below, or click Other Network Options to connect to the internet using other network devices.

Given that I did not see this issue when testing DEP, I set forth to to attempt to duplicate the issue and find out what triggered this event.

(If you don’t want to read this whole post or want to dupe the radars I submitted, scroll down to As a macadmin, does this impact me? for a TL;DR version.)

Critical Software Update?

I think it’s safe to say the macadmin community has been hearing rumblings about the future of macOS administration. Whether it was Michael Lynn’s excellent blog post, m(DM)acOS, APFS or even Sal Saghoian’s position being axed, many macadmins (myself included) are worried about the future of macOS administration being a MDM only world.

What if the new TBP Macs were the first piece to this future?

I was initially worried about this discovery due to it breaking a very typical thin imaging workflow:

  1. asr an AutoDMG thin image
  2. Install/Touch /private/var/db/.AppleSetupDone
  3. Run bootstrapping software

Upon reboot, SetupAssistant was not skipped and you were instead greeted with this lovely screen:

Critical Update 1

Attempting to skip this page would lead to an additional failure:

A critical software update is required for your Mac, but an error was encountered while installing this update.

Even more worrying was the final note: Your Mac can't be used until this update is installed. Shutdown / Try Again Critical Update 2

If you did connect your TBP to an online source, the critical update was downloaded, installed and your system was rebooted. Critical Update 3

This entire process takes about two minutes to finish and then you can login into the Mac. Obviously a lot of engineering effort went into this - this is not a fluke.

What triggers this? Is Mac imaging finally dead?

Trying to recreate the issue

Good news everyone: Mac imaging isn’t dead … yet. While trying to recreate this issue, I started off with most simple workflow and methodically tried adding features.

Uneventful Workflow tests

  • Workflow 1
    • Deploy image to current volume - no wipe
  • Workflow 2
    • Same as workflow 1
    • Add fingerprint to Touch ID
  • Workflow 3
    • Same as workflow 2
    • Convert booted volume to Core Storage
  • Workflow 4
    • Same as workflow 2
    • Enable FileVault 2
  • Workflow 5
    • In Imagr NetInstall environment
    • Open up Disk Utility
    • Delete FileVault 2 volume
    • Re-run Workflow 4

In none of these tests did I receive the critical software update. For informational purposes, workflow 5 was tested due to Imagr’s inability to wipe FileVault encrypted volumes and was the closest thing to Joe’s testing. After some discussions with Joe, we had a theory as to the true culprit of the issue.

Workflow Goldmine

  • Workflow 6
    • In Image NetInstall environment
      • Open up Disk Utility
        • Delete entire disk
    • Deploy image to newly created volume

It was with this workflow that I was finally able to recreate the issue. SetupAssistant immediately prompted the critical software update.

So what is being deleted when wiping the entire disk?

Apple’s EFI container for TouchBar

For some time, Apple has been installing EFI/firmware updates through standalone packages. Allister Banks first wrote about this during the Thunderstrike vulnerability and I have been complaining about this for some time.

Unfortunately, it looks like this how now been taken to a new level:

As a guess, I decided to look at the EFI volume

diskutil list
/dev/disk0 (internal):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme 500.3 GB disk0
1: EFI EFI 314.6 MB disk0s1
2: Apple_HFS Macintosh HD 499.3 GB disk0s2
3: Apple_Boot Recovery HD 650.0 MB disk0s3

diskutil mount disk0s1

ls /Volumes/EFI/EFI/APPLE/EMBEDDEDOS/
FDRData
combined.memboot
version.plist

Embedded OS you say. And something called FDRData

After searching for Embedded OS logs (thanks to Joe Chilcote), we found some interesting tidbits:

log show --debug --predicate 'process =="EmbeddedOSInstallService"'

EmbeddedOSInstallService: Couldnt find memboot image in ESP:
file:///Volumes/EFI/EFI/APPLE/EMBEDDEDOS/combined.memboot

So clearly, Apple is looking for this “Embedded OS” and if it can’t find it, it attempts to rebuild the boot process.

EmbeddedOSInstallService

As a test, I decided to delete the contents of /Volumes/EFI/EFI, reboot and look at the logs. It appears that the following happens:

  • During macOS system start, the TouchBar attempts to boot running it’s own derivative of iOS.
  • If it cannot find the embedded OS in the EFI volume, it triggers a repair.
  • Taking iOS files from /usr/standalone/firmware/iBridge1_1Customer.bundle/Contents/Resources and /Library/Updates/PreflightContainers:
  • If a valid preflight exists, no internet connection is required.
  • macOS will show an extended/long boot process to the user, typically taking 2-3 minutes before the desktop is available.
  • If a valid preflight does not exist, an internet connection is required
  • SetupAssistant is triggered, informing the user of a Critical Update needed
  • Once the TouchBar has either repaired itself or booted properly, biometrics/Touch ID is now available to the user.

Valid Preflight:

EmbeddedOSInstallService: network reachability check
EmbeddedOSInstallService: ---- Starting network reachability check ----
EmbeddedOSInstallService: We have a valid preflighted container, no network is required
EmbeddedOSInstallService: (EmbeddedOSInstall) prepare device

Invalid Preflight:

EmbeddedOSInstallService: No matching preflight container found
EmbeddedOSInstallService: network reachability check
EmbeddedOSInstallService: ---- Starting network reachability check ----
EmbeddedOSInstallService: Checking for reachability to gs.apple.com
EmbeddedOSInstallService: personalization
EmbeddedOSInstallService: ---- Starting personalization ----

No Internet

EmbeddedOSInstallService: Can't continue the restore because you are not connected to the Internet.

Here is an abridged version of the logging events:

EmbeddedOSInstallService: (EmbeddedOSInstall) FDR preflight
EmbeddedOSInstallService: ---- Starting FDR preflight ----
EmbeddedOSInstallService: Waiting for device to be connected
EmbeddedOSInstallService: Device connected: EOSDevice , boardID: 0x12, chipID: 0x8002, secure: YES, prod fused: YES>
EmbeddedOSInstallService: Starting FDR preflight
EmbeddedOSInstallService: Wrote preflighted FDR to memory store URL: /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/6140FB57-C5C9-4E4D-8534-1BA6F879A77A-EmbeddedOSInstall-FDRMemoryStore
EmbeddedOSInstallService: (EmbeddedOSInstall) preflight container stash
EmbeddedOSInstallService: ---- Starting preflight container stash ----
EmbeddedOSInstallService: Saved preflight container to disk: EOSPreflightContainer <file:///Library/Updates/PreflightContainers/FE0EC8F3-E032-49C1-B6E2-EA42171165AB.preflightContainer> (preflighted on 2016-11-24 07:00:02 +0000)
EmbeddedOSInstallService: Preflight time: 3.0 seconds
EmbeddedOSInstallService: Preflight was successful!
EmbeddedOSInstallService: Diagnostic summary: Preflight (14Y363 -> 14Y363 (Customer Boot), preflighted = 0, prod fused = 1, user auth = 0, retries = 0, after boot failure = 0, failing phase = 0, uuid = A21D9039-E0B8-42DA-A3D6-A037EE04484B): success
EmbeddedOSInstallService: ---- End Embedded OS Preflight ----
EmbeddedOSInstallService: [com.apple.mac.install.EmbeddedOSInstall] Adding client: loginwindow (pid = 91, uid = 0, path = /System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow)
EmbeddedOSInstallService: Checking if we should heal the device
EmbeddedOSInstallService: No data found in ios-boot-in-progress NVRAM key
EmbeddedOSInstallService: Device isn't booted yet and boot isn't in progress (EFI failed to bootstrap?)
EmbeddedOSInstallService: ---- Begin Embedded OS Boot ----
EmbeddedOSInstallService: (EmbeddedOSInstall) force reset
EmbeddedOSInstallService: ---- Starting force reset ----
EmbeddedOSInstallService: Resetting device
EmbeddedOSInstallService: (EmbeddedOSSupportHost) connection with driver establish (connect: 4907, service: 4807)
EmbeddedOSInstallService: Waiting for device to be connected
EmbeddedOSInstallService: Entering recovery mode, starting command prompt
EmbeddedOSInstallService: recovery mode device matches (using device type)
EmbeddedOSInstallService: ---- Starting memboot from EFI system partition ----
EmbeddedOSInstallService: Waiting for device to be connected
EmbeddedOSInstallService: recovery mode device matches (using locationID)
EmbeddedOSInstallService: (EmbeddedOSSupportHost) registered for 'com.apple.EmbeddedOS.DeviceConnected' darwin notification
EmbeddedOSInstallService: (EmbeddedOSSupportHost) registered for 'com.apple.EmbeddedOS.DeviceUnresponsive' darwin notification
EmbeddedOSInstallService: Waiting for Storage Kit to populate disks
EmbeddedOSInstallService: Done waiting for Storage Kit to populate disks
EmbeddedOSInstallService: Mounting ESP
EmbeddedOSInstallService: Couldn't find memboot image in ESP: file:///Volumes/EFI/EFI/APPLE/EMBEDDEDOS/combined.memboot
EmbeddedOSInstallService: Unmounting ESP
EmbeddedOSInstallService: Boot failed with error: Code=1201 "No memboot image was found in the EFI system partition."
EmbeddedOSInstallService: ---- End Embedded OS Boot ----
EmbeddedOSInstallService: Resetting the device into recovery mode since an error occurred during boot
EmbeddedOSInstallService: Waiting for recovery mode device
EmbeddedOSInstallService: Done waiting for recovery mode device
EmbeddedOSInstallService: Waiting for device boot
EmbeddedOSInstallService: Checking for healing overrides
EmbeddedOSInstallService: SMC is in app mode
EmbeddedOSInstallService: Should heal: YES, Found recovery mode device (after reboot attempt) (took 4.915 seconds)
EmbeddedOSInstallService: Enqueuing restore
EmbeddedOSInstallService: Starting restore
EmbeddedOSInstallService: Disabling retrying with AC for loginwindow
EmbeddedOSInstallService: Setting bootFailedAfterShouldHeal
EmbeddedOSInstallService: (EmbeddedOSInstall) Embedded OS Restore
EmbeddedOSInstallService: Getting information about current device for diagnostics
EmbeddedOSInstallService: Choosing between bundle specifiers: ("EOSRestoreBundle (14Y363)")
EmbeddedOSInstallService: Chose EOSRestoreBundle (14Y363)
EmbeddedOSInstallService: Using restore bundle: EOSRestoreBundle (14Y363)
EmbeddedOSInstallService: Attempting to locate preflight container for restore bundle
EmbeddedOSInstallService: Preflight container matches bundle specifier: EOSPreflightContainer <file:///Library/Updates/PreflightContainers/FE0EC8F3-E032-49C1-B6E2-EA42171165AB.preflightContainer/> (preflighted on 2016-11-24 07:00:02 +0000)
EmbeddedOSInstallService: Found preflight container: EOSPreflightContainer <file:///Library/Updates/PreflightContainers/FE0EC8F3-E032-49C1-B6E2-EA42171165AB.preflightContainer/> (preflighted on 2016-11-24 07:00:02 +0000)
EmbeddedOSInstallService: Set restore bundle: EOSRestoreBundle (14Y363) (PKBundleComponentVersion )
EmbeddedOSInstallService: Set FDR memory store: /Library/Updates/PreflightContainers/FE0EC8F3-E032-49C1-B6E2-EA42171165AB.preflightContainer/FDRData.plist
EmbeddedOSInstallService: Loading restoreOptions from plist: /Library/Updates/PreflightContainers/FE0EC8F3-E032-49C1-B6E2-EA42171165AB.preflightContainer/personalized/EOS_RestoreOptions.plist
EmbeddedOSInstallService: Updating restore options with preflight container paths
EmbeddedOSInstallService: (EmbeddedOSInstall) network reachability check
EmbeddedOSInstallService: ---- Starting network reachability check ----
EmbeddedOSInstallService: We have a valid preflighted container, no network is required
EmbeddedOSInstallService: (EmbeddedOSInstall) prepare device
EmbeddedOSInstallService: ---- Starting prepare device ----
EmbeddedOSInstallService: Waiting for Storage Kit to populate disks
EmbeddedOSInstallService: Done waiting for Storage Kit to populate disks
EmbeddedOSInstallService: Restore bundle was preflighted, forcing reset into recovery for restore
EmbeddedOSInstallService: Entering recovery mode, starting command prompt
EmbeddedOSInstallService: Device is now in recovery mode
EmbeddedOSInstallService: (EmbeddedOSInstall) personalization
EmbeddedOSInstallService: ---- Starting personalization ----
EmbeddedOSInstallService: We already have a valid preflighted container, skipping
EmbeddedOSInstallService: (EmbeddedOSInstall) bootstrap recovery mode
EmbeddedOSInstallService: ---- Starting bootstrap recovery mode ----
EmbeddedOSInstallService: Starting recovery mode restore
EmbeddedOSInstallService: Starting recovery restore
EmbeddedOSInstallService: Recovery mode restore succeeded
EmbeddedOSInstallService: Waiting for device to be disconnected
EmbeddedOSInstallService: ---- Starting restore mode restore (using restored) ----
EmbeddedOSInstallService: Waiting for device to be connected
EmbeddedOSInstallService: Starting restore mode restore
EmbeddedOSInstallService: Restore mode restore success!
EmbeddedOSInstallService: Sleeping 10 seconds to wait for nvram flush
EmbeddedOSInstallService: (EmbeddedOSInstall) force reset
EmbeddedOSInstallService: ---- Starting force reset ----
EmbeddedOSInstallService: Resetting device
EmbeddedOSInstallService: Waiting for device to be connected
EmbeddedOSInstallService: (EmbeddedOSInstall) recovery mode restore (OS ramdisk)
EmbeddedOSInstallService: ---- Starting recovery mode restore (OS ramdisk) ----
EmbeddedOSInstallService: Waiting for device to be connected
EmbeddedOSInstallService: Using OSRamdisk tag for second boot
EmbeddedOSInstallService: Starting recovery restore
EmbeddedOSInstallService: Recovery mode restore succeeded
EmbeddedOSInstallService: Waiting for device to be disconnected
EmbeddedOSInstallService: (EmbeddedOSInstall) wait for boot
EmbeddedOSInstallService: ---- Starting wait for boot ----
EmbeddedOSInstallService: Waiting for device to be connected
EmbeddedOSInstallService: Device boot complete!
EmbeddedOSInstallService: (EmbeddedOSInstall) EFI system partition installation
EmbeddedOSInstallService: ---- Starting EFI system partition installation ----
EmbeddedOSInstallService: Memboot image checksum (/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/3E465740-A3BD-4EE6-98D6-FE4DB003C10B-EmbeddedOSInstallESPSandbox/combined.memboot): 1904138235
EmbeddedOSInstallService: Mounting EFI system partition
EmbeddedOSInstallService: Error getting size of directory: /Volumes/EFI/EFI/APPLE/EMBEDDEDOS, returning 0: Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory" UserInfo={NSFilePath=/Volumes/EFI/EFI/APPLE/EMBEDDEDOS}
EmbeddedOSInstallService: Creating intermediate directory: /Volumes/EFI/EFI/APPLE
EmbeddedOSInstallService: Shoving /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/3E465740-A3BD-4EE6-98D6-FE4DB003C10B-EmbeddedOSInstallESPSandbox to /Volumes/EFI/EFI/APPLE/EMBEDDEDOS
EmbeddedOSInstallService: Memboot image checksum (/Volumes/EFI/EFI/APPLE/EMBEDDEDOS/combined.memboot): 1904138235
EmbeddedOSInstallService: Cleaning up sandbox: /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/3E465740-A3BD-4EE6-98D6-FE4DB003C10B-EmbeddedOSInstallESPSandbox
EmbeddedOSInstallService: Unmounting EFI system partition
EmbeddedOSInstallService: (EmbeddedOSInstall) purge preflight containers
EmbeddedOSInstallService: ---- Starting purge preflight containers ----
EmbeddedOSInstallService: Purging all preflight containers
EmbeddedOSInstallService: (EmbeddedOSInstall) [com.apple.mac.install.EmbeddedOSSerial]
EmbeddedOSInstallService: Restore time: 104.8 seconds
EmbeddedOSInstallService: Restore was successful!
EmbeddedOSInstallService: Diagnostic summary: Restore ((null) -> 14Y363 (Customer Boot), preflighted = 1, prod fused = 1, user auth = 0, retries = 0, after boot failure = 1, failing phase = 0, uuid = 2359AA1C-98E9-4B21-96A5-D710317AD57E): success
EmbeddedOSInstallService: ---- End Embedded OS Restore ----
EmbeddedOSInstallService: Adding client: biometrickitd (pid = 232, uid = 0, path = /usr/libexec/biometrickitd)
EmbeddedOSInstallService: Enqueing get local FDR data
EmbeddedOSInstallService: Starting get local FDR data
EmbeddedOSInstallService: Waiting for Storage Kit to populate disks
EmbeddedOSInstallService: Done waiting for Storage Kit to populate disks
EmbeddedOSInstallService: (CoreFoundation) Loading Preferences From System CFPrefsD For Search List

Local Preflight Container

The local preflight container has some interesting information, although I won’t dive into this too much on this post.

ls /Library/Updates/PreflightContainers/guid.preflightContainer
FDRData.plist
metadata.plist
personalized

ls /Library/Updates/PreflightContainers/guid.preflightContainer/personalized/
BuildManifest.plist
EOS_RestoreOptions.plist
Firmware
KernelCache_kernelcache.release.img4
OSRamdisk_058-40573-247.img4
Restore.plist
RestoreKernelCache_kernelcache.release.img4
RestoreRamDisk_058-27707-457.img4
amai
version.plist

ls /Library/Updates/PreflightContainers/guid.preflightContainer/personalized/amai
apimg4ticket.der
debug
receipt.plist

ls /Library/Updates/PreflightContainers/guid.preflightContainer/personalized/amai/debug
tss-request.plist
tss-response.plist

ls /Library/Updates/PreflightContainers/guid.preflightContainer/personalized/Firmware
all_flash
dfu

ls /Library/Updates/PreflightContainers/guid.preflightContainer/personalized/Firmware/all_flash
all_flash.x619ap.production

ls /Library/Updates/PreflightContainers/guid.preflightContainer/personalized/Firmware/all_flash/all_flash.x619ap.production/
DeviceTree_DeviceTree.x619ap.img4
LLB_LLB.x619.RELEASE.img4
RestoreDeviceTree_DeviceTree.x619ap.img4
RestoreSEP_sep-firmware.x619.RELEASE.img4
SEP_sep-firmware.x619.RELEASE.img4
iBoot_iBoot.x619.RELEASE.img4
manifest

ls /Library/Updates/PreflightContainers/guid.preflightContainer/personalized/Firmware/dfu
iBEC_iBEC.x619.RELEASE.img4
iBSS_iBSS.x619.RELEASE.img4

A few interesting pieces:

  • There appears to be a tss-request, tss-response and receipt, which seem to be signing data from Apple. One could assume this is for production activation of the OS.
  • There are img4 files, all related to booting, restoring and DFU modes. These are more than likely signed files extracted from /usr/standalone/firmware/iBridge1_1Customer.bundle
  • From the Receipt.plist, the TouchBar appears to be denoted as Watch2,5 running watchOS 3.0-14Y363.
  • Many of these plist’s contain keys, more than likely pointing to certificate based authentication/identities.
  • There appears to be logic to detect whether the TBP has a fuse set. We might also be able to send specific variants of the OS. No build variant specified and device is prod-fused, choosing customer variant.

Fun bugs

Given that iOS has had difficulty in the past with time, I hopped in my DeLorean and went back to 01-01-1970. Sure enough, the TouchBar failed to load, Touch ID was broken and the machine took a significant amount of time to boot.

While there were no logging events with EmbeddedOSInstall, the user experience was terrible. This is what the user sees for 2+ minutes.

1970 TouchBar

One can imagine that the OS is no longer validated, but why doesn’t Apple attempt to detect a network, and then run ntpdate if the time if incorrect? While I have not submitted a radar for this, I plan very soon. Why Apple continues to not test time sync situations is beyond me.

The wonderful future of hybrid hardware

It’s quite clear - Welcome to the future of Apple’s hybrid ARM/x86 platform. It’s also quite clear that destroying entire disks is going to lead to some pain points for people still imaging. I have some concerns though. For several years, Apple has moved firmware/EFI updates into the delta/combo updaters and imaging has not been able to solve this issue.

  • What happens when we build a 10.12.2 image and the EFI is out of date?
  • Does EmbeddedOSInstallService also check signing windows?
  • If it does, what happens when we deploy an image that doesn’t contain the new firmware?
  • Will there come a time that much like iOS, a MacBook Pro cannot be restored to an older OS?
  • Will Apple consider wiping the primary OS volume only a security vulnerability, and cause this message to occur during any re-image of the volume?

I think we will have our answers soon, but it will be up to the community to figure this out.

As a macadmin, does this impact me?

  • Are deploying a thin, modular or thick image?
  • Are you doing some kind of first boot scripting / boot process, ie LoginLog or DeployStudio finalize scripts?
  • Are you bootstrapping munki?

If so, the key to not encountering this issue is by targeting only the current Macintosh HD volume of the machine.

Imagr users

Imagr targets the first available volume by default. For FileVault enabled disks, you typically deleted the entire drive or just the encrypted volume via Disk Utility.

If you are deleting the entire drive, stop! Delete the current volume and then run your imagr workflow. This will allow your automation workflows/bootstraps to continue to work.

DeployStudio users

DeployStudio should be okay, but I recommend changing your restore workflow to:

Target Volume: Enter Value - Macintosh HD

While this option may not work for everyone (and may fail if people rename the default Macintosh HD naming convention), it will ensure that the volume itself is targeted. First Disk available is more than likely an option you want to stay away from, but I have not tested this theory. Please report your findings and if you run into any issues, post on the DeployStudio forums. Or better yet, move to Imagr! :)

Internet Recovery users

Unfortunately, it seems if you are using Internet Recovery and wipe the entire disk, the critical update component will still be needed to complete, even though Internet Recovery has an internet component.

This is problematic if you wipe your devices prior to re-allocation and using DEP. This may also need a radar.

Radars submitted to Apple

Apple are you listening?

Apple if you are reading this, can you please outline this process for us? A simple knowledge base will not be enough. We need a up-to-date portal with information regarding the future of mac management. Documentation should not be put on the backs of the companies that use your products.

We need answers and we need answers soon. Leaving us in the dark about future processes is bad for everyone. It’s bad for the community, bad for the companies using your products and eventually it will become bad for those millions of iOS/macOS developers you cherish.

Bring back the portion of WWDC for macadmins - invite us! We would love to talk! :)

Thanks to Michael Lynn and Pepijn Bruienne for working with me late Wednesday night. As we continue to dissect this, I hope to see more in-depth discoveries.

Table Of Contents