Signed bundles and the cost of rollback

Teams building fleet infrastructure design the update path with care and the reversal path in a hurry. The forward direction gets signatures, staged rollout, verification, and a review. The backward direction gets a runbook line that says restore the previous version, written by someone who assumed the previous version would still be there, still be valid, and still be able to run.

For machines that move, that asymmetry is a defect rather than an oversight. An update changes how a fleet behaves near people. So does undoing one. The two operations differ in which artifact ends up installed and in nothing else that matters, which means they deserve the same mechanism and the same evidence.

The forward path is the easier half, and it is worth stating briskly before spending the rest of the argument on the half that is actually hard.

What crosses the gap

An update to an air-gapped site is not a connection. It is an artifact: a signed bundle that carries its own provenance, moved across the boundary on physical media or a controlled transfer, verified inside the fence before anything is staged. The crossing is an operator's deliberate act on the operator's schedule, never a system's assumption that it may go and fetch what it needs.

The part that gets underspecified is what a bundle contains. A bundle is a complete, self-consistent set — the models, the policies, the configuration, and the constraints those artifacts were validated against — rather than a folder of files that happen to have travelled together. The distinction has teeth. A model validated against one policy set, installed beside a configuration validated against another, produces a combination nobody tested, on machines that express untested combinations as motion. Distributing files individually makes drift between them a matter of operational luck.

So the unit of distribution should be the unit of validation. The bundle exists to make the tested combination the only installable one, and to make the identity of that combination something the fleet can name, verify, and later be asked about.

Going back is also a physical change

Rollback usually gets filed under incident response, which quietly reclassifies it as an emergency action rather than a change. That reclassification is where the trouble starts. Reinstalling a previous model does not restore a safe default; it installs different behavior on machines with mass and reach. It is an update whose direction happens to be backward.

Treated as an escape hatch, the reversal path acquires the properties of an escape hatch: fewer checks, looser authorization, available to whoever is closest at the worst moment. That is a mechanism for installing unreviewed behavior on a fleet under time pressure, which is precisely when a fleet can least afford it. The rushed revert that fixes a visible problem and introduces an invisible one is a familiar story in software, and it has a heavier ending here.

The discipline that follows is straightforward to state and unpopular to implement. A reversal is verified the same way, approved the same way, staged the same way, and attested the same way as an advance. The previous bundle was approved once, for a fleet and a site in a state that no longer exists — so admitting it again is a decision, not a restoration.

The version you are not running

Rollback presupposes that its target still exists and can still be shown to be genuine. Neither is free. Both are obligations the design takes on for versions it is deliberately not running.

The artifact has to be present inside the boundary, because a site that cannot reach anything cannot go get it. Its signatures have to remain verifiable. The known-good measurements that let a node prove it is running the older image have to be retained alongside it, or the fleet will find itself in the position of running software it can install but cannot attest — which is a worse state than the one it was trying to escape. Keeping the artifact and discarding the evidence for it is a common and quiet mistake.

How far back to keep is a policy question with a physical answer. Retaining every version forever is not free inside a facility sized for its work. Retaining only the most recent one means a bad bundle installed twice leaves nowhere to go. Whatever depth is chosen defines a reversal window, and every version inside that window is a supported configuration — it may have to run real machines on a bad night, which means it cannot be a version the team has stopped thinking about.

Mixed versions are the normal state

A fleet does not update all at once, and should not. Updates go out in stages: a machine, a cell, a shift, then the rest. A mixed fleet is therefore not a transient anomaly to be tolerated for a few minutes. It is the state the system occupies during every rollout, and the state a partial reversal leaves behind indefinitely.

Correctness has to hold across the mix rather than at its endpoints. That constrains change more than most teams expect. Coordination protocols have to interoperate across adjacent versions, because machines on the old bundle and machines on the new one will share a floor, claim the same aisles, and read the same shared world model. Schema changes become additive by necessity: a field the old version ignores is survivable, while a field the old version misreads is a physical conflict rather than a parsing error.

Reversal tightens the constraint further. If the new version wrote shared state in a form the old version cannot read, going back is not a reversal at all — it is a migration, performed under pressure, in the wrong direction. Compatibility has to run both ways across the whole reversal window, which is a stricter obligation than forward compatibility alone and one that has to be enforced at design time, because it cannot be added on the night it is needed.

What does not come back

Software reverts. Some state does not. Overnight adaptation produces a model shaped by the day's data. An index gets built. A calibration is adjusted against the machine as it is now, not as it was. A migration rewrites what it touched. Reinstalling the previous bundle does not unlearn, un-index, or un-calibrate any of that.

Which means "roll back the software" and "restore the prior system" are different sentences, and a design that treats them as synonyms will be surprised at the worst time. The honest approach is to classify state explicitly: reversible state is versioned, retained, and restorable as part of the reversal; irreversible state can only move forward, and the procedure has to say so plainly. Old code paired with new state is a configuration nobody validated, which is the exact failure the bundle was constructed to prevent.

Naming the irreversible parts early tends to shrink the list. Once a team writes down that adaptation results cannot be undone, the appealing fix is to stop treating them as in-place mutations and start treating them as versioned artifacts admitted through the same mechanism as everything else. The classification exercise is not paperwork. It is a design pass that moves state out of the column where mistakes are permanent.

Reverting with nothing to call

Now put the decision where it actually happens. In an air-gapped facility, the choice to revert is made by people standing at the site, at night, with a line stopped, with no way to fetch a bundle, no vendor to bring into the room, and possibly no way to ask a question until the next scheduled crossing. Everything the reversal needs must already be inside the boundary before anyone knows it will be needed: the target artifact, its signatures, the measurements to verify it, and a procedure written for the people who will actually execute it rather than for the people who designed it.

That makes reversal capability a provisioning requirement instead of a support commitment. It is something to establish at installation and re-establish after every update, not something to discover the state of during an incident.

Reverting also has to leave a record. "We reverted" is exactly the kind of event an outside reviewer asks about later, and the questions are specific: what was running, what replaced it, who authorized it, when, on which machines, and what evidence exists that the fleet ended in the state the record claims. If the reversal path is the one path that skips the mechanism producing that record, the fleet's history develops a hole precisely where the interesting events live.

All of which resolves into a single design principle, and it is the one E31 Network is being built around here. The update path and the reversal path are one mechanism with a direction, not a feature with a fallback bolted beside it. A bundle is admitted to a site by the same verification, the same approval, and the same attestation whether the version it carries is newer or older than the one it displaces. The remaining honesty is being clear, in advance, about what no version number can restore.