Degraded modes and the order things fail in

The configuration in which everything works is the easiest state to build and the rarest state to be in. Every component is present, every link is up, every sensor is clean, every machine is on the current model. It is the state that gets demonstrated, and a working fleet spends almost none of its operating life there.

The rest of the time something is broken. Usually something small, often something intermittent, occasionally several things at once. A fleet system's real behavior is the behavior of those states, which means the design work that decides whether the system is usable is the design work nobody schedules.

Most systems have degraded modes they never designed. The modes exist anyway — assembled out of timeouts, retry loops, watchdogs, and the default branch of a switch statement — and they are what the machine does on a bad day.

The ways a fleet is partially broken

The list is unremarkable, which is the point. A machine loses contact with the facility plane while carrying a load. The facility plane loses contact with a machine that is still moving. The site loses the fleet plane, so no new signed artifacts arrive and no evidence flows out. A camera fails outright, or is occluded by a pallet, or is merely dirty, which is worse because it degrades quality without announcing itself. A region of the shared world model goes stale because nothing has observed that aisle in a while. The plane is saturated, and every machine's query comes back a little late. Time synchronization drifts or is lost. A model fails to load after an update, or loads and begins producing output that is implausible rather than absent.

None of these are exotic. Each of them is a Tuesday. What matters is that each has a correct response and that the correct responses differ. The right behavior when the facility plane is unreachable is not the right behavior when a camera is dirty, and a system that handles both with the same fallback is handling at least one of them wrong.

The first two are worth separating, because they look like one failure and are two. A machine that loses the plane knows its own state and knows nothing about the fleet. A plane that loses a machine knows the fleet and now has to assume the worst about one member of it. A single link fault produces both, and each side has to behave correctly without being able to agree with the other about what happened.

Working and stopped are not enough states

A system with two states has to place a single threshold, and both placements are wrong. Set it tight enough that the machine stops whenever anything is off-nominal, and the fleet stops constantly; the operator responds by widening tolerances, suppressing checks, or working around the system, and the property that motivated the threshold is gone. Set it loose enough that the fleet keeps running, and the machine keeps working in conditions where it should not.

There is no third placement. A binary system is miscalibrated in one direction or the other, permanently, because the situations it is trying to separate are not themselves binary.

The correct response to most degradation is a reduction in capability rather than a stop. Move slower. Shrink the working envelope. Stop accepting new tasks but finish the current one. Give up operation near people while keeping operation in a clear aisle. Hold position instead of parking. These are real, distinct states, and they are where the useful design lives.

It also has to be said that stopping is not automatically safe. A machine with momentum in a shared space stops somewhere it did not choose — in a doorway, on a ramp, across a fire lane, halfway through a transfer with a load partly supported. A fleet that stops all at once turns a floor into an obstacle course and a shift into a manual recovery. Failing safe has to be defined per situation, which makes the safe response a design decision at every rung rather than a global default.

The ladder is a designed artifact

What replaces the threshold is an ordered set of capability rungs: what is shed first, what is shed last, what the machine is still permitted to do at each level, and what conditions move it down or back up.

The ordering is the substance. Speed usually goes before the ability to complete a motion already underway. Acceptance of new work goes before work already accepted. Autonomy near people goes before autonomy in a clear aisle. Reliance on the shared world model goes before reliance on the machine's own sensors, because its own sensors are the thing it can still verify. Each of those choices encodes what the operation values, and that is a decision an operator should be making rather than one that falls out of whichever timeout expired first.

Two properties make the ladder worth the trouble. It is declared, so it exists in readable form before the failure instead of being inferred from behavior afterward. And it is ordered, so degradation is monotonic and predictable rather than a scramble in which several subsystems independently decide what to give up.

A ladder an operator can read in advance is a ladder an operation can plan around. A machine whose degraded behavior can only be characterized by testing it is a machine nobody can plan around, however sensible that behavior turns out to be.

Degradation nobody can see

A machine that has quietly dropped to reduced capability is worse than one that stopped. A stop is legible: everybody knows, and somebody responds. Silent degradation looks like normal operation while the margins are gone. Throughput falls and nobody knows why. A task completes with less clearance than anyone assumed. The operator's mental model of the fleet stays intact while the fleet stops matching it.

So every rung has to announce itself — on the machine, to the facility plane, in the operator's view, and in the record. The announcement also has to survive the failure that produced it, which rules out reporting a lost link over the link that was lost. Local indication is the part that has to be correct, because it is the only part guaranteed to be reachable.

Recovery is as designed as degradation

Coming back gets less attention than going down and deserves at least as much. A machine that has been operating alone has been acting on its own picture of a world that kept moving without it. Rejoining is not resuming.

The sequence has parts. Prove identity and integrity again, because a machine that was out of contact was also out of supervision. Reconcile local state against the shared world model, which means resolving disagreements rather than letting one side overwrite the other, since both changed. Treat stale local beliefs as claims to be verified rather than facts to act on. Then re-earn capability rung by rung instead of returning to full.

Re-earning matters because the cause of the degradation is often still present. A link that flaps comes back and leaves again. A ladder that descends slowly and ascends instantly will oscillate, and an oscillating fleet is harder to work around than a degraded one. Hysteresis is a design requirement, not a tuning parameter discovered during commissioning.

Tier independence is a claim about degraded modes

E31 Network's design goal that each tier keeps doing its job when the tier above it is unreachable reads at first like an availability statement. It is not. Availability describes how often a system is up. This describes what the system does while part of it is down: the machine without the facility plane, the facility without the fleet plane, the site without the outside world — each a named mode with a defined capability set, entered deliberately and left deliberately.

Stated that way, much of the architecture is a set of decisions about rungs. What is a machine still allowed to do when it is alone, for how long, and what does it have to prove before it is trusted with more. Those questions can be answered in advance, and the answers are the system. A fleet's character is set by the states it was designed to occupy on a bad day, not by the state it was demonstrated in on a good one.