Skip to content

TECHNICAL PREVIEW · IN ACTIVE DEVELOPMENT

ALL NOTES

PHYSICAL AI

Designing for the link that is not there

Offline is not one condition. It is at least five, and a system that treats them as a single failure state will behave badly in all of them. What it actually costs to design disconnected-first.

· 8 min read

Most systems have one word for the absence of a network, and that word does most of the damage. "Offline" gets used as though it names a condition, when it names the edge of a set of conditions that share nothing except the failure of an assumption. A machine out of coverage for a few seconds behind a steel column and a machine at a site that will never be connected to anything are both offline, and almost nothing about how they should behave is the same.

The consequence of collapsing them is predictable. A system with one code path for "no network" has one behavior for several situations, and it is correct for at most one of them — usually none, because the path was written for the case a developer imagined and tested against the case the office network produced.

Getting this right starts with refusing the single word.

Each of these is a distinct engineering situation, not a severity level of the same one.

  • Absent. No link at all, by design or by geography. A vessel underway, a mine, a site where connecting is forbidden by the environment's own rules. The honest case, and the easiest to build for, because it permits no wishful thinking.
  • Intermittent. Coverage that comes and goes as the machine moves — behind a steel column, into a container stack, through a doorway with a metal frame. The gaps are short, frequent, and unannounced, and the machine discovers the boundary by crossing it.
  • Degraded. A link that is present but slow, lossy, or violently variable. Contended radio, shared backhaul, weather on a microwave hop. Everything still completes, eventually, which is the trap.
  • Contested. A link that is present and functional and should not be used, because emitting is a signature, the path is untrusted, or policy forbids the traffic. Availability and permission are different variables, and only one is visible to a network stack.
  • Partitioned. The site is internally healthy but cut off from everything above it. Nothing local is broken. What is broken is distribution, oversight, and governance.

No two of these want the same response.

The dangerous middle

The instinct is to worry most about the absent case. The absent case is the safe one: a system built for a site that never connects has no illusions to lose.

The dangerous regimes are the middle ones, and degraded is the worst, because software keeps hoping. A request that will eventually complete is worse for a machine with a deadline than one that fails immediately. Retry logic backs off and tries again. Timeouts written for a datacenter expire on a scale meaningless to a control loop. The machine does not fault, it hesitates — and a machine holding a part in mid-air while a library works through its retry schedule is in a worse state than one that concluded instantly it was alone and acted accordingly.

Intermittent has its own signature failure, and it arrives on reconnection rather than during the gap. A system that treats a restored link as an invitation to catch up produces a burst of queued traffic, a flood of stale writes, and a reconciliation storm across every machine that crossed back into coverage at the same moment — which, in a facility with one bad corridor, is most of them. The gap was survivable. The rejoin is what takes the site down.

Contested is the regime most architectures cannot express at all, because they conflate reachable with allowed. A stack that transmits because it can has no way to represent a situation in which transmitting is the failure. That distinction has to exist as a policy state separate from the connectivity state, or it does not exist.

Autonomy is not a degraded mode

Here is the structural error, and it determines everything downstream of it.

If a system is built as a connected system with a disconnected fallback, the disconnected path is the least-exercised code in the product. Development happens in offices with good connectivity. Integration tests run against reachable services. The offline branch accumulates across releases without ever being the path anyone is watching, and it is invoked for the first time under conditions that are already adverse.

A failure path entered only under stress and never under test is not a failure path. It is a latent outage with a scheduled arrival time.

Worse, the degraded mode is usually specified as a subset: the machine can still do these reduced things until the link returns. That framing puts capability in the connected case and lets it leak away as connectivity does. For a machine whose entire justification is working where people and networks cannot reliably go, this is backwards. The disconnected case is not the exception the design tolerates. It is the case the design is for.

The inversion that fixes it is easy to state and expensive to honor: build disconnected-first, and treat connectivity as an optimization that can be removed without loss of capability. If removing the link costs convenience, coordination scope, or update latency, the design is working. If removing the link costs function, the fallback was a story.

What disconnected-first costs you

The discipline only means something if it forbids things, so the forbidden list is worth stating plainly.

No license check — nothing may verify entitlement against a remote service at start, at intervals, or anywhere that failure to reach it changes behavior. No cloud control plane: the authority to schedule work and admit a machine to the fleet lives inside the boundary, because an external controller is a runtime dependency whether or not it is called one. No telemetry callback, including the well-intentioned heartbeat that becomes load-bearing the day someone writes an alert on its absence. No remote model fetch on the critical path: a policy is either resident or the machine does not claim the capability that needs it.

Identity and authorization have to be verifiable locally, a deeper constraint than it first sounds. Every permission decision must be answerable from an artifact the site already holds — a signature, a certificate, a signed policy — rather than by asking anyone. Anything requiring a conversation with an external authority at the moment of use is a runtime dependency wearing a security label.

There are real losses here and it would be dishonest to skip them. Central rollout gets slower and coarser, because a change reaches a site when someone carries it there. Fleet-wide dashboards stop being live. Remote debugging during an incident is not available in the moment. The argument for paying is not that these costs are small; it is that the alternative buys convenience with the machine's ability to work at all.

What disconnected-first makes you build

The forbidden list has a mirror image, and the mirror image is the more interesting half.

Local decision authority. The machine and the site have to decide and commit, not decide provisionally pending confirmation. A decision that becomes real only when something distant acknowledges it is not authority; it is a queued request with optimistic rendering. Authority means the site's answer is the answer, including when the site turns out to be wrong, which forces the design to specify who resolves that and how.

Local state that is authoritative rather than cached. A cache implies a source of truth somewhere else, and if the truth is elsewhere the system is connected-first with extra steps. The site's record of what it did, what it observed, and what it believes has to be the record — complete enough to operate from indefinitely, with no expectation that anything upstream will eventually correct it.

Reconciliation on rejoin rather than replay. This is where most designs go wrong. While disconnected, the site did real work: tasks completed, exceptions handled, state advanced. That is not a queue of pending intentions to be re-executed when the link returns; it is a completed history to be merged. Replay assumes the disconnected period was a delay, merge assumes it was operation, and merge forces an answer in advance to what happens when two histories disagree.

Distribution changes shape too: updates arrive as signed bundles that cross the boundary on the operator's schedule and verify against local trust, rather than as a stream from a repository. A boundary crossing becomes an event with a decision attached, which is what makes it auditable.

Disconnected as a supported state

The test for whether a system has done this is not whether it survives a link failure. Most systems survive a link failure for a while. The test is whether the disconnected configuration is a configuration — something an operator can select, run indefinitely, and be supported in — rather than a condition the system endures until circumstances improve.

E31 Network is being designed against that test. A facility that never connects to anything is intended to be a fully supported deployment rather than an exception negotiated case by case: decision authority on the machine and in the facility plane, state that is authoritative where it is produced, and updates that move as signed bundles across a boundary the operator controls. Where a link exists it is an optimization, and removing it costs speed and closeness of oversight, nothing else.

Disconnected is a supported state, not a failure state. Everything above is what it costs to mean that sentence literally.

DESIGN PARTNER PROGRAM

Build this with us.

We are working with a small number of teams operating real fleets in constrained environments. If the cloud is not an option where your machines work, we want to talk.