Skip to content

TECHNICAL PREVIEW · IN ACTIVE DEVELOPMENT

ALL NOTES

ARCHITECTURE

Time is the first shared resource

Clock synchronization gets treated as plumbing. In a fleet it is a correctness dependency — fusion, deconfliction, and incident reconstruction all fail quietly when the participants disagree about when things happened.

· 8 min read

Nothing in a fleet system works before the clocks agree. That reads like an overstatement, and it is not one. Sensor fusion, the shared world model, deconfliction, and every reconstruction of what happened afterward all rest on the same assumption: that two observations carrying nearby timestamps describe nearby moments. Break the assumption and each of them fails, mostly without announcing it.

Time synchronization is nevertheless filed under infrastructure plumbing — a daemon someone enables during provisioning, a service that either works or produces a log line nobody reads. That filing is a mistake with consequences. In an embodied system, a clock is not a convenience for correlating logs after the fact. It is a dependency of correctness, sitting underneath components that will produce confident, well-formed, entirely wrong output when it is violated.

The case is easiest to make from the consumers.

What depends on agreement about when

Start on one machine. A robot carries sensors that do not sample together: cameras exposing on their own cadence, depth returning at a different rate, encoders sampling continuously, force integrated over intervals. Fusing them requires knowing which readings correspond to the same instant — when each was captured relative to the others, not when it arrived at the process consuming it. Get that wrong and the machine assembles a scene in which the arm is where it was a moment ago and the object is where it is now. A policy conditioned on those inputs is reasoning about a world that never existed, and it will act in that world decisively.

Extend the same requirement across the network and it becomes the precondition for the shared world model. Two machines observing the same region contribute views that either can or cannot be placed on a common timeline. If they cannot, the model has no way to distinguish a genuine disagreement — two sensors seeing different things — from an artifact of ordering, where both saw the same thing at different moments and the newer report merely looks older.

Deconfliction raises the stakes. A claim on space is a statement about an interval: this lane is mine from now until I clear it. That means nothing unless the participants agree on when now is and when the interval ends. Two machines with drifting clocks can hold claims each believes are sequential and which in fact overlap. That is not a data-freshness problem with a data-freshness consequence. It is two machines in the same aisle.

Afterward, reconstruction depends on the same foundation. An investigation has to order events across several machines and the facility plane — what was perceived, what was decided, what was commanded, what moved, what stopped. If the clocks disagree by more than the intervals separating those events, the ordering cannot be recovered, and the central question of whether the system saw the hazard before or after it acted has no answer. The record can be complete, signed, and useless.

Bounded and known beats accurate

The property that matters here is not absolute accuracy against civil time. Nothing on a floor cares what an official reference thinks the hour is. The machines care about each other, which makes the real requirement a bounded offset between participants — and, in the part that most often gets dropped, a bound that is actually known.

An unknown offset of a small amount is worse than a known offset of a larger one. A system that knows its participants may differ by up to some quantity can design against it: widen the margin on a claimed interval, decline to fuse observations whose timestamps sit closer together than the bound, mark a sequence as indeterminate rather than guessing at it. A system that believes its clocks are perfect and is quietly wrong cannot do any of that, because it never thinks to ask.

The practical form of this is that a timestamp should be understood as an interval rather than a point, with the uncertainty travelling alongside the value instead of being discarded at the first interface. It also means a machine's sense of elapsed time and its view of the shared timeline are different quantities and should not be the same clock, since a correction that steps the shared view backward will produce negative durations in any control code that trusted it.

A clock that is quietly wrong

A clock known to be unsynchronized is a manageable condition. The machine can be excluded from fusion, denied new claims, held to conservative local behavior, and flagged for attention. Everything downstream can adapt because everything downstream was told.

A clock that is wrong while believed correct is the dangerous case, and it is dangerous in the specific way that silent failures are. Nothing stops. Every consumer continues at full confidence on a false premise, and the resulting misbehavior presents as a perception problem or a planning bug — investigated in the wrong place, for as long as it takes someone to suspect the clock.

So synchronization state has to be an exposed, first-class signal rather than a metric on a dashboard. Fusion should ask whether its sources are inside the bound before combining them; the claim protocol should ask before granting an interval; the recorder should ask before writing a timestamp it may later have to defend. A participant that falls outside the bound loses those privileges immediately rather than coasting on a synchronization that has expired. It is the platform's integrity rule carried over to time: a node that cannot demonstrate a property is treated as not having it.

The awkward cases

Coverage gaps come first, because they are routine rather than exceptional. A machine that spends stretches behind steel or beyond radio has been free-running, and its offset has been growing at whatever rate its oscillator and its temperature dictate. On rejoin it cannot simply resume participating. It has to re-establish its bound before its observations are admitted or its claims honored, and everything it recorded while away has to carry honest uncertainty rather than being back-dated as though it had been disciplined the whole time.

The air-gapped facility breaks conventional designs. A site with no external connectivity cannot reach any external time source, and since that is a supported posture rather than an outage, no outside reference can be a precondition for anything. What matters inside the fence is internal agreement, which does not require an external authority. The honest consequence is that such a facility's timeline may drift relative to civil time, and the system has to say so plainly in any record meant to be read outside the building.

Cold start is the third. At power-on a node knows nothing about its offset, and the tempting behavior is to assume the last known good state and carry on. The correct behavior is to treat the unsynchronized period as unsynchronized: withhold participation until a bound exists, and never emit a timestamp implying more confidence than was present at capture.

The fourth is the firmest. Nothing safety-relevant may depend on the network's clock. Interlocks, stop authority, and motion limits run on the machine's own control path with its own local timing, because a safety function that waits on distributed agreement about the current time has imported every failure mode of the network into the one place where failures are physical. The determinism split applies to clocks as much as it applies to compute.

A facility has to hold its own time

A site that depends on an external reference for its internal ordering has taken on a dependency it cannot inspect and cannot defend. It fails when the link fails, and it is trusted without verification when the link is up, which is the worse of the two conditions.

So the facility needs to hold its own time discipline locally: an authority inside the boundary, serving the fleet, with bounds it can state, monitor, and report. Machines discipline against it. It requires nothing from outside in order to be useful, and where an external reference is available and permitted, that reference is an input to be weighed rather than a prerequisite to be depended on.

This is the same shape as the rest of the plane, for the same reason. The facility has to be able to do its job with the fence closed.

Time belongs in the trust chain

Signed records are only orderable if the timestamps inside them mean something. An attestation that a node was in a given state, an audit record of a decision, a retained episode handed to an investigation — each is a claim about a moment, and a signature protects the contents of that claim while saying nothing at all about whether the moment is right. A perfectly signed record with a wrong timestamp is a well-protected falsehood, and it is worse than an unsigned one because it invites belief.

The consequence for the design is that synchronization state belongs inside the signed record, not beside it: what the node's offset bound was, which authority it was disciplined against, whether it was synchronized at that moment at all. A reviewer can then reason about ordering instead of assuming it, and an ordering that cannot be established can be identified as unestablished rather than quietly invented.

Which is the argument compressed. Time is not a service running beside the system, offering a convenience to whichever components ask for it. It is part of what the system's evidence means, and E31 Network is being designed to treat it that way — facility-scoped, with stated bounds, exposed state, and explicit degradation — because every other guarantee in the architecture is a statement about a moment.

DESIGN PARTNER PROGRAM

Build this with us.

We are working with a small number of teams operating real fleets in constrained environments. If the cloud is not an option where your machines work, we want to talk.