Skip to content

TECHNICAL PREVIEW · IN ACTIVE DEVELOPMENT

ALL NOTES

PHYSICAL AI

The latency budget is a physics problem

A control loop's budget is set by the dynamics of the task, not by product requirements. Working through where the milliseconds go — and why a network round trip can't be one of the line items.

· 6 min read

Every machine that acts in the world runs the same loop: sense, perceive, plan, actuate. The loop has a deadline, and the deadline is not negotiable, because it comes from the task itself. An object slipping in a gripper gives you the time it takes to slip. A pedestrian stepping into a vehicle's path gives you the time it takes to step. A part moving down a belt gives you the belt's speed. Nobody chose these numbers; they are properties of matter.

That is the sense in which the latency budget is a physics problem. In web systems, latency is a quality metric — slower is worse, but slower still works. In a control loop, latency is a correctness condition. Exceed the budget and the system is not degraded; it is wrong. The grasp fails, the vehicle brakes late, the weld lands off-seam.

Where the budget goes

Walk the loop and the budget spends itself quickly.

Sense. Cameras expose and read out. Encoders sample. Force sensors integrate. Before any computation happens, the data itself is already milliseconds old — sensor readout is a physical process with a physical duration.

Perceive. The dominant spender in modern stacks. Learned perception — detection, segmentation, pose estimation, or the perception front of a vision-language-action model — consumes most of what the budget has to give, which is exactly why so much engineering effort goes into making it fast on local accelerators.

Plan. Deciding what to do with what was perceived: trajectory updates, grasp adjustments, policy outputs. Cheap in some architectures, expensive in others, never free.

Actuate. Motor controllers ramp currents through real windings attached to real inertia. Like sensing, actuation takes physical time that no software can compress.

Notice what this accounting leaves: nothing. A well-engineered loop spends its whole budget on the work. There is no line item labeled "wide-area network," and there never will be.

The round trip is not a number

The usual response is that networks are getting faster. This misses what a control engineer actually needs, which is not a good average but a bound — a worst case that can be designed against.

A wide-area round trip is a sum of terms the machine does not control: local network contention, uplink scheduling, transit routing, load-balancer queues, service-side batching, the return path, and the retry logic wrapped around all of it. Most of the time the sum is small. Occasionally it is enormous. The distribution has a tail, and the tail is the problem, because a control loop that works "most of the time" is a machine that fails at an unpredictable moment for an unobservable reason.

You can engineer around a known delay — controllers compensate for fixed latencies routinely. You cannot engineer around a delay that is sometimes three units and sometimes three hundred. Variance, not magnitude, is what disqualifies the round trip. Determinism is the actual requirement, and determinism is exactly what a shared network cannot provide, at any bandwidth.

What belongs where

None of this means every computation must live on the machine. It means computations must live where their deadlines can be met.

The perception-action loop belongs on the machine — that is the edge node's job, and it must keep working when nothing else is reachable. Fleet coordination, shared world models, and heavy inference tolerate a local network hop measured on a facility's own switches: still bounded, still engineered, still inside the boundary. That is the facility plane's job. What tolerates arbitrary latency — provisioning, signed distribution, audit — can span sites, and that is the fleet plane's job.

Draw the tiers by deadline and the architecture follows. That is the design rule E31 Network is built on: the budget decides where the compute goes, because the budget was never ours to set.

DESIGN PARTNER PROGRAM

Build this with us.

We are working with a small number of teams operating real fleets in constrained environments. If the cloud is not an option where your machines work, we want to talk.