An onsite inference plane is not a small cloud

The facility plane runs accelerators, serves inference, schedules workloads, and lives in a rack. From a distance it looks like a small private cloud, and the comparison is useful right up until it starts making design decisions on your behalf.

The problem with the analogy is not that cloud architecture is bad. It is very good, at what it was built for. The problem is that it is a set of answers to a particular set of conditions, and it encodes those conditions as assumptions so thoroughly that they stop being visible as assumptions at all. Move the design into a working facility and they come along silently. Most of them are false there.

Naming them one at a time is the useful exercise, because each one has an architectural consequence.

The fleet is the load

A cloud service under pressure buys its way out. More instances, more nodes, a larger tier — capacity is a purchasing decision made at runtime, and an enormous amount of cloud engineering exists to make that decision automatic and invisible. Onsite there is no such lever. The hardware in the building is the hardware in the building, sized against a fleet, installed once, and unchanged until someone physically brings more.

That inverts the design goal. Instead of absorbing peaks by growing, the plane has to behave predictably at saturation: know its own limits, refuse work rather than accept it and quietly miss, and shed capability in an order somebody chose in advance. Autoscaling is not merely unavailable here — the habit it teaches, that saturation is a temporary condition which resolves itself, is exactly the wrong reflex for a system whose consumers are machines in motion.

The compensation is that the load is not arbitrary either. A cloud service faces demand from an unknown internet. The facility plane faces a known population of machines with known duty cycles doing known work, which means capacity can be engineered against the actual fleet rather than a forecast. Fixed supply is a real constraint. Bounded, characterizable demand is a real gift, and it is the reason fixed supply is workable at all.

Multi-tenancy, inverted

A cloud's central isolation problem is many mutually distrusting tenants sharing hardware. Hypervisors, namespaces, network policy, quota accounting — a large fraction of the stack exists so that one tenant's behavior does not show up in another tenant's results.

The facility plane has one tenant. The customer owns the building, the machines, the data, and the plane itself. The isolation problem does not go away, but it rotates onto a different axis: what has to be isolated is not tenants but criticality. Fleet coordination and the shared world model sit on one side. Heavy inference, batch evaluation, and overnight adaptation sit on the other. There must be no path by which the second class can starve the first, no matter how much work it is given.

The failure model differs accordingly. In a multi-tenant cloud, an isolation failure is a security event, and the mitigation is a boundary strong enough to resist a hostile neighbor. Here the neighbor is not hostile — it is your own training job, running with permission, doing what it was asked. The failure is not disclosure. It is a coordination decision arriving after the moment it was needed. So the boundaries that matter are timing boundaries: reservations of compute, memory bandwidth, and accelerator access that hold under load, rather than fair-share schemes that come out even over an interval nobody on the floor cares about. Those are different mechanisms with different costs, and a design that inherits the cloud's isolation toolkit tends to get the strong version of the wrong one.

Nothing above it, and nobody logging in

A cloud region assumes a control plane above it: a global service that holds configuration, mediates identity, tracks fleet membership, and can be reached. Entire categories of failure are handled implicitly by the expectation that something upstream will notice and intervene.

The facility plane cannot assume that, because the fleet plane may be absent by design. An air-gapped site, a forward location, a plant whose security posture simply does not permit the connection — these are supported configurations rather than outages. Self-sufficiency is therefore not a resilience feature layered on top. It is the baseline. Everything the plane needs in order to boot, verify itself, admit nodes, coordinate a fleet, and keep running has to already be inside the boundary.

The second absence is a person. Cloud operations assume an engineer with credentials who can open a shell, inspect a running process, install a missing package, and restart something by hand. The facility plane is a sealed node: a signed, read-only image carrying one defined workload, with nothing to log into. That is deliberate, and it is what makes the integrity question narrow enough to answer cryptographically. The price is that every problem a cloud would solve by logging in has to be solved in the design instead — recovery paths, health gating, safe restart, state that is observable without a debugger attached. Anything that would normally be fixed by hand must be handled by architecture, or honestly acknowledged as not handled.

Authoritative state, not a cache

Most cloud services are built to be disposable. State lives in a managed store, compute instances are interchangeable, and recovery means starting a fresh one and letting it reload from the source of truth. That whole pattern rests on the source of truth being somewhere else.

Onsite, it is not. The shared world model is the authoritative picture of the facility, and there is no upstream copy to reload from. The day's episode data is the original record and the only one. That turns restarts, upgrades, and recovery from routine operations into events with real consequences. Restarting the component that holds the fleet's picture of the space is not free. An upgrade has to be designed around state that must survive it. Recovery cannot mean coming back empty and resyncing, because there is nothing to resync against.

The practical effect is that storage onsite is an archive rather than a cache tier, and durability, integrity, and orderly handover during upgrades become first-order design requirements rather than properties inherited from a managed service.

What failure costs

A cloud service's failure produces errors. Requests return the wrong thing or nothing, clients retry, a status page updates, an incident review follows. This can be enormously expensive, but the consequence is informational and the recovery is a redeploy.

This plane's failure produces stopped machines. Not slower throughput — motion that ends, in a building where those machines were carrying loads, occupying aisles, and working near people. Even a correct, clean stop is disruptive in a way an error page never is, because it leaves physical objects in physical places and requires people to walk out and resolve them.

That reorients the objective. It matters more that the plane's failure mode is one the fleet can absorb — machines falling back to local autonomy, claims expiring safely, work in progress finishing or halting in a known posture — than that failures are statistically rare. Designing carefully for what your consumers should do without you is not a common cloud pattern, because a cloud's consumers are software, and software can wait.

What it borrows anyway, and what that leaves

A great deal does transfer, and pretending otherwise would be its own mistake.

Scheduling discipline transfers: resources requested explicitly, accounted, and reclaimed, with workloads declaring what they need rather than taking what is available. Resource isolation transfers as a set of mechanisms even though the axis changes, because the primitives for partitioning compute, memory, and device access are the right primitives. Signed, immutable artifacts transfer directly, and matter more here than in a cloud, since the sealed node's entire security argument depends on the running image being exactly the approved one. Health and readiness gating transfers almost unchanged — refusing to send work to a component until it has demonstrated readiness is precisely the right model for admitting a node to a fleet. Declarative configuration, deliberate upgrade orchestration, and serious observability all keep their value.

What does not transfer is the ring of assumptions around those techniques: that capacity is purchasable at runtime, that the dangerous neighbor is someone else's workload, that something above will notice, that a person can get in, that state is a cache, and that failure is counted in errors.

So the facility plane is being built as a fixed-capacity, self-sufficient, sealed appliance with a saturation behavior it declares in advance — not as a region with fewer racks. It takes the discipline the industry earned and leaves behind the conditions that made that discipline cheap, because inside a facility, none of those conditions hold.