Simulation to real without leaving the site

The simulation-to-real loop is well understood as a diagram. Build a model of the environment. Train or refine a policy inside it. Evaluate. Deploy to hardware. Collect what happened. Use it to correct the model, and go around again. Every serious robotics program has some version of this drawn on a wall.

What the diagram omits is where each artifact comes from. The scene is a particular building. The assets are a particular company's products and fixtures. The disturbances are a particular floor's lighting, surface, and traffic. The evaluation scenarios, if they are any good, are a particular fleet's recorded failures. Nothing in the loop is generic once it is useful.

That has an uncomfortable consequence for anyone planning to run the loop outside the fence, because the inputs it needs most are exactly the inputs that cannot be exported.

Every artifact in the loop is site-specific

Take the pieces one at a time.

The scene is the facility's layout — not a representative warehouse but this one, with its column spacing, its aisle widths, the racking reconfigured last spring, and the pallet that always ends up slightly out of position near the dock door. A layout is a security artifact in its own right. There are sites where the floor plan is the most controlled document in the building.

The assets are the products, packaging, and fixtures the machines actually handle. Getting them into a simulator means getting geometry and material properties out of the engineering systems that hold them, which is to say the company's design data. For a contract manufacturer, much of that data belongs to customers and is covered by agreements written specifically to prevent it being copied anywhere.

The disturbances are the site's own physics of nuisance: sun through the skylights in the late afternoon, the section of floor not quite flat where the lift trucks turn, the vibration when the press runs, the traffic pattern at shift change. Nobody models these who has not stood in the building, and the only record of them is sensor data from machines working there.

The evaluation scenarios are the most valuable and the most sensitive, because a good scenario is a real episode. The near-miss in the aisle by the door. The box that presented at an angle no one designed for. The grasp that slipped on a humid morning. These come from the fleet's recorded operation, the material a site is least willing and least permitted to release.

Assemble that list and the asset pipeline for a site-specific simulator turns out to be an export pipeline for the plant's layout, design data, and process video. Approval for that pipeline is not going to be granted, and it should not be.

The reality gap is mostly a local gap

There is a version of this problem where none of the above matters, because a general-purpose simulator built on general-purpose scenes transfers well enough. That version is not the one practitioners encounter.

General simulation is reasonably good at the general parts. Rigid-body dynamics, common object categories, plausible camera models, and gross kinematics are approximated well enough to be useful, and a policy trained against them starts from a better place than one trained against nothing. The residual — what is still wrong after the general parts are right — is where deployments fail, and the residual is local. It is this gripper on this material, this camera's response to this lighting, this floor's friction under this load, this fixture's tolerance stack, this operator's habit of setting the tote down half-rotated.

That inverts the usual assumption about where simulation value lives. The most valuable simulation work is not the part that could run anywhere. It is the part that depends on knowledge of a specific building, which is the part that cannot leave that building.

There is a second-order version of the same point, easy to miss. A simulator tuned to a site is itself constructed from site data, which makes it a sensitive artifact in its own right: a compressed, queryable representation of the facility. It cannot be hosted outside the boundary either. Rendering elsewhere and bringing the frames back does not help, because the scene description is the disclosure, not the pixels.

What the loop asks of the building

If the loop runs onsite, it makes demands on the infrastructure, and they are not the demands the fleet makes.

The most useful property is a rhythm. Serving a working fleet is a latency-bound workload that peaks when the machines are moving. Rendering and rollout are throughput-bound and elastic, and they do not care when they run. Those profiles are close to complementary: accelerators pinned to inference during a shift are available for simulation when the floor is quiet. Sizing a facility's compute against the union of the two workloads and scheduling between them is a materially different exercise from sizing for peak inference and buying a separate cluster elsewhere for the training work. The second option is also the one that requires the data to travel.

The second demand is a scenario library built from recorded episodes rather than authored by hand. Authored scenarios test what an engineer imagined; recorded episodes test what happened. A site that retains its edge cases at full fidelity — which it can, because they never had to survive an uplink — accumulates a regression set no external party could construct, and it accumulates more of it every shift. That library is the asset that makes onsite simulation worth building, and it exists only if retention was designed in from the beginning.

The third is reproducibility, which is unglamorous and load-bearing. A scenario that does not replay identically cannot gate anything. Seeds, simulator revision, asset revision, and policy version have to be part of the run record, or evaluation produces impressions rather than results.

Promotion is where the loop is governed

The interesting control point in a sim-to-real loop is not training. It is promotion.

Improvement in simulation is not evidence of improvement on hardware, and it must not function as permission. The rule worth designing around is that a policy becomes eligible for a machine only after it passes the site's own regression set: the recorded edge cases, the safety-relevant scenarios, and the ordinary tasks that must not quietly regress while the interesting metric improves. A policy better at the new thing and worse at the routine thing is the most common bad outcome of a refinement loop, and the only defense is a test suite made from the site's own history.

The artifact that crosses back onto hardware then has to be a specific, identifiable thing rather than a directory somebody copied. A signed, versioned bundle whose origins are recorded — which base model, which data window, which simulator revision, which evaluation run, which person approved it — is what makes the question "why did the machine do that" answerable months later, and what makes rollback a defined operation instead of an archaeology project.

Promotion is also the natural boundary crossing. If artifacts are the only thing that moves between sites, then a reviewed, signed policy bundle is the unit of shared improvement, and the data that produced it never has to be part of the transaction.

What is not solved

A note like this earns its argument by being specific about the parts that do not work yet, and there are two large ones.

The first belongs to the whole field. How much of the reality gap simulation can close for contact-rich manipulation is genuinely open. Rigid pick-and-place transfers reasonably well. Deformables, friction at the moment of slip, cluttered multi-contact interaction, and materials that behave differently on a humid day do not, and confident claims to the contrary deserve to be read carefully. A loop that closes well for navigation may close poorly for the manipulation task the site actually cares about, and there is no way to know which case you are in without running it.

The second is specific to the onsite version. A site-specific simulator is a maintenance burden that has to earn its keep. Buildings change: layouts move, products turn over, fixtures get replaced. A model that has drifted from the building is worse than no model, because it produces confident answers about a place that no longer exists. Keeping it current has to be largely automatic, derived from what the fleet already records as it works, because a simulator maintained by hand will be maintained until the first busy quarter and no further. How much of that maintenance can be driven from ordinary operation is an open design question, and the answer at a given site decides whether the loop is worth running there at all.

The loop runs inside the fence

Put the pieces together and the shape is not subtle. The inputs to a useful sim-to-real loop are the facility's layout, design data, disturbances, and recorded episodes. The compute it needs is the same compute that serves the fleet, available on the opposite schedule. The output worth moving is a policy bundle and an evaluation record — small, reviewable, signed.

E31 Network's facility plane is being designed for that arrangement: heavy inference during the shift, rendering and refinement when the floor is quiet, evaluation against the site's own history as the gate on promotion, and only reviewed artifacts crossing the boundary in either direction. Whether simulation can close enough of the gap to matter for the hardest manipulation work is not something anyone can promise today. But if it closes, it will close where the building is, because everything the loop needs is already there and none of it is going anywhere else.