Keeping what the fleet saw

A fleet records everything it perceives, all shift, every shift. Camera streams per machine, depth, force, joint telemetry, the intermediate outputs of every perception stage, and the decisions taken on top of them. Keeping all of it permanently is not an option — not because of someone else's policy or someone else's bill, but because storage inside a building has a physical footprint, a power draw, and a purchase order, and the recording never stops.

So retention is a design problem with a forced choice at its center: what to keep, at what fidelity, for how long, and on what basis to decide. Answering that well requires first answering a question that usually goes unasked. Who is the record for?

There are three answers, and they are not variations on each other.

Three consumers that want different things

Learning wants the rare, the surprising, and the corrected. A thousand ordinary pickups teach a model almost nothing. The one where the load was wrapped wrong, or the light came in at an odd angle, or a person stepped in and took over, is the entire training signal. Learning wants those episodes at full fidelity, because reduced data cannot be un-reduced, and it wants the synchronized sensor streams alongside the actions actually taken. It also wants them findable by situation rather than by timestamp, since an episode nobody can locate is not in the training set no matter where it is stored.

Investigation wants something structurally different. It wants a complete, unedited, tamper-evident record of a bounded window around one specific event. Not the interesting parts — everything, in order, with nothing selected out, because ruling out selection is precisely what an investigator has to be able to do. It wants that record on demand, possibly long after the fact, when an incident review or a regulator's question surfaces, and it wants to demonstrate that nothing changed between the event and the review.

Operations wants neither. Rates, cycle times, utilization, where the fleet loses time, which machine is trending toward service. Aggregates and trends over long windows, computed continuously. Operations needs no frames at all, and handing it frames converts a cheap question into an expensive one.

Conflate the three and the result is a storage design that fails all of them at once: full fidelity held too briefly for the investigation, summaries too coarse for the learning, and raw episodes retained indiscriminately because nobody could say which ones mattered.

The tiering that follows

Separating the consumers produces a structure with distinct paths rather than one archive with a policy attached.

Everything passes through a short, complete, high-fidelity window first. This is not a decision, it is a buffer, and its purpose is to buy time. Promotion decisions made in the moment are made with the least information anyone will ever have about the episode; a window lets them be made afterward, when the outcome is known and the operator has had a chance to weigh in.

Out of that window, episodes that matter are promoted to long-lived storage at full fidelity. Everything else is reduced aggressively before it ages out — statistics, event markers, coarse trajectory outlines, whatever preserves the operational question without preserving the frames. Reduction is not deletion, and the distinction is worth keeping in the design, because a record that says an ordinary episode occurred and went normally is cheap and occasionally decisive.

Alongside all of that runs a separate path with harder guarantees for anything an investigation may need: write-once, integrity-protected, with a retention clock set by policy rather than by remaining capacity, and independent of whatever the learning pipeline judged interesting. It has to be separate, and the reason is a conflict of interest. A record whose contents were selected by the same system whose behavior is under review is not evidence. It is a summary produced by the party being examined.

Promotion is a policy problem that fails quietly

Deciding what to promote is the hard part, and it does not reduce to a fixed rule. Some triggers are obvious: an intervention, a fault, a safety stop, a failed task, a manual flag from an operator. The valuable ones are less obvious — an episode where the policy's uncertainty spiked, a situation the fleet has not encountered before, a case where two machines' observations disagreed, a near-miss that produced no event because the avoidance worked. Those are learned judgments, made by models, against a distribution that shifts as the facility changes. They will be wrong sometimes.

What makes this dangerous is that promotion failures are silent. Set the policy too narrow and nothing breaks, no alarm fires, no operator notices, throughput is unaffected. The cost arrives months later, when a new failure mode appears, someone goes looking for prior examples, and finds that they were reduced to summaries the week they happened. The bill for a bad promotion decision is paid entirely in the future, by people who did not make it.

The design responses are unglamorous. Bias toward keeping when the policy is uncertain, within whatever footprint the site can carry. Version the promotion policy itself and keep it reviewable, so that a change in what gets kept is a change someone can point at rather than a drift nobody can reconstruct. And retain enough about what was reduced to know later that something unusual happened, even when it can no longer be replayed. Knowing an episode existed and was not kept is a much better position than not knowing it existed.

A pile of video is not an archive

Retained data that cannot be located is a liability rather than an asset. It occupies the footprint, carries whatever obligations attach to its contents, and returns nothing, which is a precise description of retention without an index.

So the index is part of the storage design and not a feature to add later. Episodes need boundaries, because a recording is not a continuous stream but a sequence of bounded units with a start, an end, a machine, a task, and an outcome. Events need tags applied at capture, while the context still exists, rather than reconstructed afterward from correlated timestamps. And the archive needs to be searchable by situation rather than only by time and machine: find the cases where a machine approached an occluded corner carrying a load, where the surface was wet, where the policy hesitated before committing.

That last capability is the one that forces the schedule. Searching by situation implies representation and indexing work that has to happen while the data is still in the full-fidelity window, because it cannot be done retroactively against material that has already been reduced. Findability is not a query problem bolted onto a store. It is a capture-time obligation.

The investigation path needs the same discipline for a different reason. Producing the complete window around a specific event requires knowing which retained material corresponds to that event, and an archive that cannot answer that question cannot serve the request no matter how faithfully it kept the bytes.

The boundary is what makes it possible

None of this is affordable if the record has to leave. A single camera stream at working resolution outruns a typical site's uplink, and a fleet's worth of them is not a bandwidth problem to be optimized but an architecture that does not exist. Any design requiring the raw record to travel has to reduce first, in the moment, before anyone knows what will matter — which is exactly the decision the full-fidelity window exists to defer.

Keeping the record inside the facility removes that forcing function. Fidelity is traded against the site's own storage rather than against a link, and the trade can be made after the fact instead of during capture. Full-fidelity retention is possible precisely because nothing has to survive an uplink.

The discipline that comes with it is that what leaves is derived. Model updates, evaluation results, aggregate metrics, attestation evidence, audit records — artifacts produced from the record, reviewed, and signed. The raw record itself does not leave, and not as a strongly discouraged option. As a path the architecture does not provide. Sites share capability through artifacts; they do not pool footage.

The tension we have not resolved

Retention depth argues against physical footprint, and the argument does not have a clean winner. Every additional day of full-fidelity window is rack space, power draw, and rejected heat inside a building that was never designed as a datacenter and rarely has room to spare. The learning case always wants more depth. The siting case always wants less equipment. Storage density improves, but a fleet's output grows with every machine and every sensor added to it, and the two curves do not obviously converge.

We do not think that tradeoff has a universal answer, and we are suspicious of anyone who offers one. It is a per-facility decision about what that site's record is worth to it, and the right architecture is the one that makes the decision explicit and adjustable rather than burying it in a default. Retention on the facility plane is being designed as several paths with different guarantees for that reason: the three consumers of a fleet's experience want genuinely different things, and a single tier can only be tuned to disappoint all of them evenly.