The safety case and the learned component

A safety case is a structured argument, backed by evidence, that a system's hazards have been identified and that each one is adequately controlled. It is not a statement of intent and not a record of diligence. It is a claim, a chain of reasoning that supports the claim, and evidence at the bottom of the chain, written for a reader whose job is to find the joint where the reasoning gives way.

That structure was built for machines whose behavior could be written down. It does not accommodate a large learned policy, and the mismatch is structural rather than procedural — the evidence a hazard analysis asks for and the evidence a trained model can supply are different kinds of thing.

The interesting question is what to do about it, because one answer is available today, and it is architectural rather than analytical.

What a safety case is asking for

Strip away the formality and the argument has a simple shape. Here is how this machine can injure someone or destroy something. Here is what prevents each of those things. Here is why you should believe the prevention holds. The unit of the argument is the hazard, and every hazard has to be discharged on its own terms.

What discharges a hazard is evidence, and the traditional toolkit is narrow on purpose. A specification states what a component must do, so an implementation can be reviewed against it. Inspection and static analysis examine the artifact directly. Test coverage carries weight because the structure being covered is finite and can be enumerated. Failure mode analysis walks each way a component can break and follows the consequence downstream. Where force or speed or travel matters, a bound on worst-case behavior converts an open question into a closed one.

Those tools have a property in common. Each produces evidence about the structure of the artifact rather than about how it behaved on a sample. That is what makes the argument portable: a reviewer who was present for none of the engineering can still check it.

Why the evidence does not attach

Carry that toolkit to a large learned policy and every tool slides off.

There is no specification to review against. The model's behavior is defined by a dataset and a training objective, not by a statement of what it must do in each situation. The usual three-part relationship — the requirement, the implementation, and the argument that the second satisfies the first — is missing its first term. Whatever the model does is what it was going to do.

The behavior is not enumerable. Inspection is available in principle and useless in practice. The weights are visible, and they do not decompose into anything a reviewer can reason about. There is no branch to trace to a hazard, no path through the artifact that corresponds to the situation under discussion.

Coverage of the input space is not a quantity. The input is the space of scenes a sensor can produce. It has no dimension a test plan can partition and no boundary an argument can lean on. A large evaluation set is a sample from a distribution nobody can characterize, and the entire difficulty of a deployment is that the site keeps generating inputs the distribution did not contain.

The failure modes are distributional, not structural. Traditional analysis assumes a component fails in identifiable ways — stuck, drifting, open, silent — and asks what happens downstream of each. A learned policy does not fail like that. It is subtly wrong under conditions that were thin in training, and those conditions cannot be enumerated in advance, which is the coverage problem arriving from the other direction.

What remains is statistical evidence about average behavior. That evidence is real, and it is not nothing. It is also not the kind of evidence a hazard analysis wants, because it answers a different question: how well a policy does across a population of situations, rather than what happens in this situation, with what margin, when the thing that must not happen is at stake. Aggregate performance cannot discharge a specific hazard, and accumulating more of it does not change its type.

This is worth saying plainly. Nobody in the field has closed this. It is not a gap in one vendor's homework.

Remove the component from the argument

The response available today is not to make the learned component safe. It is to remove it from the safety argument entirely.

Treat the policy as an untrusted source of requests. It proposes; it does not dispose. Hazards are controlled instead by mechanisms small enough to specify and simple enough to analyze: interlocks, motion and force limits, speed restrictions, protected zones tied to sensed occupancy, stop authority that answers to nothing above it, and the plant's existing safety systems, which already hold final say over the actuation they own.

The claim to be argued changes shape as a result. It stops being that the model will not ask for anything dangerous, which is unarguable with the evidence available, and becomes that the machine cannot exceed its limits regardless of what is asked of it. That is a claim about the constraining mechanism. The constraining mechanism has a specification, has enumerable states, has analyzable failure modes, and has a worst-case response time that can be bounded. Every tool in the traditional kit attaches to it.

The model is now outside the argument, which also means it has to be outside the mechanism's fate. A constraint layer whose timing or availability depends on the inference plane has not been separated from it, only drawn separately.

What the trade costs

What this buys is freedom of movement on the learned side. A model can be updated, replaced, or fail outright without reopening the hazard analysis, because no hazard was discharged by an argument about the model. That matters more than it first appears. An argument that has to be redone for every change in weights is an argument that prevents updates from happening, and a fleet that cannot update is a fleet that cannot improve.

What it costs is capability. The machine can do exactly what the constraint layer permits and nothing more, so limits conservative enough to be obviously safe may be conservative enough to be useless. All the interesting engineering moves into making the constraint layer expressive without making it unanalyzable: limits that vary with task phase, envelopes that tighten near sensed people, zones that open and close with the state of the cell.

Every increment of expressiveness adds state, and state is what makes analysis hard. There is a second cost that is easy to miss. Expressiveness usually means the constraint layer needs sensing of its own, and anything it depends on inherits the full evidence burden. Move presence detection into the trusted layer to get a better limit, and presence detection is now safety-relevant, with all the specification, analysis, and failure-mode work that implies. The temptation is always to push one more function across the line in exchange for capability. Each push enlarges the thing that has to be argued.

What this does not cover

Containment discharges the hazards that come from a machine exceeding its limits. It does not touch the hazards that come from the policy doing something permitted and wrong. A machine can stay inside every envelope, respect every interlock, move at a sanctioned speed, and still set a load down where it does not belong, obstruct an egress route, or drive a process toward a state nobody wants. The constraint layer permits those by construction; permitting is what it does. Reducing that class of hazard is work on the policy, on the task representation, and on supervision. It is not work the limits can do.

The larger question stays open as well. A safety argument for a system containing learned components that a regulator, an accreditor, or an insurer will accept is not something anyone has produced, and the industry should be suspicious of the first party to claim otherwise. Containment narrows what has to be argued. Narrowing is a real contribution and not a solution.

E31 Network's answer today is the architectural one: everything learned stays off the safety path, so the property under argument is a property of the constraining mechanism rather than of the model. That is the honest position available now. It makes the argument smaller. It does not make the argument won.