Evaluation jobs and regression sweeps

Every model a team ships should have been measured first, and every change to it should have been measured against the version before. Evaluation jobs do that measuring: run a model over a held-out set, score the results, compare them with a baseline. Regression sweeps are the same thing at scale: several candidate models, several evaluation suites, all combinations, so that nothing has quietly got worse.

This work is heavy, periodic, and never urgent in the live sense. It is a strong fit for E31 Network, which is being designed to hold compute for exactly this kind of bounded, unattended pass and release it when the scores are in.

Why evaluation belongs on the network

Evaluation is inference with bookkeeping. The model runs over a fixed set of inputs, and the outputs are scored against known answers or judged by a procedure. The input set exists in advance, the scoring is deterministic or at least repeatable, and the result is a report. Nothing about it needs to happen in the moment.

What makes it a network workload rather than something to run on a desk is volume. A thorough evaluation suite, run across every candidate a team is considering, is a great deal of inference. It happens whenever there is a new candidate, which is often but not continuously. Compute owned for that purpose sits idle between candidates. Compute rented for it is held for the pass and released.

Describing an evaluation job

The work is the model under test and the evaluation procedure: the inference pass, the scoring, the comparison. The data is the evaluation set, staged for the job, along with the baseline results it will be compared against if the team has them. The request is the compute and either a deadline or a duration, which for evaluation is usually predictable because the set is fixed. The output is the scored results and a summary: metrics per suite, deltas against baseline, the specific examples that changed.

An evaluation job is divisible in the same way batch inference is. Examples can be scored independently, so the job can run wide and fast or narrow and slow. The design intent is that the team declares which it prefers and whether a deadline matters. A team that wants scores before a release decision in the morning says so. A team that is building a record over time can let the job take whatever slot the scheduler finds, with the meter stopped while it waits.

Regression sweeps

A regression sweep multiplies the evaluation: every candidate model against every suite, so the team can see not just whether the new version is better on average but whether it has regressed on anything in particular. This is where the volume comes from, and where owning compute stops making sense for most teams.

As a job, a sweep is a matrix. The work lists the candidates and the suites. The data is the union of the evaluation sets, staged once for the job. The output is the full grid of scores plus the comparisons that matter. Describing the matrix precisely is what lets the network schedule it well; how precisely a job must describe itself is one of the program's named open problems, and a well-defined evaluation matrix is a good test case because the shape is so regular.

The output, and what stays local

The output of an evaluation job is small and dense: a report, not a model. It is returned to the submitter when the job finishes, the compute is released, and the hours held are accounted for. The report is also something an auditor could read later, which matters for teams that need to show what a model was tested against before it was used. Rates are not yet published and will be when access opens; the cost has the same shape as every other network job, compute held multiplied by hours held, with nothing charged while the job waited in the queue.

A guardrail that inspects a model's answer before it reaches a user, a monitor that scores a live stream of outputs as they happen, a check that gates a robot's next action: these are evaluation in a loose sense, but they must answer now, so they stay on the team's own systems or at the edge. The network measures models before and between deployments. It is not in the path of a live request. See Fine-tuning and retraining overnight for the job that usually precedes an evaluation, Batch inference as a job for the general form evaluation is a special case of, and /workloads for the rest.