Simulation sweeps as a batch job

A simulation sweep is a large number of runs of the same simulator with different inputs: parameters, seeds, scenarios, initial conditions. Each run is independent. The point is the set, not any single run: a distribution of outcomes, a map of where a design fails, a body of experience for a policy to learn from. The illustrative request the program uses is "I need to run ten thousand simulations."

Sweeps are among the most natural batch jobs there are. They are embarrassingly parallel, they have a known end, and nothing is waiting on any individual run. E31 Network is being designed to take a sweep as one job: compute held for it, the runs spread across that compute, the results collected and returned, the compute released.

Why sweeps fit the network

A team that runs sweeps needs a great deal of compute for the sweep and very little between sweeps. The peak is high and the trough is near nothing. Owning hardware for the peak means paying for it at the trough, and that is the mismatch the network is designed to resolve. Rent compute for the sweep, run it, release it.

Sweeps also tolerate delay well. A sweep that starts tonight and finishes by morning is as useful as one that starts now, for almost every purpose a sweep serves. The job can wait in a queue with the meter stopped, start when compute is free, and return its results when done. The meter runs only for the hours compute is held.

Describing a sweep as a job

The work is the simulator and the sweep definition: what the simulator is, how each run is configured, and how the set of runs is generated. The data is whatever the simulator needs to read: models, environments, assets, scenario files, staged for the job. The request is the compute needed and either a duration or a point by which results are required. The output is the collected results of all runs, in whatever form the team wants: raw logs per run, a summary table, a dataset for a downstream training job.

The sweep definition is where precision pays. A sweep that is described as a set of independent runs, each with a known approximate cost, can be spread across whatever compute is held and will finish predictably. A sweep whose runs vary wildly in length, or depend on one another, is harder to schedule. What a job must declare to be scheduled well is one of the program's named open problems, and sweeps are a clear case to work it out on.

Independence and the scheduler

Because the runs are independent, a sweep can be shaped to fit the compute available rather than the other way round. Many GPUs for a short time or fewer for longer produce the same results. The design intent is that the team states its real constraint, usually a deadline or a budget of hours, and the network places the sweep accordingly.

The program lists scheduling mixed job sizes fairly as unsolved. A sweep is a large total amount of work made of small pieces, which is different from one long training run and different again from a quick evaluation job. How these shapes would share a pool fairly is being worked out in the preview, and the candid answer is that it is not finished.

Results, and what stays local

The output of a sweep is the aggregate. Per-run artefacts may be returned too, but the team usually wants what the set says: which parameters held up, where the failures cluster, what the distribution looks like. Returned to the submitter, that is the deliverable. The compute is released and the hours accounted for. No rates are published yet; the cost of a sweep has the shape of compute held multiplied by hours held.

Sweeps often feed training. A set of simulated episodes becomes a dataset; a dataset becomes a training job. Those are separate jobs on the network, each described on its own, and Rendering and synthetic data generation describes the version of this where the simulator's purpose is to produce training data in the first place.

A simulator running inside a live control loop, predicting what a machine should do next in the moment, is not a sweep and is not a network job. It must answer now, so it stays on the machine or on local infrastructure. The network runs the sweep that designs or validates the controller. It does not run the controller. See /examples#simulation for the illustrative request and /workloads for the full list.