Batch inference as a job

Batch inference is the plainest shape a network job can take. You have a trained model, a large set of inputs that already exist, and no one waiting on any single answer. The inputs could be a backlog of support transcripts, a catalogue of product photographs, or every document in an archive. The model runs over all of them, writes the results somewhere, and the work is done.

On E31 Network, that whole pass is one job. You describe it, compute is held for it, it runs unattended, the results come back, and the compute is released. There is no service to keep warm afterwards and no endpoint to pay for while it sits idle between batches.

What the job declares

A batch inference job declares the same four things every network job does. The work is the model and the inference pass: which model, in what form, over what inputs. The data is the input set, staged for the job so the model can read it where the compute is. The request is how much compute the pass needs, and whether it must be finished by a particular point or can simply run until done. The output is the result set: labels, embeddings, scores, generated text, whatever the model produces, packaged for return.

Declaring the request honestly matters more than it might seem. A pass over a large corpus can be split into many small units that run in parallel, or run as one long unit on fewer GPUs. The network is being designed to schedule either, but it can only schedule well if the job says which shape it is. How much a job must declare to be scheduled well is one of the problems the program lists as unsolved, and batch inference is a good place to work on it because the shape is so regular.

Why the queue is acceptable here

The test for any network workload is whether it can wait for a queue. Batch inference nearly without exception can. The inputs are already collected; they are not arriving in real time. The consumers of the output are downstream systems or people who will read a report, not a user staring at a spinner.

That tolerance is what makes the economics work. A job that can wait goes into a queue until compute is free, and the meter does not run while it waits. When compute is held and the job runs, you pay by the hour for those hours. When the pass finishes, the meter stops. The design intent is that a team pays for the inference it ran, not for a cluster sized to the biggest batch it ever expects.

If a team instead finds that each result is needed within moments of the input arriving, that is not batch inference. It is serving, and serving belongs on infrastructure the team owns or operates close to where the requests originate.

What comes back

The output of a batch inference job is a result set, returned to the submitter. For a classification pass that might be one record per input with a label and a confidence. For an embedding pass it might be a vector per document. For a generation pass it might be a file of completions keyed to prompts.

The design intent is that the output is returned, the compute is released, and what was held is accounted for in a record the team can read. Retention and deletion specifics for staged data and returned output are not yet published, and a team planning sensitive batch work should treat that as an open item rather than assume a policy.

What stays with you

A few things in the batch inference picture are not network jobs. The model itself is yours; the network runs it but does not own it. Any step that must answer a live request stays on your systems. And the decision about which inputs to include, and what to do with the outputs, is entirely upstream and downstream of the job.

The useful habit is to draw the line at the point where waiting becomes acceptable. Everything after that line can be described as a job with work, data, request, and output. Everything before it stays local. See /workloads for the full list of workloads the network is being designed for, and /examples#dataset for an illustrative request of this shape: "I need eight GPUs for four hours to process this dataset." The large-model case of the same pattern is covered in Large-model inference that can wait.