Large-model inference that can wait

Large-model inference splits into work that must answer now and work that does not. An assistant answering a person in a chat window is the first kind. Summarising every document in a repository, drafting a description for every item in a catalogue, extracting structured fields from a backlog of contracts, or scoring every transcript in an archive with a large model is the second. The second kind can wait, and because it can wait it can be a network job.

E31 Network is being designed for that second kind: large-model inference run as a batch, over inputs that already exist, with results returned when the pass is done. It is not being designed to serve the first kind, and it is important to say so plainly.

Interactive and asynchronous inference

The difference is not the model. The same large model can answer a live question or work through a queue of documents overnight. The difference is who is waiting. Interactive inference has a person or a system on the other end that needs the answer in the moment. Asynchronous inference has a pile of inputs and a downstream consumer that will read the results later.

The network's test applies directly. If the request must be answered now, the inference stays on infrastructure the team owns or operates close to its users. If the pile can be processed tonight, it can be submitted as a job, wait in a queue with the meter stopped, run when compute is free, and return its output.

Why large models change the calculation

Large models need a great deal of memory and compute even to run, let alone to run over a large input set. Small inference workloads can live on modest hardware a team already has. Large-model inference over a large corpus usually cannot. The team either provisions substantial GPU capacity it will use occasionally, or it does not run the pass at all.

Renting compute for the pass fits here the way it fits training. Compute is held for the job, the model runs over the inputs, the outputs are returned, the compute is released. Between passes the team holds nothing. For a team whose large-model needs come in bursts, a backlog this month and nothing next month, that is the whole case.

Describing the job

The work is the model and the inference procedure: which large model, in what form, with what prompt or instruction per input, and how outputs are to be parsed or structured. The data is the input set, staged for the job, together with the model weights if the team is supplying its own. The request is the compute and either a deadline or a duration. The output is the set of model outputs, one per input, in a form the downstream consumer can read.

Large-model jobs are divisible across inputs, but each unit may be heavy, which affects how the job is shaped. The design intent is that the team declares the model's demands and the size of the input set, so the network can hold the right compute for the right span. What a job must declare to be scheduled well is an open problem the program publishes, and large-model inference raises the version where each unit of work is itself substantial.

The queue is the point

For asynchronous large-model inference, the queue is not a compromise. It is what makes the work affordable. A job that can wait does not need compute to be sitting ready for it. It needs compute to be available at some point before its deadline. The network is being designed to hold a job in a queue until that point, with the meter not running, and then to hold compute for it and run it.

That is also why rates, when they are published, will be by the hour for the hours held. No rates are published today. The cost of a pass is compute held multiplied by time held; a job that waited in a queue for most of the night and ran for a fraction of it pays for the fraction.

What must not come here

A chat assistant, a copilot in an editor, an agent acting on a user's behalf in a live session, a model answering a voice call: these are large-model inference, and they stay on the team's own systems. Not because the network is unwilling, but because they must answer now, and a queue-based network cannot promise now. Element 31 builds Czar for teams that want this kind of serving on compute in their own rack. The network is for the backlog.

The useful exercise is to walk a team's large-model workloads and sort them by who is waiting. Those with someone waiting stay local. Those with a pile and a deadline can be jobs. Batch inference as a job describes the general form; this article is the large-model case of it. See /workloads for the full list.