Training a model on rented compute
A training run needs a lot of GPUs for a stretch and then none of them. How a run is described as a job on E31 Network, why staging the data is the hard part, and what the meter covers.
· 4 min read
IN SHORT
A training run fits E31 Network's job model directly: the script and configuration are the work, the staged training set is the data, the GPUs and duration are the request, and the trained weights are the output. Compute would be held for the run and released when it finishes, with the meter running only while compute is held. Staging large training sets is a named open problem.
Training a model is the workload that makes owning compute feel necessary and renting it feel obvious at the same time. A run needs a lot of GPUs for a stretch, and then it needs none of them. Teams that own a cluster sized for the run spend most of their time paying for idle hardware. Teams that have none cannot run it at all. E31 Network is being designed for the gap between those positions: compute held for the run, then released.
The run as a job
A training run fits the network's job shape without much translation. The work is the training script and its configuration. The data is the training set, staged so the compute can read it at full speed. The request is how many GPUs, for how long or until the run converges, and whether there is a point by which the model is needed. The output is the trained weights, along with whatever logs and metrics the run writes.
Once submitted, the run goes to a queue if compute is not immediately free. When compute is held, the meter starts and the run proceeds unattended. It is isolated from other jobs while it runs. When it finishes, the weights are returned to the submitter and the compute is released back to the pool. Nothing about the run requires anyone to watch it, and nothing is left behind afterwards that the team continues to pay for.
Staging the data is the hard part
Getting a large training set to where the compute is sits on the program's published list of problems not yet solved, and training is the workload where it bites hardest. A training set can be far larger than the model it produces, and the run cannot start until the data is readable by the GPUs that will train on it.
The design intent is that data is staged for the job as a declared step, not something the team improvises. What that staging looks like in practice, how long it takes relative to the run, and how it is accounted for are open questions the preview exists to work through. A team considering training on the network should expect to talk about the dataset's size and location in the same breath as the model's architecture.
Sizing the request
The request for a training job has a shape that most other workloads do not: it may not have a known end. A run might be specified as a fixed count of steps, in which case the duration is predictable, or it might run until a validation metric stops improving, in which case it is not.
The network is being designed to accept both. A job can ask for compute for a stated span or until done. The honest version of "until done" is a request that also carries a ceiling the team is comfortable paying for, so the meter has a limit even when the run does not. Scheduling a long, large request fairly beside many short, small ones is the other published open problem, and large training runs are exactly the kind of job that makes it hard.
Paying for the run
You pay for the hours compute is held for the job. The meter does not run while the job waits in a queue. It starts when compute is held, runs through the training itself, and stops when the run finishes and the compute is released. There is no charge for the hardware's existence between runs, because between runs the team holds none of it.
No rates are published yet. They will be published when access opens. What a team can plan on now is the shape: the cost of a run is the compute held multiplied by the time it was held, and nothing else. Anything a team hears beyond that shape is not from this program.
After the run
The trained model is the team's output, returned to them. What the team does with it next is outside the network. If the model is going to serve live requests, those requests belong on infrastructure the team owns or operates near its users, not on the network. If the model is going to be checked against a held-out set, that check can be its own batch job, described separately; Evaluation jobs and regression sweeps covers that shape.
For a team that wants compute in its own rack permanently, Element 31 also builds Czar, a sealed appliance on its own subscription. The network is for the runs that do not need a rack of their own. The illustrative request on /examples#retrain is the near neighbour of this one: "I need compute tonight to retrain my model." Fine-tuning and retraining overnight follows that thread.
Questions this raises
- What if the run has no known end?
- The network is being designed to accept a request for a stated span or until done. The honest form of until done also carries a ceiling the team is comfortable paying for, so the meter has a limit even when the run does not.
- Do I keep the trained model?
- Yes. The weights are the job's output and are returned to the submitter. What the team does with them next is outside the network; serving them to live users stays on the team's own infrastructure.
- Is renting the right choice for every training team?
- Not for every team. A team that trains continuously may want compute in its own rack, which is what Element 31 builds Czar for. The network is for runs that do not need a rack of their own.
KEEP READING