Fine-tuning and retraining overnight
A model that exists, new data that has piled up, and a refreshed model wanted by morning. Why the overnight window suits a queue, and how a fine-tuning or retraining run is described as a job.
· 4 min read
IN SHORT
Fine-tuning and retraining are recurring, bounded runs that can wait a few hours, which is the profile E31 Network is being designed around. The job declares the starting model, the new data, the compute, and the window by which the refreshed model is needed. It waits in a queue with the meter stopped, runs when compute is free, and returns new weights.
One of the four illustrative requests the program uses is "I need compute tonight to retrain my model." It describes a situation most teams with a model in use will recognise. The model was trained on data that has since aged. New examples have accumulated. The team wants an updated model, and it wants it by the start of the next working day, not in a week.
Fine-tuning is the smaller cousin of the same situation: a base model, often an open-weight one, adapted to a team's own data in a run that is short compared with training from scratch but still far too heavy for a laptop. Both are recurring, both are bounded, and both can wait a few hours. That is the profile E31 Network is being designed around.
Why this is a network job
Retraining does not happen continuously. It happens when enough new data has arrived to justify a run, or on a cadence the team has chosen. Between runs, the compute that would do the work has nothing to do. A team that owns that compute pays for the gap. A team that rents it pays for the run.
The other property that matters is tolerance for delay. A model refreshed overnight is good enough for almost every use that is not live. The job can go into a queue in the evening, start when compute is free, and have the output back before anyone needs it. The meter does not run while the job waits. It runs only while compute is held and the run proceeds.
Describing the job
A fine-tuning or retraining job declares the work as the training procedure and the starting model: the base weights, the adaptation method, the configuration. The data is the new examples, staged for the job, along with any held-out set the run uses to check itself. The request is the compute needed and the point by which the refreshed model must be back. The output is the new weights and the run's logs.
The starting model matters here in a way it does not for training from scratch. It has to be staged alongside the data, because the job begins from it. For an open-weight base model that is straightforward to describe. For a team's own previous checkpoint it is a file the team supplies as part of the job. Either way the job carries its own starting point and does not depend on anything the network keeps between runs.
The overnight window
"Tonight" is a deadline with a soft edge, and that softness is useful. A job that must finish by a fixed moment constrains the scheduler. A job that should finish by morning, and can start any time after it is submitted, gives the scheduler room to place it where compute is free. The design intent is that a team states the real constraint, which for most retraining is a window rather than an instant.
Scheduling jobs of mixed size fairly is on the program's list of unsolved problems, and overnight retraining is one of the cases that list is about. Many teams want the same window. Some runs are short and some are long. How the network would order them so that each finishes when it is needed is being worked out, and the preview is candid that it is not finished.
Fine-tuning versus retraining
The words are often used loosely, and for the network the distinction is mostly about size. A fine-tuning run adapts a base model with a comparatively small amount of data and a comparatively small amount of compute. A retraining run may rebuild a model on a full, refreshed dataset and look much more like a training job in its demands.
The network does not need the team to pick a label. It needs the job described accurately: how much compute, for how long, over what data. A small fine-tuning job and a large retraining job go through the same steps, submit, rent, run, output, release, and are paid for the same way, by the hour for the hours compute was held.
What comes back, and what stays with you
The refreshed model is returned to the submitter. Deploying it is the team's own step. If the model serves live requests, those requests stay on the team's infrastructure or at the edge; the network trained the model but is not where it answers. If the new model should be compared with the old one before it replaces it, that comparison is a separate evaluation job, which Evaluation jobs and regression sweeps describes.
No rates are published yet. The shape of the cost is the compute held for the run multiplied by the hours it was held, with the meter stopped while the job waited and stopped again when it finished. See /examples#retrain for the illustrative request this article grew from, and Training a model on rented compute for the larger run this one is a smaller version of.
Questions this raises
- Is fine-tuning handled differently from retraining?
- Not as a category. The distinction is mostly size. Both go through the same steps and are paid for by the hour for the hours compute is held; the network needs an accurate job description, not a label.
- Can I use an open-weight base model?
- The design intent is that the starting model is staged with the job, whether it is an open-weight base or the team's own previous checkpoint. Either way the job carries its own starting point.
- How do I know the new model is better than the old one?
- That comparison is a separate evaluation job, described on its own. The retraining job returns the weights; an evaluation job scores them against the previous version on a held-out set.
KEEP READING