Staging data for a job
Every job declares the data it reads. Staging is how that data is made ready for the job before the meter starts, as E31 Network is being designed.
· 4 min read
IN SHORT
Staging data for a job means preparing the specific inputs the job will read so they are ready when compute is held for it. On E31 Network, data is one of the four things a job declares, and the design intent is that it is staged before the run so the metered hours go to the work. The mechanics of moving large datasets are still an open problem.
A job on E31 Network declares four things: the work, the data, the request, and the output. The data is the one most teams underestimate. The work is usually obvious, the request is a matter of sizing, and the output is whatever the work produces. The data is where the job touches everything else the team owns, and getting it ready is a step with its own name. That step is staging.
Staging means the inputs a job will read are prepared and in place before compute is held for the job. This article is about why that step exists, what it asks of the team, and what about it is still unsettled.
Why staging is separate from running
The network meters compute by the hour, for the hours compute is held for a job. The meter is being designed to start at the rent step, when compute is held, and stop when the job finishes and the compute is released. It does not run while a job sits in the queue.
If data arrived only after compute was held, the first stretch of every job would be paid-for compute waiting on inputs. That is wasted rent, and across many jobs it is a lot of wasted rent. Staging moves the waiting to before the meter. The design intent is that when a job's turn comes, its data is already where the job can read it, and the held compute goes straight to work.
There is a scheduling reason as well. A job whose data is ready can be started the moment compute frees up. A job whose data might or might not be ready is hard to place in a queue beside other jobs. Declaring and staging data up front is part of what lets the network schedule a job well, which the program lists as one of the things it is still working out.
What a team stages
A job should declare the data it actually reads, not the data the team happens to have. A retraining run reads the new examples and whatever it needs from the old ones. A footage processing job reads the footage. A simulation sweep reads the scenario definitions and the model being tested. An evaluation job reads the candidate model and the test set.
Choosing that subset is a team decision and it is worth making carefully. Staging less means less to move, which matters given how unfinished the moving part is. It also means the job is isolated to what it needs: while it runs, it reads its staged data and nothing else, and the staged data is what gets accounted for when the job ends.
In the terms of the four illustrative requests, the team that says it needs eight GPUs for four hours to process a dataset has to stage that dataset. The team with forty hours of warehouse camera footage stages the footage. The team that needs compute tonight to retrain a model stages the training data. The team that needs to run ten thousand simulations stages the scenarios. In each case the staged data is the concrete thing the job will read.
What staging asks of the team
Staging is work on the team's side before the job is submitted. The data has to be selected from wherever it lives on the team's own systems. It has to be in a form the job can read. And it has to get to where the compute is, which is the hard part.
That last part is not solved, and the program says so plainly. Getting large datasets to where the compute is appears on the published list of open problems. The article on getting large datasets to where the compute is goes into why. For now, the honest statement is that the design calls for staged data and the mechanism for very large stages is still being worked out.
What happens to staged data afterward
Once a job finishes, the output is returned to the submitter, the compute is released, and what was held is accounted for. The staged data was part of what was held. Its retention and deletion after the job are not yet published; rather than invent a policy, the program has said that these specifics are still to come, and the article on what has not yet been published about retention sets out exactly what is and is not known.
What can be said is the shape of the design: data is staged for a job, read by that job in isolation, and accounted for when the job ends. The details that sit on top of that shape are among the things a technical preview exists to settle. The how it works page shows where staging sits in the life of a job.
Questions this raises
- Is staging something the team does or something the network does?
- The team decides what the job reads and prepares it; the network is being designed to receive it as the job's declared data. Where the boundary of responsibility falls for very large datasets is part of what has not been solved yet, and the program says so.
- Does the meter run while data is being staged?
- The design intent is no. The meter is tied to compute being held for the job, which happens at the rent step. Staging happens before that, which is why it matters that the data is ready when the job's turn comes.
- Can a job start with partial data and pick up the rest later?
- That is not how the design is framed. A job declares its data up front so it can be scheduled well and run unattended. A job that needs data it does not have yet is better described as two jobs, or as one job submitted later.
KEEP READING