Working through a large dataset once

Some compute needs happen exactly once. A team has a dataset it has never processed, or has processed only in part, and it wants to run something heavy over all of it: compute embeddings for every record, run a model over every row, transform a raw dump into a form the rest of the pipeline can use. When it is done, it is done. The dataset will not need that pass again, or not for a long while.

The illustrative request the program uses for this is "I need eight GPUs for four hours to process this dataset." It is the shape of a team that knows what it wants to do, knows roughly what it takes, and has no reason to own the compute that does it. E31 Network is being designed so that this team can rent that compute, run the pass, take the output, and walk away.

What "once" changes

A recurring workload gives a team a reason to think about owning compute, because the hardware will be used again. A one-time pass gives it none. Whatever is bought for the pass sits idle afterwards, and whatever is provisioned in a cloud account tends to stay provisioned longer than intended.

The network's model is built for the opposite. Compute is held for the job and released when the job finishes. The meter runs for the hours held and stops. There is nothing left over to decommission, nothing to remember to turn off, and no capacity sized for a peak that will not recur. For a genuinely one-time pass, that is the whole argument.

Describing the job

The work is the processing step: a model to run, a transformation to apply, an analysis to compute. The data is the dataset, staged for the job. The request is the compute and the duration, which for a one-time pass is often well understood in advance because the team has usually run a sample. The output is the processed dataset or the derived artefact: embeddings, features, a cleaned table, a set of summaries.

A dataset pass is usually divisible, since records can be processed independently, which gives the team a choice between a short, wide request and a long, narrow one. The design intent is that the job says which it wants. If the output is needed by a particular point, the request should say so. If it is not, the job can take whatever slot is free, and the meter does not run while it waits in the queue.

Staging is the hard part

Getting large datasets to where the compute is sits first on the program's published list of things not yet solved, and the one-time pass is the purest form of the problem. For a recurring job a team might amortise the effort of moving data. For a single pass the staging may take a meaningful fraction of the total effort, and the team will do it exactly once.

The preview does not claim this is solved. The design intent is that staging is a declared step of the job, visible and accounted for, rather than a surprise. A team with a large dataset and a single heavy pass to run would be contributing to this question as a design partner. Intake is not yet open; /design-partners describes what the program is looking for.

After the pass

The output is returned to the submitter. The compute is released. The hours it was held are accounted for in a record the team can keep. No rates are published yet; when access opens they will be, and the cost of the pass will have the shape of compute held multiplied by hours held.

What happens to the staged copy of the dataset after the job is a retention question whose specifics are not yet published. For a dataset a team would not want to leave behind, that is an item to raise before the job, not after. The one-time pass is a good workload for the preview precisely because it is simple enough to make these questions concrete. See /examples#dataset for the illustrative request, Image processing at volume for a dataset made of pictures, and Batch inference as a job for the case where the pass is a model run over every record.