Getting large datasets to where the compute is

Describing a job is the easy part. A team can say, in a sentence, that it needs eight GPUs for four hours to process a dataset. The sentence is honest and complete as a request. It says nothing about where the dataset is, how large it is, or how it is supposed to arrive at the compute that will read it. For many jobs, that unstated part is the hardest thing about the whole exercise.

E31 Network lists getting large datasets to where the compute is as one of the problems it has not solved. This article takes that line seriously and explains what makes the problem hard, what the design already assumes about it, and what the program is and is not willing to say.

What the job model assumes

The life of a job has five steps, and the data has to be in place before the second one. SUBMIT describes the work, the data, the request, and the output. RENT holds compute and starts the meter. RUN executes unattended. OUTPUT returns results. RELEASE frees the compute and stops the meter.

The meter is being designed to run only while compute is held, so the design wants data staged before RENT. That is sound for the team's budget: nobody wants to pay for held compute while files trickle in. But it moves the burden of data movement to a point before the job exists in the pool. The time and effort of moving the data are a separate problem, and for large datasets a serious one.

Why large is different

A small dataset is easy to reason about. It can be staged quickly and the job dwarfs the staging. The difficulties start when the data is large relative to the work.

The first difficulty is time. Moving a large dataset across any distance takes long enough that it has to be planned, not just done. If the job is meant to run tonight, the staging has to start well before tonight.

The second is cost in the general sense. Large data has to live somewhere while it waits to be read, and holding it is not free in any system. The program has not published how, or whether, any of that is reflected in what a team pays, and no rates of any kind are published yet. It is a real cost somewhere, and the design has to decide where it lands.

The third is scheduling. A queue works best when every job in it is ready to start the instant compute frees up. A job whose data is still arriving is not ready, and a scheduler that cannot tell how long staging will take cannot place that job well beside others. The question of exactly what a job must declare to be scheduled well is itself on the list of unsolved problems, and data readiness is a large part of why.

The fourth is the opposite direction. Some jobs produce outputs as large as their inputs, processed footage being the obvious case. Returning that output to the submitter is the same movement in reverse, and the same difficulties apply.

What the team controls

The program does not have the mechanism settled, but a team is not helpless in the meantime, because the size of the problem is partly set by what the job declares.

A job should declare the data it reads, not the data the team owns. A retraining job that reads this quarter's new examples and a small reference set is a much smaller stage than one that reads the entire history. An evaluation job that reads a candidate model and a test set is small by nature. A footage job has to read the footage, but it can read the backlog that needs processing rather than the full archive.

The ratio of input to output matters too. A simulation sweep reads a compact set of scenario definitions and a model, and produces a set of results. A large-model inference pass over a text dataset reads the dataset and produces something of comparable size. Jobs with large inputs and small outputs, or small inputs and large outputs, each move data in one direction only, which cuts the problem down. Jobs with large inputs and large outputs move it in both.

None of this solves the problem. It makes the unsolved part smaller, which is the practical thing a team can do now.

How the program is treating it

The honest position is the one on the design partners page: this is a problem to be worked through with a small number of teams who can describe a job precisely and run it for real when the time comes, and intake is not yet open. A design partner with a backlog of footage or a large dataset to process once is exactly the kind of team whose job exposes what the data movement design has to handle.

Until then, the program's commitment is to say what it does not know. The mechanism for staging very large datasets is not settled. The cost treatment is not published. The retention of staged data afterward is not published either; the article on what has not yet been published about retention collects those gaps in one place. Naming the problem as open is the better outcome, and the preview is built on that choice.