Running unattended: what a job does when nobody is watching

The run step is the middle of a job's life, and it is the step nobody is meant to see. On E31 Network, a job is designed to start when compute is held for it, work through its declared task, and finish by producing its declared output, with the submitter disconnected the whole time. This is the step that makes the network useful for a team that wants to submit in the afternoon and have results in the morning. It is also the step that puts the most weight on the job description.

This article covers why unattended is the default, what the job is designed to be given while it runs, and what remains open about a job that does not go as planned.

Why unattended is the default

Batch and asynchronous work is, by definition, work that does not need a person present. A training run does not need anyone to watch the loss go down. A processing pass over forty hours of warehouse camera footage does not need someone to confirm each clip. A sweep of ten thousand simulations is finished when the last one is, not when someone checks.

Designing the run step around this fact simplifies the network. There is no session to keep alive, no interactive terminal to hold open, and no state that lives in the submitter's connection. The job is handed everything it needs at submission and left alone. If that sounds like a constraint, it is the same constraint that makes a job's cost predictable: the compute is held for the work, not for someone's attention.

Isolation while running

While a job runs, the design intent is that it is kept apart from other jobs sharing the pool. Compute held for one job is not shared with another job during the hours it is held. One job's work is not meant to be visible to or affected by another's.

The preview states this as design intent and does not publish the mechanism. How isolation is enforced, what it covers, and how it is verified are details the program has not released, and nothing in this article should be read as a security claim. The commitment is to the shape of the design: a job gets its compute, does its work apart from everyone else's, and gives the compute back.

What a job carries with it

Because nobody is watching, a job has to be self-sufficient. In practice that means the four declarations covered in what a job declares are not administrative detail but the job's survival kit.

The data must be staged before the run begins, because a job cannot pause mid-run and ask the submitter for a file. The request must include a limit, because a job cannot ask whether it is allowed to keep going. The output must be declared, because a job cannot ask whether what it produced is what was wanted. The design intent is that once the run step begins, everything the job needs is already present on the compute.

This is also why a task that needs a human decision partway through does not fit. Such a task should be cut at the decision point, so that the part before it and the part after it are each a job that can run on its own.

When the run does not go to plan

The hard case is a job that stops before it finishes. A run that fails on bad input. A training run that diverges. A job that hits the limit in its request with work still to do. The design intent is that the compute is released whenever the job stops, whether it stopped because it finished or because it could not continue, and that the meter stops with it.

What comes back from a job that stopped early, and how the hours held by a failed job are accounted for, are open questions the program has not published answers to. A job that writes intermediate output may have something to return; a job that does not may return only a record that it ran and stopped. The article on what happens when a job fails goes through the cases.

For the overall sequence, how it works shows where the run step sits, and the examples page describes four illustrative jobs that are each designed to run start to finish without supervision.