What happens when a job fails partway through

Some jobs do not finish. A training run diverges. A processing pass hits a corrupt file. A simulation sweep exhausts its limit with work remaining. On E31 Network, a job that stops early is still a job that ends, and the design intent is that the end looks the same at the step level: the compute is released, the meter stops, and what was held is accounted for. What is different is what comes back, and on that point the preview is honest about how much it has not yet decided.

This article goes through where a job can fail, what the design intends at each point, and what remains open.

Three places a job can stop

A job can stop before compute is held, during the run, or while the output is being returned. The three cases are not alike.

Before compute is held, a failure is cheap. The job was submitted or queued, nothing was set aside for it, and nothing was metered. A job that is rejected because its declaration is incomplete, or withdrawn by the submitter while it waits, ends with no compute ever having been held. The design intent is that nothing is owed in this case.

During the run, the job holds compute. If it stops here, the compute has been held for some number of hours and work has been partly done. This is the hard case, and the rest of this article is mostly about it.

During output, the work is finished but the result has not yet reached the submitter. The compute is still held, because the design does not release until the output is returned. A failure here is a transfer problem rather than a computation problem, and the program has named moving large data as an unsolved problem in both directions.

What the design intends at the step level

Whatever the cause, the design intent is that a job that stops releases its compute. There is no state in which a failed job keeps compute held. The compute goes back to the pool, becomes available to the next queued job, and the meter stops at that moment. The article on release describes that step for the normal case, and the intent is that the abnormal case follows the same path.

The accounting record is produced in the same way. What was held, and for how long, is recorded whether the job finished or not. This is a statement about the record existing, not about how the hours in it are treated, which is a separate and still open question.

What comes back

A job that finishes returns its declared output. A job that stops early may have nothing that matches that declaration. The design's answer is intermediate output: a job that declares, at submission, that it will produce checkpoints or partial results along the way gives the network something to return if the run stops.

This puts a burden on the submitter, and the design accepts that. A long training run that declares only its final model has nothing to show for a run that stops short. The same run that declares a checkpoint at intervals can return the most recent one, and a second job can be submitted that starts from it. The piece on what a job declares covers the declaration itself; the point here is that the declaration is the submitter's main protection against losing a long run.

Whether the network would return intermediate output automatically, or only on request, and in what form, is not yet published.

What is not yet decided

Three questions are open, and the program has chosen to publish them as open rather than answer them with policy that does not exist.

The first is billing for a failed run. The meter is designed to run while compute is held, and compute is held during a run whether it succeeds or not. How those hours are treated when the job did not complete is not published, and no rates are published for any hours at all.

The second is attribution. A run can stop because the job was wrong or because something on the network side went wrong. Distinguishing the two cases, and what follows from the distinction, is undecided.

The third is partial output. The design intends that intermediate output can be declared and returned. The mechanism for doing so, and what is retained of it afterward, is unpublished, like all retention specifics in the preview.

A team that depends on long runs should treat these as questions to raise when design-partner intake opens. For how a job is meant to behave when it does not fail, see running unattended and how it works.