What we have not yet published about retention

A resources library is supposed to answer questions. This article exists to be precise about one it cannot answer yet. E31 Network has not published how long the data staged for a job is retained after the job ends, how long returned output is kept on the network side, or how either is deleted. Those specifics are not yet published, and no page on this site should be read as implying that they are.

Writing that down is more useful than leaving the gap for a team to discover. What follows is what has been published, what has not, and how a team might plan in the meantime.

What has been published

The design of data and output on the network has a stated shape, and that shape is public.

Data is staged for the job. A job declares what it reads, and that data is prepared for the job before compute is held. The job runs isolated from other jobs; while it runs, it reads its own staged data on its own held compute and no other job crosses that boundary. The output is returned to the submitter, not pooled or shared. The compute is released back to the pool. What was held is accounted for, in a record meant to be plain enough that an auditor could read it.

Each of those sentences is a design intent, and together they describe the life of a job from the point of view of its data. They say what happens to data while a job exists. They do not say what happens to it after.

What has not been published

Here is the list, stated as plainly as the published list of unsolved problems.

How long staged data is retained after the job's compute is released is not published. Whether it is deleted at release, kept for some period, or handled some other way has not been stated.

How long returned output is kept anywhere other than with the submitter is not published. The output is returned to the team; whether a copy persists on the network side after that, and for how long, has not been stated.

How deletion works, who initiates it, and how a team would confirm it has not been published.

Whether any of this differs by kind of data, kind of job, or kind of output has not been published.

None of these gaps is being filled by implication elsewhere on the site. If another article seems to assume an answer, the assumption is wrong and this article is the correction.

Why the gap is named rather than papered over

The program could have written a plausible retention policy for the preview and revised it later. It chose not to, and the reason is the same reason the site carries a list of what it has not solved.

A technical preview earns attention by being candid about what is unfinished. A team deciding whether to describe a real job to a program like this one needs to know which parts of the design are settled and which are not, so it can judge for itself whether its data belongs in a job yet. A retention policy that was invented to look complete would undermine that judgment.

There is also a connection to the open problem of moving large datasets. How data is staged for a job, and how large stages are handled, is not settled. Retention sits downstream of staging. It would be strange to publish precise rules about how long staged data is kept before it is clear how the staging itself works.

And there is the matter of claims. The program makes no certification or compliance claims for the preview. A detailed retention policy, published early, would invite readings as exactly that kind of claim. Stating the gap avoids promising something the preview cannot yet stand behind.

How a team can plan around it

Unpublished is not the same as unknowable, and a team can make sensible choices now.

Declare only the data a job reads. The less that is staged, the less whose retention is in question. This is good practice for staging in any case, and the article on staging data for a job makes the argument on its own terms.

Keep the team's own copies. The design returns the output to the submitter; treat that returned copy as the authoritative one, and keep the original data on the team's own systems as it already is. Nothing in the design asks a team to rely on the network as a store.

Read the record. When a job ends, what was held is accounted for. The article on what is accounted for when a job ends describes what that record is being designed to contain. It is the place where a team would see what was staged and what was returned, which is the starting point for any later question about retention.

When retention and deletion specifics are published, this article will be out of date, which is the intended outcome. Until then it stands as the plain statement that they are not. The about page explains why the preview is run this way.