What we have not solved yet
The network publishes a list of problems it has not solved and treats it as part of the product. This is the list as it stands, and why each item on it is hard.
· 4 min read
IN SHORT
E31 Network has not yet solved four problems it publishes on purpose: getting large datasets to where the compute is, scheduling jobs of very different sizes fairly, where the capacity in the pool comes from, and exactly what a job must declare to be scheduled well. Each is an open design question, and items come off the list only when they are actually solved.
E31 Network publishes a list of problems it has not solved. This is not a disclaimer tucked into a footer; it is a section of the site, maintained on purpose, and the program treats it as part of the product. A technical preview that claimed to have everything worked out would be making a claim the preview cannot support. This article goes through the list as it stands and says why each item is hard.
Getting large datasets to where the compute is
A job's data has to be staged for the job before it runs. For a small dataset this is simple. For a season of warehouse footage or a large training corpus, moving the data is itself a significant piece of work, and it has to finish before the meter has any reason to start.
The open questions are about the mechanics: how data arrives, how long staging takes relative to the job that follows it, what happens to data between related jobs, and how the cost of moving it relates to the cost of computing on it. The design intent is that staging is part of the job's life and not a separate thing the team arranges by hand, but the program has not settled how.
Scheduling mixed job sizes fairly
The pool will receive jobs of very different shapes. One job needs many GPUs for several hours. Another needs a small amount of compute for a few minutes. A third needs compute until a backlog is clear, however long that takes. Packing those together without leaving gaps, and without making small jobs wait behind large ones indefinitely or large jobs wait endlessly for a window, is a hard scheduling problem.
Fairly is the difficult word. A scheduler that maximizes pool utilization may starve some job shapes. A scheduler that treats every job equally may waste capacity. The program does not yet know which trade-offs a team would accept, and it does not want to design the scheduler in isolation from the people whose jobs it would schedule.
Where the capacity comes from
The site does not say where the compute in the pool comes from, and this is deliberate. The question is unresolved, and the program does not describe a supply arrangement it has not settled or a location it has not chosen. When there is something to say, it will be said plainly. Until then the honest answer is that this is on the unsolved list.
What a job must declare to be scheduled well
A job declares the work, the data, the request, and the output. That is enough to describe a job. It may not be enough to schedule one well. A scheduler that knows only that a job wants some compute for some hours has less to work with than one that knows how the job behaves, whether it can be paused, how its demand changes over its run, and what happens if it is given slightly less or slightly more than it asked for.
The program does not yet know exactly which of those details matter enough to require, and which would make submitting a job tedious without improving scheduling. Asking for too little makes the pool inefficient. Asking for too much makes the job description a chore. Finding the right set is something the program expects to learn from design partners rather than decide alone.
Why the list is published
A preview exists to find out what is wrong with a design before it ships. Publishing the open problems invites the people best placed to help, the teams with real jobs, to say which problems matter to them and what a solution would need to look like. It also sets expectations correctly. A team reading this site should know that no jobs can be submitted yet and that the hard parts are still hard.
The list will change. Items will be removed when they are actually solved, not when they become inconvenient, and items will be added as the program learns what else it has underestimated. For what a team would be asked to contribute toward solving them, see what a design partner would be asked to do. For the rule the program uses to decide when something moves from this list to the rest of the site, see how the preview decides what to publish.
Questions this raises
- Why publish problems you have not solved?
- Because a preview exists to find out what is wrong with a design before it ships. Publishing the open problems invites teams with real jobs to say which ones matter and what a solution would need to look like, and it tells every reader honestly that no jobs can be submitted yet.
- How does something come off the list?
- When the problem is actually solved and the solution can be described as design intent. Items do not come off because they have been there a long time or because they are awkward, and new items are added as the program learns what else it underestimated.
- Does the unsolved list mean the network might not work?
- It means the network has not shipped and the hard parts are still hard. The design intent is settled: jobs, hourly metering, queues, isolation, release. The unsolved items are about how to do each of those well at the scale the network would need.
KEEP READING