GPU cluster scheduling at Ai2: budgets and fair share cut p90 debug queue waits from 2 hours to 30 seconds. Here is how the numbers work.
GPU cluster scheduling sounds like plumbing until you notice that the plumbing decides how fast a research lab can learn anything. On October 9, 2026, the Allen Institute for AI (Ai2) published an account of how its infrastructure team replaced a priority-based scheduler with budgets and a hierarchical fair-share system, and the headline figure is hard to ignore: the 90th-percentile queue wait for debugging jobs fell from about two hours to about 30 seconds.
The post is useful beyond Ai2 because it is specific. It names the failure modes of the old system, describes the scheduling contract that replaced it, and reports where the new design made things worse. This article walks through the mechanism, checks the arithmetic, and flags what the numbers can and cannot support.
What Ai2 was dealing with
According to the post, Ai2’s AI Infrastructure team manages thousands of NVIDIA H100, B200 and B300 GPUs, split across clusters of between 88 and 1,024 GPUs, for roughly 150 internal researchers. The constraint is not the hardware count. It is that outstanding requests run at two to three times the number of GPUs available, so every GPU hour has two or three competing claimants.
A queue under that kind of pressure is a market with no prices. The old scheduler used priority levels, plus an option to mark a workload non-preemptible so it could not be kicked off its GPUs. Both knobs were free to use, and people used them.
Three ways the old scheduler broke
The team describes three pathologies. The first is GPU squatting: researchers who needed fast access for debugging would park a no-op workload on a node so the hardware would still be theirs when the real job was ready. The second is priority inflation. When a high priority is free and a low one means waiting, rational users drift upward until, as the post reports, all scheduled workloads were using the HIGH level. A priority scale on which everyone sits at the top carries no information.
The third is operational. Non-preemptible workloads running on a host with a hardware fault cannot simply be moved. On-call engineers ended up spending most of their ticket time negotiating with owners to shut jobs down so a machine could be repaired.
The post frames this as a tragedy of the commons and points to a 2011 paper on Dominant Resource Fairness that describes users adding infinite loops to inflate their measured usage. That citation is the post’s own; the paper was not opened for this article.
Budgets instead of priorities
The central change is conceptual. Instead of letting users declare how important a job is, managers allocate proportional shares of GPU time through a hierarchy that mirrors the research organization. Lead researchers divide a project’s share, principal investigators divide a program’s share, and program managers (or the CEO, for cross-program questions) divide the whole. A project might hold, in the post’s example, a 35 percent claim on total capacity.
Every request must be funded from some budget, or it runs unprotected. Allocated occupancy is charged against the budget and is protected from preemption for the job’s declared minimum runtime. Unallocated occupancy costs nothing, but any funded request can preempt it at once.
The stated design goal is worth quoting because it explains the whole system: make gaming the scheduler more expensive than honestly arguing for a larger budget. Squatting now burns the squatter’s own allocation, so it stops being clever.
The scheduling contract and the fair-share clock
Two ideas make the budgets enforceable. The first is a contract each workload signs when submitted: a minimum runtime that is protected from preemption, and a flag for whether the job is resumable. Setting the minimum runtime to zero makes the work unallocated. Ai2 capped the maximum value at 8 hours, so nobody can hold a protected claim for a week.
The second is the fair-share calculation. The scheduler tracks how much each node of the budget tree has actually occupied over a sliding lookback window, which defaults to 7 days, and gives priority to the nodes that are furthest below their share. The post notes the lineage honestly: the Hadoop Fair Scheduler, SLURM’s Fair Tree and the YARN Fair Scheduler all work along similar lines. What is new, they say, is that the tree follows research program structure and the weights are budgets that managers set deliberately.
A side effect helps the on-call rotation. Because every protected job has a bounded minimum runtime, an unhealthy host can be marked to drain and will empty on its own as those runtimes expire. Nobody has to be talked into stopping anything.
- Allocated time: charged to a budget, protected until the minimum runtime ends.
- Unallocated time: free, resumable work, preemptible by any funded request.
- Lookback window: 7 days by default, chosen with the help of a simulator.
- Maximum minimum runtime: 8 hours.
Working through the numbers
Start with what the team reports for a 30-day measurement period after rollout began at the end of July (the post gives no year, though it was published in 2026). Teams received 98 percent of the GPU hours they were owed. Thirteen of 15 team allocations received at least 95 percent, and the worst case received 90 percent. Cluster occupancy stayed at 98 percent, the same as before the change, while demand remained at two to three times capacity.
That last pair of facts is the important one. Fair share did not buy its gains by raising utilization; the clusters were already nearly full. It redistributed who waited. Also, 18 percent of delivered GPU time was unallocated. Combining the two figures gives a rough picture (this is arithmetic on the post’s numbers, not a figure the post states): about 80 percent of capacity was running protected, budgeted work, and about 18 percent was running free work that could be bumped at any moment.
Now the latency results. For debugging workloads, the p90 queue wait fell from 2 hours (7,200 seconds) to 30 seconds, a 240-fold improvement. The team’s simulator had predicted 6 hours falling to 5 minutes, so reality beat the model, though the post cautions that the debug sample is small and the variance higher. On the largest H100 cluster, median queue wait dropped from 5 minutes to 24 seconds, about 12.5 times faster, while p90 moved from 2.8 hours to 1.8 hours, a reduction of roughly 36 percent.
Why would the median and tail respond so differently? Debug jobs are short and small, so they slot into the free, preemptible capacity almost immediately. The tail on the big cluster is probably dominated by large jobs that need many GPUs at once, and those are the ones a fair-share scheduler cannot hurry. The post does not state this explanation; it is an inference from the shape of the numbers.
Repairs that needed a human in the loop fell by 74 percent. One researcher quoted in the post, Chris Clark, said the new scheduler makes it feel like the lab has an extra 30 percent of compute. Treat that as an impression from a user, not a measurement.
Why simulation mattered
Scheduler parameters interact in ways that are hard to reason about on paper. A longer lookback window rewards steady users and punishes bursty ones; a longer maximum minimum runtime protects long training jobs but makes the free pool smaller. Ai2 built a simulator and replayed historical data through candidate configurations before touching production.
There was a catch. The historical records contained too few debug-style jobs, because the old system had taught people to avoid submitting them. The team had to construct synthetic debug workloads by hand. That is a quiet but real limitation: a simulator fed with behavior shaped by the old rules cannot fully predict behavior under new rules. The gap between predicted and observed p90 wait times is a small demonstration.
What got worse
The post is candid about costs. Interactive sessions suffered. Researchers used to hold sessions for up to a week; now they face the 8-hour protection cap and can be preempted, losing in-memory state. Planned fixes include a CPU-only cluster for data preparation and restorable sessions.
Capacity fragmentation is the larger open question. A scheduler that can no longer interrupt many small jobs at once may make large jobs wait longer for a contiguous block of GPUs. The team says it is studying this with the simulator and production measurements and has no verdict yet. The rollout itself was also bumpy: clusters were migrated one at a time, so behavior differed between them, and old vocabulary such as the word priority kept misleading people. Live explanation sessions worked better than written documentation.
What other teams can take from it
Few readers run 1,000-GPU clusters, but the lesson scales down. If a shared pool has no cost to claim and no cost to hold, users will over-claim and over-hold. Moving the argument from queue behavior to budget size makes the conflict explicit and puts it with the people who can resolve it, which is a management decision, not a software one.
The reported gains also depend on circumstances that may not transfer. Ai2 had a lot of short, resumable work to fill the free pool, and a user base small enough for managers to know one another. A lab with mostly long, rigid training runs would see less benefit from preemptible filler. The data in the post comes from one organization over 30 days, with some figures resting on small samples, so it is an encouraging case study and not a benchmark.
Frequently asked questions
What is a hierarchical fair-share scheduler?
It is a scheduler that gives each node in an organizational tree (program, project, person) a weight, measures how much capacity each node has used over a recent window, and prioritizes the nodes using less than their share. Ai2 sets the weights as budgets decided by managers.
Did Ai2 buy more GPUs to get these results?
The post does not say so. It reports that occupancy stayed at 98 percent before and after the change, with demand at two to three times capacity, so the gains came from changing who waits, not from added hardware.
What is GPU squatting?
Holding GPUs with a placeholder workload so they are available when the real job is ready, usually to avoid queue delays while debugging. Under a priority scheduler it is cheap. Under Ai2’s budgets it draws from the squatter’s own allocation.
Are the 30-second and 24-second figures guaranteed?
No. They are p90 and median wait times measured over one 30-day period on Ai2’s clusters, and the post notes that the debug-job sample is limited.
The interesting part of this story is how little of it is technical. Preemption, fair-share trees and lookback windows have been around since Hadoop, and Ai2 says as much. What changed the outcome was deciding that GPU time is a budgeted resource with a visible owner, then writing a scheduler that makes the honest path cheaper than the clever one. The unresolved items, fragmentation and lost interactive sessions, are the ones to watch if other labs copy the design. Ai2’s follow-up measurements on those two would be more informative than another round of queue-time charts.
Sources
