Inference compute changes AI benchmark scores. UK AISI and EvalEval released verified results showing how token budgets and retries shift results.
Inference compute is the quiet variable behind many AI benchmark scores. Two labs can run the same model on the same test and report different numbers because one allowed a few thousand tokens of reasoning and the other allowed a few hundred thousand, or because one let the model retry after a failure. On September 22, 2026, the UK AI Security Institute (AISI) and the EvalEval Coalition published verified evaluation results meant to make that dependence visible.
The release rests on a paper, How Inference Compute Shapes Frontier LLM Evaluation, whose authors are Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei and Cozmin Ududec. This article explains what they varied, what the abstract says they found, and what the numbers imply for anyone reading a leaderboard. Where the sources give no figures, I say so.
The problem with a single budget
Early benchmarks were short. A model read a question and produced an answer in a few hundred tokens, so the test measured what it knew. Current evaluations ask for longer, tool-using, iterative work: fixing a repository bug, proving a theorem, breaking into a capture-the-flag server. On such tasks, the score depends on how long the model is allowed to keep working.
The paper’s central claim, as stated in its abstract, is that many evaluations still apply one restrictive budget, so a low score may reflect the setup, not the model. For safety work this matters in both directions. A cyber evaluation run at a small budget may report that a model cannot do something that it can do, given more room.
What the authors varied
According to the arXiv abstract (version 3, dated July 16, 2026, first submitted June 16), the team evaluated up to 12 frontier language models on seven hard benchmarks spanning software engineering, mathematics, medicine and cybersecurity. They applied three simple interventions, chosen because they are things any evaluator can do.
The AISI release page names five benchmarks from the paper’s main experiment: HealthBench, FrontierMath, Humanity’s Last Exam (HLE), SWE-Bench Pro and Terminal-Bench 2.0. It also names six models: Claude Opus 4, Opus 4.5, Opus 4.6, GPT-5, GPT-5.2 and GPT-5.4. Two cyber evaluations, Cyber CTFs and The Last Ones, use a different and partly overlapping model set. That is five of the seven benchmarks and six of up to 12 models, so the public release covers part of the study, not all of it.
- Larger token budgets: let the model reason and act for longer before stopping.
- Context compaction: summarize earlier work so a long run fits in the context window.
- Repeated submission attempts, guided either by the model itself or by minimal correctness feedback.
What they found
The abstract gives three findings and no percentages. First, larger budgets substantially improve performance across several domains, including cybersecurity, FrontierMath, HLE and Terminal-Bench. Second, fixed-budget evaluations can increasingly understate newer models, which do better at large budgets because they can attempt and reliably solve harder tasks. Third, the interventions do not help equally everywhere: repeated submission broadly helps, but the benefit of bigger budgets, external feedback and parallel attempts varies by benchmark.
The second finding deserves attention. It implies that gaps between model generations are partly artifacts of the budget. An older model may saturate at a modest number of tokens, while a newer one keeps improving past that point. A test that stops early compares them where the newer model has not yet had room to use its advantage, and so understates the difference. I could not read the full text, so the size of this effect is not available to me, and the abstract does not quantify it.
Working through the retry arithmetic
Repeated attempts are the easiest intervention to model, and the arithmetic shows why the type of retry matters. The following is an illustration with invented numbers, not data from the paper.
Suppose a model solves a task with probability 0.2 on a single attempt, and attempts are independent. After five blind attempts, the chance that at least one succeeds is 1 minus 0.8 to the fifth power, which is 1 minus 0.328, or about 67 percent. Ten attempts give about 89 percent. But that figure counts a success only if someone can recognize it. If nothing tells the evaluator which attempt was right, a scoring rule that takes the final or a random answer still returns about 20 percent.
This is where feedback enters. The paper tests retries guided by the model’s own judgment and retries guided by minimal correctness feedback, such as a bare signal that the last answer was wrong. With the second kind, the harness knows which attempt to keep, and the pass-at-k curve is what counts. The AISI post’s HLE figure shows this directly: it plots the cumulative share of attempted tasks solved against tokens used, counting each task’s earliest observed success, and per its caption models given correctness feedback after each attempt kept solving additional tasks as tokens rose.
The caution is what that curve means. A model that solves a problem on the fifth try with an oracle telling it the first four were wrong is not the same as a model that solves it in a single unaided pass. Both are capability, but of different kinds, and a safety assessment should specify which one it is measuring.
Why reproducibility is the real topic
The Hugging Face post that accompanies the release is about a plainer problem. Evaluation results are reported in many formats and places, often without enough detail to rerun, and rerunning can cost too much. The EvalEval Coalition and AISI began working on this at a joint workshop alongside NeurIPS 2025, and AISI’s feedback shaped the Every Eval Ever (EEE) schema, a shared format for reporting results.
AISI is now publishing verified results, context and configuration through EvalEval’s platform, along with the data behind the paper. A related platform, Evaluation Cards, joins benchmark metadata, run data and model metadata into one record per result. The post also names two other AISI projects: OptStop, which aims to make evaluation more efficient (arXiv 2608.14425), and HiBayES, which uses hierarchical Bayesian modelling for more rigorous statistics (arXiv 2505.05602). I did not open either paper.
Same model, different published score
One chart in the post puts AISI’s Terminal-Bench 2.0 results next to other reported evaluations of the same models under different setups. The post says the point is that setup choices can change the reported score; it states no numbers in the text, and the chart is interactive, so I cannot quote values from it.
Think about what goes into one Terminal-Bench number. The token limit, the number of attempts, the scaffold that wraps the model, the timeout on each command, whether failures are retried and how the final answer is chosen all vary. Each is a legitimate choice. Without a record of them, a score is a point on an unlabeled curve. The proposal in the paper is to report the whole curve: capability as a function of inference-time compute, with the protocol written down.
What this means for reading leaderboards
A reader of a model announcement can ask four things. What was the token budget, and was it the same for every model compared? Was the score from a single attempt or the best of several? If several, did anything tell the harness which was correct? And was context compaction used on long runs? Announcements seldom answer all four, which is why the AISI-EvalEval effort puts them into a schema.
There is a policy angle as well. The authors recommend comparing model generations over a large shared compute range at matched budgets, especially in safety- or policy-relevant settings. If regulators or developers set thresholds for dangerous capability, a threshold measured at a low budget may be crossed later on paper than it is in practice. A fixed-budget evaluation tells you what the model does under that budget and nothing about what it does under a larger one.
Limits of what is known so far
Three limits apply. The abstract reports no effect sizes, and I could not retrieve the paper’s full text, so how much any intervention helps on a specific benchmark is not something I can state. The AISI post says it shares material where appropriate, so not all of its evaluation work is public. And the post describes the verified results as reference points, not a definitive ranking.
Also note the tone: the release is written by the organizations that built the infrastructure it advertises, and it asks developers and evaluators to adopt it. That does not make it wrong, but the call to adopt EEE is partly a pitch.
Frequently asked questions
What is inference compute in AI?
It is the computation a model uses when answering, as opposed to training. In practice it shows up as how many tokens the model may generate while reasoning, how many tool calls it can make and how many attempts it gets.
Why do benchmark scores change with token budgets?
Hard tasks need multi-step work. A small budget cuts the model off before it finishes, so the score reflects the budget as well as the model. The AISI paper reports that larger budgets substantially improve results on several benchmarks.
Does retrying a task count as cheating?
Not inherently, but it measures something different. Retries with a signal that the last attempt failed test a different capability from a single unaided answer, so an evaluation should state which it used.
Which benchmarks and models does the AISI release cover?
Five benchmarks (HealthBench, FrontierMath, HLE, SWE-Bench Pro, Terminal-Bench 2.0) and six models (Claude Opus 4, 4.5 and 4.6; GPT-5, 5.2 and 5.4), plus two cyber evaluations using a partly different model set.
The tidy way to summarize this work is that benchmarks need error bars on a variable nobody printed. A better summary is that the number in a model announcement is an answer to a question the announcement rarely states. Until budgets and retry rules travel with the score, comparisons across labs, and across model generations, carry an asterisk. The useful follow-up is the full paper’s per-benchmark curves, which will show whether the understatement of newer models is a modest correction or a large one.
Sources
