ThinkingBox is an agent reliability benchmark that checks database state over 20 runs. See why 66.5% pass@1 becomes 47.5% and what it costs per success.
An agent reliability benchmark has to answer an awkward question: if a model does the job correctly once, how often will it do it correctly again? A Microsoft and Hugging Face write-up published on October 3, 2026, takes that question seriously. The benchmark, called ThinkingBox-Bench, runs each of its 507 business workflows 20 separate times and grades only what the backend database looks like afterward.
The results are a useful corrective to leaderboard culture. Models with similar headline accuracy turn out to differ widely in how often they repeat a success, and the cheapest model per correct answer is not the cheapest per dependable one. Below, the mechanism and the arithmetic, including a few places where the sources disagree with each other.
The failure that started it
The post opens with a customer-service example. A $745 appliance was stuck in a courier exception, 15 days past its delivery date. The agent read the policy correctly, made nine well-formed tool calls, and closed the ticket as resolved. The policy required the case to be put on hold. Every step looked fine in the transcript. The database ended up in the wrong state.
That is the gap the benchmark targets. Most agent evaluations score the final message or check whether tool calls were valid. Neither catches an agent that does something syntactically perfect and semantically wrong, then reports success.
How the benchmark is built
ThinkingBox is the sandbox; ThinkingBox-Bench is the task set. According to the Hugging Face post, it contains 507 stateful workflows across five domains: retail (98 tasks), auto insurance (100), travel (104), neobank (104) and consulting (101). The accompanying arXiv paper, by Zhuochun Li and colleagues, describes the domains slightly differently (hospitality instead of travel, and internal IT or IT/HR support for the neobank and consulting groups), so treat the domain labels as approximate.
Each task starts from an identical clean backend inside an isolated MCP session, which keeps trials from contaminating one another. A simulated user holds private context and reveals it only when the agent asks. Grading is by deterministic judges that compare the side effects an attempt leaves behind with a golden end state. Any path to the correct state passes; wrong values, missing effects and unwanted extra effects fail. Of the 507 tasks, 477 are graded on state alone and 30 also have response rubrics.
The post summarizes the philosophy in three short sentences: a trajectory is a claim, database state is the evidence, and repetition is the trust test.
Three metrics, and why they differ
Every task is attempted 20 times, which allows three views of the same data. Pass@1 is the share of all attempts that succeeded, a measure of typical performance. Pass@20 is the share of tasks solved at least once, which measures what the model can do on a good day. The third, observed 20/20, counts tasks passed on every attempt, with no statistical estimator involved.
These answer different questions. A support team that can review each result might care about pass@20, since a human can retry. A system that acts unattended on refunds or insurance claims needs the 20/20 view. The arXiv abstract reports a related figure it calls pass^20, which is an estimator; for Claude Opus 5, it matches the post’s raw count at 47.53 percent, but for Kimi-K3 it gives 17.60 percent against the post’s observed 13.41 percent. The sources do not explain the gap. A plausible reading is that the estimator and the literal tally are different things, but that is an inference.
Working through the headline numbers
Start with what the leaders score. Claude Opus 5.5 leads with 67.16 percent pass@1, Opus 5 follows at 66.50, and GPT-5.4 is at 65.36. The strongest open-weight model, Kimi-K3, scores 57.37. The post also reports Qwen3.8-27B at 51.70 and DeepSeek-V4-Pro at 43.26. A gap of 0.66 points between the top two is well inside what anyone would call a tie.
Now look underneath. Kimi-K3 solves 476 of 507 tasks at least once, 93.89 percent, the best coverage in the table. Yet it passes all 20 attempts on only 68 tasks, or 13.41 percent. Claude Opus 5 solves 401 tasks at least once (79.09 percent) but passes all 20 on 241 (47.53 percent). Opus 5.5 also lands at 241 tasks. The post draws the sharp conclusion that half a point of headline accuracy bought no additional dependability at all.
A back-of-envelope check shows why this happens. If a model had a flat 66.5 percent chance on every task, passing 20 in a row would have probability 0.665 to the 20th power, about 0.03 percent. Observed, Opus 5 passes all 20 on nearly half the tasks. So success is not spread evenly; tasks sort into ones the model nearly always solves and ones it nearly always misses, with a band in between. That arithmetic is mine, not the authors’, and it assumes independent attempts. It does suggest that task difficulty and ambiguity matter more than random sampling noise.
The retention figures tell a related story. Across 20 repeats, GPT-6 Astra keeps 78 percent of its pass@1, and Opus 5.5 and Opus 5 each keep 71 percent. GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8 percent.
Why valid-looking runs fail
The failure statistics are the most instructive part. In an ablation over 12 models, the authors collected 121,680 valid trials, of which 79,853 failed the executable checks, about 65.6 percent. Of those failures, 67.24 percent ended cleanly: the agent invoked a state-changing tool and reported no final tool error. Applied to the failed pool, that is roughly 53,700 runs that looked like successes to anyone not inspecting the database.
Among those clean-ending failures, the checks found wrong field values in 77.61 percent, unintended extra effects in 43.30 percent and missing required effects in 25.36 percent. The categories overlap, so the shares exceed 100 percent together. The point is that the dominant error is not a crash. It is a plausible action with the wrong parameter.
A separate unweighted classification put the failure signature as tool usage in 79.9 percent of failed traces, wrong state updates in 10.3 percent, incomplete user resolutions in 7.0 percent, and no state-changing action in 2.9 percent. The authors caution that these signatures are not unique causal explanations.
Cost per success versus cost per dependable task
Vendors usually quote price per token. The authors compute something closer to what a business pays. Cost per successful attempt divides the estimated cost of running 507 attempts by the number of successes. Using undiscounted OpenRouter list prices from a September 20, 2026 snapshot, the cheapest per success is GPT-5.6 Sol at $0.127. GPT-5.4 adds 3.45 points of pass@1 for $0.004 more, and Opus 5.5 adds 1.80 points more for $0.276.
Then the reliability-adjusted figure: cost of the whole 20-run campaign divided by tasks that passed all 20. GPT-5.4 comes out at $6.80 per dependable task across 128 tasks, GPT-6 Astra at $7.45 across 231, Opus 5.5 at $7.80 across 241, and GPT-5.6 Sol at $9.76 across 82. Multiplying back (my arithmetic), that implies a campaign of roughly $870 for GPT-5.4 and $1,880 for Opus 5.5, which bought nearly twice as many dependable tasks.
The authors’ phrase for it is that the cheapest way to get a right answer is not the cheapest way to get a dependable one. They also warn that these are comparative estimates, not invoices, and that they price single successes rather than consistency.
- GPT-5.4: 128 tasks at 20/20, $6.80 per dependable task.
- GPT-6 Astra: 231 tasks, $7.45.
- Claude Opus 5.5: 241 tasks, $7.80.
- GPT-5.6 Sol: 82 tasks, $9.76.
Domain effects and mitigations
Averages hide domain swings. Claude Opus 4.6 scores 68.62 percent on retail but 8.30 percent on auto insurance. Averaged over the 12-model ablation, pass@1 is 59.52 percent for retail and 33.83 percent for auto insurance. A model chosen from a retail demo may not transfer to claims processing, even from the same vendor.
The authors suggest checking terminal state before committing, classifying errors so retries target recoverable ones, narrowing the tool surface, and requiring human approval for irreversible changes. They state plainly that they have not measured how much any of these helps, so they are reasonable ideas, not demonstrated fixes.
Limits of the benchmark
Every public task is a synthetic reconstruction, and the customers are not real. Operational errors go to a separate sidecar file and count as unsuccessful trials. The 20/20 count is a literal tally with no smoothing, so a task that fails once in twenty by chance counts the same as one that always fails. A community comment on the post also raised a methodological question: one task check delegates its pass condition to a database validation function, and unequal hashes fail even when a diff shows nothing. The commenter’s replay used hashes only and did not run an agent, so it does not overturn anything, but it shows the grader deserves independent inspection.
The harness runs through OpenEnv, with the dataset on Hugging Face, and needs Python 3.11 or later, uv, Docker and a pinned data release. Teams can therefore test their own agents against it, which is probably the most useful thing about the release.
Frequently asked questions
What does pass@1 mean for an AI agent?
It is the share of individual attempts that succeed. In ThinkingBox-Bench it is averaged over 20 attempts on each of 507 tasks.
Why does the benchmark grade database state instead of the final answer?
Because an agent can report success while leaving the wrong records behind. State-based grading accepts any route to the correct end state and rejects wrong, missing or extra changes.
Which model is most reliable in the results?
On pass@1, Claude Opus 5.5 leads at 67.16 percent. On tasks passed every time, Opus 5.5 and Opus 5 are tied at 241 of 507. These are the benchmark authors’ results, not independent ones.
Can I run ThinkingBox myself?
According to the post, yes: the harness is available through OpenEnv and the dataset is on Hugging Face, with Python 3.11 or later, uv, Docker and a pinned data release as requirements.
The reliability numbers do not say agents are bad. They say that a single demo, or a single pass rate, is weak evidence for letting software change records without supervision. If a task can be retried by a human, pass@20 may be the right measure and the economics look good. If it cannot, the 20/20 count is the one to read, and it is far lower than the leaderboard suggests. The practical move for anyone deploying agents is to build the same kind of check into their own system: define the correct end state, run the task many times, and look at the distribution before trusting the average.
Sources
