Microsoft's ThinkingBox: Claude Opus 5.5 passes all 20 runs on just 241 of 507 stateful agent tasks

Microsoft and Hugging Face have released ThinkingBox, a benchmark that grades AI agents on the backend state and side effects they leave behind, not on their final responses or tool calls. It covers 507 stateful business workflows, each run 20 times per model. It is available through Hugging Face and can be run via OpenEnv.

The benchmark is described in a joint blog post and an accompanying paper. All results below are the authors' own and have not been independently replicated.

How it works

Each task runs against isolated MCP tool sessions. Every attempt starts from an identical clean backend, and the grader checks the terminal database state with executable checks. The 507 tasks span five domains: retail (98), auto insurance (100), travel (104), neobank (104) and consulting (101).

Three metrics are reported:

• pass@1: share of all attempts that succeeded.
• pass@20: share of tasks solved at least once in 20 tries.
• Observed 20/20: the literal count of tasks passed on all 20 attempts, with no estimator or smoothing.

Tool calls hide failures

In a common-set ablation of 121,680 valid trials across 12 models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool and reported no final tool error.

The checks found wrong field values in 77.61% of failures, unintended extra effects in 43.30%, and missing required effects in 25.36%. These categories overlap.

The post's example is a retail agent that made nine well-formed tool calls and correctly read the refund policy. It then set a ticket to "solved" when the required end state was "hold" because a carrier exception remained open.

Results

Overall task-weighted pass@1, selected models:

Model	Overall pass@1 (%)
Claude Opus 5.5	67.16
Claude Opus 5	66.50
GPT-5.4	65.36
GPT-5.6 Sol	61.91
Claude Sonnet 4.6	59.19
GPT-6 Astra	58.31
Kimi-K3 (open-weight)	57.37
Qwen3.8-27B (open-weight)	51.70
DeepSeek-V4-Pro (open-weight)	43.26
Grok-4.3	14.38
Mistral-Large-3	4.66

Domain variance is large. Claude Opus 4.6 scores 68.62% on retail but 8.30% on auto insurance.

Consistency

GPT-6 Astra retains 78% of its pass@1 rate across 20 repeats. Claude Opus 5.5 and Claude Opus 5 each retain 71%. GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%.

Kimi-K3 has the broadest coverage. It solves 476 of 507 tasks (93.89%) at least once, with only 31 tasks defeating it entirely. It passes all 20 attempts on just 68 tasks (13.41%).

Claude Opus 5 solves 79.09% of tasks at least once, with 106 defeating it entirely. It passes all 20 attempts on 241 tasks (47.53%).

Claude Opus 5.5 beats Opus 5 on pass@1 and solves more tasks at least once. It passes the same 241 tasks on all 20 attempts.

What this means

Most agent evaluations score the transcript or the tool-call sequence. ThinkingBox's data suggests that approach misses the majority of real failures: two-thirds of failed attempts looked clean from the outside.

The Opus 5 to 5.5 comparison is the sharpest finding. A 0.66-point pass@1 gain produced no additional fully reliable tasks. Teams should treat headline accuracy and repeatability as separate selection criteria.

The Kimi-K3 versus Opus 5 split, 75 more tasks solved at least once against 173 more solved every time, matters for deployment. Pass@20 suggests what a model might do with retries and verification. Observed 20/20 suggests what it can do unsupervised on records that matter.

Caveats: the benchmark and results come from Microsoft, and the task suite is synthetic business workflows. Pricing and context-window figures are not part of this release.