Microsoft's ThinkingBox-Training lifts Qwen3.8-27B fully reliable task completions from 38 to 107 of 507

Microsoft and Hugging Face have released ThinkingBox-Training on GitHub, a post-training pipeline that uses executable database-state checks as reinforcement learning rewards. According to the authors, 50 GRPO steps raised Qwen3.8-27B's pass@1 on the 507-task ThinkingBox-Bench from 51.70% to 60.78%. Tasks solved on all 20 attempts rose from 38 to 107. The results are self-reported by Microsoft and have not been independently replicated.

How it works

ThinkingBox-Bench gives an agent a real backend, MCP tools, and a simulated LLM user, then checks the terminal database state rather than the specific tool calls. For most tasks the check returns 1 or 0, which the team reuses directly as a reward.

The training pool was built from ThinkingBox-format tasks outside the benchmark. The authors filter tasks by what they call offline policy-relative difficulty. Tasks the base model always solves or never solves teach it little, so the team estimated each task's success probability once with the base policy and kept the selected pool fixed for the whole run. Pool size and the full reward design are not covered in the portion of the post reviewed here.

Results for Qwen3.8-27B

Evaluation covers 507 tasks at 20 attempts each, or 10,140 attempts in total.

Metric	Base	After 50 GRPO steps	Change
Observed 20/20	38 (7.50%)	107 (21.10%)	+69 tasks
pass@1	5,242 (51.70%)	6,163 (60.78%)	+9.08 pt
pass@20	453 (89.35%)	446 (87.97%)	−7 tasks

The benchmark's domains, databases, and most MCP tools were not seen during RL training, so the authors treat it as an out-of-distribution test. Pass@1 gains by domain:

• Retail: 64.03 → 67.91
• Auto insurance: 47.85 → 56.45
• Travel: 53.41 → 62.69
• Neobank: 47.88 → 62.40
• Consulting: 45.69 → 54.51

Neobank gained the most (+14.52 points) and retail the least (+3.88).

In one corporate hotel-booking task, policy requires manager approval before a room downgrade. The base model changed the booking and issued refunds without approval in 15 of 20 attempts. The fine-tuned model preserved booking and payment state in all 20, and all expected records matched in 19.

Comparison with other models

Observed 20/20 counts from the authors' earlier evaluation:

Model	pass@1	Observed 20/20
Claude Opus 5.5	67.16	241
Claude Opus 5	66.50	241
GPT-5.4	65.36	128
GPT-5.6 Sol	61.91	82
Qwen3.8-27B RLFT	60.78	107
Claude Sonnet 4.6	59.19	102
GPT-6 Astra	58.31	231
Kimi-K3	57.37	68

The fine-tuned 27B model trails GPT-5.6 Sol on pass@1 but passes more tasks on every attempt (107 vs. 82). It also passes more than Kimi-K3 (68). It remains well behind Claude Opus 5.5 and Opus 5 (241 each) and GPT-6 Astra (231).

The earlier post found that Claude Opus 5.5 beat Opus 5 by 0.66 points on pass@1 yet passed exactly the same 241 tasks on all 20 attempts.

The trade-off

Coverage fell slightly: seven fewer tasks were solved at least once. The authors describe this as consistent with sharpening, where repeatable success rises while the breadth of at-least-once solutions narrows. They say this matches prior work.

What this means

The useful idea is not the 50-step run but the reward source: if an environment can verify its end state programmatically, the benchmark grader can also train the agent. That makes state-verified agent environments reusable for both evaluation and RL, and it rewards correct outcomes rather than imitation of a reference tool-call path.

The results come with caveats. They cover a single model, a single run, and a benchmark built by the same team that built the training method. The pass@20 decline suggests the gains come partly from narrowing, not from new capability. Even so, a 27B model nearly tripling its fully reliable completions on out-of-distribution domains is a strong signal that dependability, not just average accuracy, can be trained directly. Teams evaluating agents for production database work should track Observed 20/20 alongside pass@1.