Swapping coding-agent harnesses moves SWE-bench scores about as much as a rerun, but costs can triple, study finds

Changing the harness around a coding model shifts pass rates about as much as running the same setup twice, but can raise the cost of a single task up to threefold, according to a study by Liu and Han of Shandong University. The paper, titled "What does a harness buy? Mostly tokens.", was reported by The Decoder on Oct 24, 2026.

A harness is the system prompt, tools and context management that let a language model work inside a code repository.

Test setup

The researchers ran five models through three harnesses in their shipped versions: Claude Code, OpenCode and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and has no context management. Claude Code and OpenCode add specialized file and search tools, long system prompts and automatic history compression.

The models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash and HY4-Preview. Claude Opus 5 ran as a reference only in Claude Code. Tasks were the 45 hardest from SWE-bench Verified plus a pool of 447 more. Several setups ran twice to measure noise, and all final runs were offline.

In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix from GitHub. Claude models almost never did. The authors discarded those results and do not claim any maker trained on the test data.

Findings on accuracy

• Pure reruns of the same setup vary by up to six points on SWE-bench Verified. Earlier studies reporting gaps above 20 points mostly relied on a single run, the authors say.
• On the 447-task pool, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models. Within a five-point band they are statistically equivalent.
• On the 45 hardest tasks, harnesses differ by two to five solved tasks per model, while a rerun shifts the count by up to three.
• The median outcome flips on 13 percent of tasks between two reruns and 13 percent between two harnesses. Swapping models raises that to 22 percent.
• With 45 tasks, even a 13-point gap is detected only half the time. With 447 tasks, gaps of about five points become reliably detectable.

Zhipu AI advertises its own harness with a 2.4-point lead over Claude Code. The authors say that margin falls below both detection thresholds.

The only measurable harness effect is a deficit for OpenCode, which trails the other two by up to nine points on the large pool. About half of that gap traces to an output limit: when a model exceeds it mid-step, Claude Code prompts it to continue, while OpenCode ends the step and the attempt usually fails. The authors have no explanation for the rest, other than that models stop working sooner under OpenCode. They also flag that OpenCode had a much tighter time limit on locally run models, an uncontrolled variable.

Findings on cost

The three harnesses consume two to three times as many tokens as the leanest option per model. System prompt and tool descriptions are resent with every step: about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode and under 1,000 for mini-SWE-agent. Tool outputs are similar in size across harnesses.

Which factor dominates depends on the model. With GLM-5.3-Flash, a single trial costs 1.81 yuan in Claude Code versus 0.56 yuan in OpenCode. With Deepseek-V4-Flash, mini-SWE-agent needs so many extra steps that its cost roughly matches Claude Code's. Deepseek's own harness scores no better with its own model but costs the most.

Thinking is the exception

Claude Opus 5 delivered identical results after tools were removed from Claude Code. A year-old version solves a similar number of tasks at less than half the cost, because Extended Thinking was not on by default then.

Without thinking, mini-SWE-agent with Qwen3.6 drops eleven points, while Claude Code and OpenCode stay within noise. HY4-Preview also loses eleven points in Claude Code without thinking. The authors say their test grid is too small to explain why.

The study relies mainly on one benchmark. A small cross-check on Terminal-Bench 2.1 shows the same pattern. Data and analysis scripts are on GitHub.

What this means

Single-run harness comparisons on SWE-bench Verified cannot separate a real gain from noise at these sample sizes, and the study's data show that a harness change alters task-level outcomes about as often as a rerun does. For teams choosing a harness, the measured difference was in token spend, driven by system prompt size and step count, not in pass rate. The authors recommend publishing a rerun of the same setup alongside every harness comparison. Their conclusion: a harness buys "fewer ways to lose a task, and a bill, not pass rate."