Co-Design Bench: best coding agent fully solves only 26.4% of real user change requests

Even the best coding agent fully completes only 26.4 percent of tasks in Co-Design Bench, a new benchmark from researchers at Alibaba and several Asian universities. According to the paper, GPT-6 Astra, the top performer among ten models tested, meets about 70 percent of a task's requirements on average. In most cases the agent does the bulk of the work but leaves at least one detail unfinished, so a human usually has to clean up by hand. The report comes from The Decoder.

What Co-Design Bench tests

Agents must rework existing web apps based on user requests, sometimes using images, audio recordings, videos or documents as input. The tasks come from real coding sessions. The researchers favored cases where users had to rephrase a request multiple times or roll back changes, so the set leans toward tasks that already gave AI agents trouble in practice.

Tasks range from small tweaks, such as pulling an amount from a receipt and updating every dependent total, to full-scale project overhauls. In the multimodal track, agents extract values from receipts, tables or videos and update all dependent views consistently.

Grading is strict. One AI judge tests the finished app in a browser. A second judge re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app.

Why the authors built it

The authors point to saturation of existing benchmarks. When models can run, check and fix their own code, they score above 99 percent on those tests, and differences between models become hard to measure. On older benchmarks such as WebCompass-edit, agentic coding models reach near-perfect scores, while Co-Design Bench leaves a gap of more than 35 percentage points.

Results

• Ranking: GPT-6 Astra is first. Claude Opus 5 and Qwen3.8-Max take the next spots. Grok 4.6 is last.
• Track results: GPT-6 Astra leads both tracks but fully solves 41 percent of multimodal tasks and just under 18 percent of repository tasks.
• Larger overhauls: The much smaller Qwen3.8-27B climbs to second, though the authors do not derive a general advantage for the model from this.
• Task complexity: All ten models score worse once a task requires three or four different types of changes, including layout, design, interaction and content.
• Compute: GLM-5.2 uses nearly six times as many tokens as GPT-6 Astra on smaller changes and still scores significantly worse.

The harness matters

In a separate test, the researchers ran the same models in different harnesses, the programs through which an agent reads, writes and executes code. Qwen3.8-27B scored 75 or just 56 percent on larger overhauls depending on the environment. Which of two models came out ahead also depended on the environment. The authors recommend always reporting model and working environment together.

Limits and context

The benchmark evaluates one fixed request per task. Follow-up questions and mid-task corrections from users are not covered, and the AI judges can make mistakes.

The Decoder cites earlier work from the research organization METR, which found that the acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on SWE-bench Verified. Flaws in test environments can also let agents pass tasks without solving them.

The harness effect is not clear-cut. A recent study on coding agents found barely any measurable benefit from more elaborate harnesses, even though they consumed up to three times as many tokens.

What this means

A 26.4 percent full-correctness rate for the top model, against a roughly 70 percent average on requirement coverage, shows that partial completion hides much of the remaining work. Because tasks with several intertwined change types hurt every model tested, and such combinations are common in real projects, single-change tests give an incomplete picture of agent reliability.

The harness results show that a model's score depends on the environment it runs in. A leaderboard that lists only the model therefore tells part of the story, which is why the authors recommend that teams selecting a coding agent test the model and harness as a pair. The benchmark's single-request design and reliance on AI judges are stated limits on how far its numbers generalize.