Opus 5.5 and Astra solve fewer than one in ten tasks in simulated developer workplace test
A new test places AI coding agents in a simulated developer job with colleagues, project boards and incomplete handoffs. Opus 5.5 and Astra each solve fewer than one in ten tasks completely, according to The Decoder.
Opus 5.5 and Astra solve fewer than one in ten tasks completely in a new simulated workplace test for AI coding agents, according to a report by The Decoder.
How the test works
The benchmark places coding agents in a simulated developer job. The environment includes colleagues, project boards and incomplete handoffs. The agents must keep developing software on their own and reliably close open tickets in a messy daily workflow.
Results
The Decoder reports that both Opus 5.5 and Astra fall far short in this setting, each solving fewer than one in ten tasks completely. The models finish clear, standalone assignments quickly, but they lose track when working as autonomous developers.
The Decoder's headline contrasts this with isolated tests, where the agents perform well. Specific scores, pricing, context window sizes and the benchmark's name are not part of the summary available for this report.
What this means
The results separate two kinds of coding ability: finishing a clearly defined, standalone assignment, and sustaining work across a workflow with handoffs, colleagues and open tickets. According to The Decoder, Opus 5.5 and Astra do the first quickly but lose track in the second. For teams evaluating coding agents, isolated-task performance and autonomous-developer performance are measured by different tests, and in this simulation the gap between them is large.
Related Articles
METR adds open-ended tasks to its AI time horizon plot; models fall short on judgment-heavy work
AI evaluation nonprofit METR has updated its time horizon plot with longer and more open-ended tasks. The report finds some capabilities still improving exponentially, but models fall short of humans on judgment-heavy tasks such as designing AI evaluations.
GPT-6.2 Astra and Sol cut answer length 34 to 40 percent but fall sharply on Admits Uncertainty
An Artificial Analysis evaluation finds OpenAI's GPT-6.2 Astra and Sol produce answers 34 to 40 percent shorter than their predecessors and lead with the answer more often. Both score lower on uncertainty and transparency, and trail Anthropic's Claude Fable 5.2 and Opus 5.5 on the Transparency and Organization indexes.
Tencent Yuanbao and KCL release GroupAssistBench: 2,395 questions from 80,299 group chat messages
Tencent Yuanbao and the King's College London NLP team released GroupAssistBench, an agent memory benchmark of 2,395 questions built from 50 real group chats and 80,299 messages. Across 14 configurations, better recall did not guarantee better service: EverMemOS scored 0.621 on L1 recall but 0.247 on L2 open-ended service.
GPT-6.2 Astra scores 41.2% on XDOF's Ember-0 robot benchmark, up from 19.6% for GPT-6 Astra
Robotics infrastructure company XDOF reports that GPT-6.2 Astra reached a 41.2% average success rate across the 16 tasks of its Ember-0 benchmark, versus 19.6% for GPT-6 Astra. It is the first model to complete rope knot-tying and T-shirt unfolding on the benchmark. Average prediction latency fell from 15 seconds to 10.1 seconds.
Comments
Loading...