CloudOpsBench: Claude Fable 5.1 solves 60% of AWS DevOps tasks, GPT-6 Astra 53%
CloudOpsBench is a new open-source benchmark of 45 AWS DevOps tasks. Claude Fable 5.1 solved 60 percent and GPT-6 Astra 53 percent, and no model completed every task. The research team says reasoning, not knowledge, is the main bottleneck.
CloudOpsBench, a new open-source benchmark for AI agents performing cloud operations, shows that even the strongest tested models fail many routine DevOps tasks. Claude Fable 5.1 solved 60 percent of tasks and GPT-6 Astra solved 53 percent. No model completed every task.
The benchmark was reported by The Decoder on November 4, 2026.
What the benchmark covers
CloudOpsBench measures how well AI agents handle typical DevOps work in the cloud. Tasks include:
- Fixing services
- Removing resources
- Restricting access rights
- Rolling out updates
- Decommissioning projects
Version 1.0 contains 45 tasks on AWS. The tasks were built by experienced DevOps engineers. Every task runs identically in the real AWS cloud and in an emulator.
Results
| Model | Share of tasks solved |
|---|---|
| Claude Fable 5.1 | 60% |
| GPT-6 Astra | 53% |
No model solved all tasks. According to the team, led by Ang Chen at the University of Michigan, the main problem is not a lack of knowledge but reasoning.
Why agents fail
The team's failure analysis attributes about 80 percent of failures to three behaviors:
- Agents misjudge the scope of a task.
- Agents do not check all the relevant services.
- Agents stop too early.
Roughly 20 percent of errors are wrong actions. According to the researchers, some of these have irreversible consequences for the infrastructure.
Availability
The project is on GitHub under the Apache 2.0 license, and a detailed failure analysis is also available. Extensions for GCP, Azure, and OCI are planned. Pricing, context windows, and parameter counts for the tested models are not part of the reported results.
What this means
The results put numbers on where agents break down in infrastructure work. The researchers locate the problem in scope judgment and incomplete checking, not in missing cloud knowledge. Those are failures of process, and they are separate from failures to recall AWS details.
The roughly 20 percent of errors that are wrong actions matter most in operations. The researchers say some of these are irreversible, and deleting resources or changing access rights leaves little room for retries.
Running identical tasks in real AWS and in an emulator gives teams a way to compare agent behavior in both environments. The Apache 2.0 license allows others to run, inspect, and extend the benchmark. The planned GCP, Azure, and OCI extensions would widen the 45-task AWS set in version 1.0.
The source is The Decoder, reporting on the CloudOpsBench project.
Related Articles
Google releases 29 public tasks from 85-task engineering benchmark PLE and opens agent competition
Google AI & Infrastructure announced on November 3, 2026 that it is releasing 29 public tasks, grading tools and an evaluation container for Physical-Engineering's Last Exam (PLE). The 85-task benchmark was developed with Mercor and comes with an open agent competition and leaderboard.
OpenAI publishes LifeVal, an evaluation built partly from a 3,995-respondent survey to test AI on everyday tasks
OpenAI published LifeVal on November 2, 2026, an evaluation built partly from a 3,995-respondent survey. It tracks AI model performance over time on practical tasks such as finding childcare, creating budgets and ordering groceries.
Gemini 4 Carbon ranks second on SemiAnalysis GDPx at just over $16 per task, behind Claude Opus 5.5
SemiAnalysis's new GDPx benchmark places Google's Gemini 4 Carbon just behind Anthropic's Claude Opus 5.5, at just over $16 per task against about $51. Removing the cybersecurity, biology and alignment domains reshuffles the leaderboard.
SemiAnalysis launches GDPx agentic benchmark across 11 domains; best model, Opus 5.5, scores 12%
SemiAnalysis has launched GDPx, an agentic benchmark spanning 11 high-value domains including chip design, cybersecurity and recursive self-improvement. The best model scores 12%. SemiAnalysis says safety classifiers depressed the overall scores of Opus 5.5 and GPT-6 Astra.
Comments
Loading...