Augment Code's agent matches Claude Code quality at 33% lower cost on Opus 4.7
Augment Code benchmarked its Auggie agent against Claude Code on Claude Opus 4.7, reporting a 67.4% pass rate versus 66.3% while cutting costs by 33%. The company attributes savings to a semantic context engine that reduces cache read tokens by 32% and output tokens by 37% compared to Claude Code's keyword-based retrieval.
Augment Code's agent matches Claude Code quality at 33% lower cost on Opus 4.7
Augment Code benchmarked its Auggie CLI agent against Anthropic's Claude Code on Claude Opus 4.7, reporting a 67.4% pass rate compared to Claude Code's 66.3% on Terminal Bench 2.0, while reducing total cost by 33%.
Token efficiency drives cost reduction
According to Augment Code, the savings come from reduced token usage across all categories. On Terminal Bench 2.0 using Opus 4.7, Auggie consumed 367.6 million total tokens versus Claude Code's 543.1 million — a 32% reduction. Cache read tokens dropped 32%, output tokens fell 37%, and cache write tokens decreased 29%. Total cost per benchmark run: $463.04 for Auggie versus $694.50 for Claude Code.
The company ran tests on a GCP n4-highcpu-16 VM with five attempts per task and four parallel tasks using Harbor framework with default settings.
SWE-Bench Pro results show similar pattern
On SWE-Bench Pro, Augment reports Auggie achieved a higher pass rate than Claude Code while costing 23% less per task. Total tokens: 1.65 billion for Auggie versus 2.35 billion for Claude Code, a 30% reduction. Cache reads dropped 30%, cache writes fell 17%. Total cost: $1,448.63 for Auggie versus $1,869.97 for Claude Code.
Semantic indexing versus keyword search
Augment Code attributes the efficiency gains to its Context Engine, which maintains a semantic index of codebases rather than using grep and keyword search. The company claims this approach reduces unnecessary file crawling and irrelevant code retrieval that require additional model turns.
"Most coding agents assemble context through grep and keyword search," according to Augment's blog post. "Agents burn turns crawling files, reading large spans of code, and pulling in irrelevant matches just to find the few lines that actually matter."
Model-agnostic approach and routing
Augment tested Auggie with alternative models on Terminal Bench 2.0. According to the company, Auggie with GPT-4.5 (likely referring to an internal designation) achieved 9.3% higher pass rate than the Claude Code baseline at 54% lower cost. Auggie with GPT-4.4 matched the baseline pass rate at 73% lower cost.
The company also introduced Prism, a model router that selects models per task. Augment claims Prism provides an additional 20-30% cost reduction on top of per-task efficiency gains.
Internal testing on private repositories
On an internal evaluation suite of private repositories, Augment reports Claude Code passed 62 tasks at $6.49 per passing task ($402 total), while Auggie passed 61 tasks at $3.90 per passing task ($238 total).
What this means
This benchmark represents a vendor-run comparison rather than independent third-party testing. The 1.1 percentage point quality difference falls within normal benchmark variance, making cost the primary differentiator. If verified independently, the token efficiency gains would be significant for organizations running coding agents at scale. However, the results reflect Augment's optimized retrieval system paired with the same underlying model, not a fundamentally different AI capability. The model routing approach could provide additional savings, though switching between models mid-task introduces complexity that may affect consistency.
Related Articles
GPT-6.2 Astra and Sol cut answer length 34 to 40 percent but fall sharply on Admits Uncertainty
An Artificial Analysis evaluation finds OpenAI's GPT-6.2 Astra and Sol produce answers 34 to 40 percent shorter than their predecessors and lead with the answer more often. Both score lower on uncertainty and transparency, and trail Anthropic's Claude Fable 5.2 and Opus 5.5 on the Transparency and Organization indexes.
AWS launches SageMaker JumpStart Model Leaderboard covering more than 70 models
Amazon Web Services introduced the Amazon SageMaker JumpStart Model Leaderboard on November 18, 2026. It gives SageMaker Studio users one view of benchmark results, response performance and estimated instance pricing across more than 70 models.
Vals tests: Claude Opus 5.5 picked Minab school as military target in 22 of 32 runs, Bloomberg reports
AI models from eight companies often marked civilians, ambulances, surrendering soldiers and a school in Iran as military targets in simulated tests, Bloomberg reported, citing a report by evaluator Vals. Anthropic's models did worst, while OpenAI's were among the best but still made mistakes.
Claude Fable 5.2 paints images through Python code, using about 700,000 simulated strokes in one piece
Anthropic's Claude Fable 5.2 produces images by writing Python programs that drive a simulated paint engine, with no external image model. Anthropic reports 64.9 percent on its internal RoboDojo Sim benchmark against 29.8 percent for OpenAI's GPT-6 Astra, though a real-world robot arm test showed limits in speed and fine motor control.
Comments
Loading...