Replit's ViBench 2.0: Top models pass every test plan in only 49-65% of full software builds

Replit released ViBench 2.0, a benchmark for long-horizon software development, and GPT-6.1 Sol leads it with a pass@1 of 65%. GPT-6 Astra follows at 60% and Opus 5.5 at 49%. Pass@1 is the share of builds in which every test plan passes. The benchmark joins the Specialized Intelligence Index (SII) from Fireworks, Replit's partner for open-weight model inference and training.

How the benchmark works

Most coding benchmarks, such as DeepSWE, give an agent a defined task inside an existing codebase. In ViBench 2.0, the agent starts with an empty project and builds a product modeled on services like Uber, GitHub or Google Docs. It writes a first version from a short spec, then adds up to eighteen features on top of its own code.

The benchmark contains seventeen products, 241 feature requests and 544 test plans. Eight products are public on GitHub in the Harbor format, and nine are held out. Results can be rerun with one command from release v2.0.6 on Replit's OpenHands-based harness.

A grading agent runs the test plans in a real browser, clicking and typing as a user would. The tests probe clean deployment on an empty database, existing data, invariants (split-fare shares must add up exactly to the ride cost), stale state, and concurrency (two return forms must never refund the same unit twice).

Grader validation

Replit uses Opus 5.5 as the grader because it made the fewest mistakes of the models tested. According to Replit, it planted 34 bugs one at a time in working builds, and Opus never passed a plan while the bug was visible. When GPT-6.1 Sol re-graded 477 of Opus's passes, 99.2% were confirmed correct. Replit says it saw no sign that Opus favors builds from Claude models.

Results

The official results come from Fireworks' runs: four builds of each of the seventeen products per model, at medium effort and temperature 1.0. Models charted: GPT-6.1 Sol, GPT-6 Astra, Opus 5.5, Grok 4.7, DeepSeek V4.1 Flash, GLM 5.3, Kimi K3, Gemini 3.8 Flash and Sonnet 5.5.

• GPT-6.1 Sol: 65% pass@1, about $12 per build.
• GPT-6 Astra: 60% pass@1, about $90 per build.
• Opus 5.5: 49% pass@1, about 70 minutes per build on average, roughly 2.5 times faster than Sol. Replit says Opus produces only about 6% fewer tokens, so the gap mostly comes from API provider speed.
• DeepSeek V4.1 Flash: passes 77% of its test plans for about $2.50 per build. No open-weight model passes a full build.

All three leading models pass more than 90% of their test plans, so most of their failed builds miss by only a few plans.

Scores across GPT generations

On the eight public products, Replit ran earlier GPT models itself, separately from the official results. Pass@1 rises from 0% for GPT-5.4 to 31% for GPT-6 Sol and 53% for GPT-6.1 Sol. Plan pass rate climbs from 42% to 95%. GPT-6.1 Sol's 53% here sits below its 65% overall. GPT-6.1 Terra, which Replit calls OpenAI's newest model at time of writing, passes 47% of builds at about half the cost of GPT-6.1 Sol.

Where top models fail

Nearly three-quarters of the top models' failed builds fail only where features intersect. Examples include a draft pull request that never shows its merge conflict, no way to comment on a word inside a table cell, and a server owner with no roles who cannot moderate anyone. In about one in 40 of the top models' builds, the product never starts on the empty database because schema code references a table before the statement that creates it. In the design-tool undo test, GPT-6.1 Sol, GPT-6 Astra and Opus 5.5 fail the plan in eight of their twelve builds.

ViBench does not grade design. Replit notes that models from the same family produce similar designs: GPT models tend toward illustrations and big headlines, while Claude models build plainer screens closer to the products they imitate.

What this means

The headline gap is between plan pass rate and full-build pass rate. The leading models pass more than 90% of test plans yet pass every plan in only 49-65% of builds, so a single missed interaction between features decides the result. Because the agent must commit to schema and routes before it sees later requests, the benchmark measures planning under uncertainty as well as code generation. ViBench 1.0 found that models score lower when extending their own code, and ViBench 2.0 extends that setting across up to eighteen features. The cost and time figures give builders a concrete tradeoff: Sol's about $12 per build against Astra's about $90 for a lower pass@1, and DeepSeek V4.1 Flash's 77% plan pass rate at about $2.50 per build, though without a fully passing build.