GitHub launches ReviewBench, an open benchmark for AI code review agents built on real pull requests

GitHub has launched ReviewBench, an open benchmark for evaluating AI code review agents, built on representative GitHub pull requests. The company announced it on the GitHub Blog under its Copilot coverage.

What GitHub says ReviewBench does

According to GitHub, the benchmark rests on four design elements:

• Representative GitHub pull requests. Evaluation cases are drawn from pull requests meant to reflect real-world review work, not synthetic or narrowly curated tasks.
• Multi-source ground truth. The reference answers for what a good review should catch come from more than one source, not a single annotator or a single signal.
• Calibrated evaluation. GitHub says the scoring process is calibrated. The announcement summary does not specify how.
• Production-aligned metrics. The metrics are designed to match what matters when a review agent runs in live workflows.

What has not been disclosed

The source material available at publication does not include the following. We are not estimating them.

• Number of pull requests or repositories in the benchmark: not disclosed
• Programming languages covered: not disclosed
• Scores for any model or agent, including GitHub Copilot: not disclosed
• Specific metric definitions, such as precision, recall, or noise rate: not disclosed
• License and hosting terms for the "open" release: not disclosed
• Models or agents evaluated, and their pricing or context windows: not disclosed

This is a benchmark release, not a model release. No new model, weights, or API version accompanies it.

Why code review is a hard target

Existing code benchmarks mostly measure code generation or issue resolution, where a test suite can verify the result. Code review has no single correct output. Two competent reviewers can flag different issues on the same diff, and a comment can be technically correct but unhelpful or distracting. That makes ground truth difficult to define, which is likely why GitHub emphasizes multi-source ground truth and calibration in its description.

The focus on production-aligned metrics also points to a known problem with review agents: a tool that flags many issues can look strong on recall while producing comments developers learn to ignore. Whether ReviewBench's metrics penalize that behavior will depend on details GitHub has yet to make visible in the summary.

What this means

ReviewBench addresses a real gap. Code review agents are shipping in products, but there has been no widely adopted shared yardstick for them comparable to what SWE-bench became for issue resolution. An open benchmark built from real pull requests could give tool vendors and teams a common basis for comparison.

Two caveats apply. First, GitHub makes a competing product, Copilot, so the benchmark's credibility will depend on transparency: published methodology, data access, and results for third-party agents, not only its own. Second, the ground-truth and calibration claims cannot be assessed until the full methodology and dataset are examined. Until independent teams reproduce results, treat the design description as GitHub's claim, not a validated standard.

The first scores for competing agents will show whether ReviewBench becomes a shared standard or stays a vendor-specific evaluation.