OpenAI Field Report: Coding Agents Speed Up Research Software 60x But Can't Verify Scientific Correctness

The core finding

A field report from OpenAI and academic partners shows that AI coding agents such as Codex and Claude Code can modernize aging research software, but cannot reliably judge whether their own output is scientifically correct. The report documents eight case studies, mostly in biology, ranging from simple build fixes to full GPU-native rewrites.

Much of the value shifted from writing code to verifying it. According to the report, agents completed well-defined technical tasks quickly, but repeatedly presented buggy code with full confidence when the underlying logic was wrong.

Speedups across eight case studies

The projects span a wide range of complexity:

• cyvcf2: GPT-5.5 modernized the build and installation process for this genetic-data-reading Python library.
• MHCflurry: Claude Code and Codex alternated as developer and reviewer to port roughly 10,000 lines of an immunology model from TensorFlow to PyTorch.
• rustar-aligner: A full Rust rewrite of STAR, a 20,000+ line C/C++ genome-mapping tool no longer actively maintained. On 10,000 yeast sequencing reads, the rewrite matched STAR's output in 99.815% of single-end cases and 99.883% of paired-end cases, with no reads mapped by one tool and missed by the other.
• RustQC: Combined 15 separate quality-control tools into one program, cutting runtime on a large dataset from 15 hours 34 minutes to 14 minutes 54 seconds — a speedup of more than 60x.
• HelixForge: A GPU-native rewrite of BamSurgeon for synthetic genomic data generation, completing a full pipeline test 59.6x faster, with the core compute step running 98.6x faster.
• hifiasm: GPT-5.5 optimized this genome assembly tool, cutting runtime on real human genome data by nearly 15%.
• HI.SIM: GPT-5.2, followed by a newer model in a second pass, reduced runtime by about 31% without changing output.
• bayesm: A Rust rewrite ran 2–20x faster than the original but contained hard-to-detect bugs, including an inverted control parameter and an incorrectly scaled correction factor in one method.

Confident but wrong

Across projects, researchers describe agents as capable of producing plausible-looking results even when those results were incorrect. Philip Ewels, who led RustQC, called the agents "eloquent, convincing, and confidently wrong in ways that are easy to miss," and said he never let models judge their own accuracy, instead building an independent test harness.

In the bayesm case, errors in two advanced statistical methods were not caught by initial agreement tests. One bug — an inverted control parameter causing the program to use the reciprocal of intended values — was only found after running detailed calibration tests against thousands of synthetic datasets with known results. A second method, HART, produced plausible overall results while still containing needlessly expensive calculations and an incorrectly scaled correction factor.

cyvcf2 developer Brent Pedersen summarized the pattern: "With coding agents, it's quite easy to go fast; for now, to go far in science, there's still a need for expert guidance, understanding, taste, and care."

Division of labor and maintenance risk

Across all eight projects, humans defined goals, success criteria, and validation methods while agents handled implementation. The report estimates that if agents resolved 25–50% of installation problems across 100 research software packages, saved research time would be worth $600,000 to nearly $5 million. For NumPy alone, agents could reportedly save about 650 maintenance hours per year.

The report also flags a maintenance risk: cheap AI-generated rewrites could fragment user communities and stretch already scarce maintainer time further. rustar-aligner was handed to the scverse research consortium after STAR's original maintainers stopped active development. The FastQC author declined to replace the original Java tool with its Rust rewrite, instead folding the improvements back into the original codebase.

The authors note the report is not a representative study but a collection of self-reported accounts. A related METR study found that actual project maintainers would reject roughly half of the solutions that the SWE-bench Verified benchmark rates as passing, and the curl project shut down its bug bounty program after AI-generated vulnerability reports overwhelmed maintainers without providing proportional value.

What this means

This report reinforces a pattern showing up across software engineering benchmarks: AI coding agents are increasingly capable of producing functional, fast code, but the bottleneck is moving from code generation to verification. In research software specifically, that verification requires domain expertise — statistical calibration, biological plausibility checks, and knowledge of what "correct" output actually looks like — that general-purpose coding agents don't possess. For labs considering AI-assisted modernization of legacy scientific tools, the practical lesson is that agent output should be treated as a fast first draft requiring independent, often labor-intensive validation, not as a finished, trustworthy product. The maintenance and ownership questions the report raises — who owns a rewrite, who maintains it, and whether it fragments an existing user base — may prove harder to solve than the technical rewrites themselves.