GPT-6-Astra solves 49 of 50 theoretical physics problems on TheoPhysEval; Opus 5.5 scores 92%

GPT-6-Astra, running inside OpenAI's Codex environment, solved 49 of 50 problems (98%) on TheoPhysEval, a new benchmark of unpublished theoretical physics research problems. The full run cost just under $116, according to the benchmark's authors.

What TheoPhysEval tests

Thirty-five researchers from institutions including the International Centre for Theoretical Sciences (ICTS), Harvard University, KU Leuven, and Google DeepMind built the benchmark. It contains 50 unpublished problems with verifiable answers across eleven subfields, from fluid flows and biological processes to particle physics and cosmology.

The authors say a leading expert in the relevant subfield would need about a day or less per typical problem without AI help. Specialists from neighboring fields would take much longer. The problems are deliberately "turn-the-crank" work: contributors modified values in a calculation from one of their own recent papers so the answer couldn't be guessed from the original. The set excludes major open questions.

The authors tested commercially available agents with no human hints. Each agent ran twice: once with web search and once offline.

Results

Agent / model	Score (with web search)
GPT-6-Astra (Codex)	98% (49/50)
Opus 5.5 (Claude Code)	92%
Gemini 3.8 Flash	68%
DeepSeek-V4.1-Flash (open-weight)	54%
GLM 5.3 (open-weight)	42%

Without web access, Astra and Opus each scored 92%. The authors read the small gap as a sign that leading models have effectively memorized much of the theoretical physics literature.

On cost, the source headline cites $2.40 per problem, while the text says the average was under $3 and the total just under $116. Those figures imply roughly $2.30 per problem. Per-token pricing, context window, parameter count, and training cutoff for GPT-6-Astra were not disclosed in the source.

The one problem Astra missed also stumped every other model. It asks for an equation that produces a given curve, and the experts' solution relies on special mathematical forms that count as the required differential equation only in a broader sense. The authors acknowledge it may seem "unfair" but call it realistic.

Failure modes and answer-key errors

Weaker models made errors an expert would catch immediately. On a black hole problem, Gemini and GLM copied formulas unchanged from an older paper during the web-search run. Those formulas apply to five dimensions, not the required four.

For several problems, follow-ups with the experts revealed small errors in the reference solutions. After corrections, Astra's answer was correct every time. The authors suggest flawed answer keys may explain why agents score higher here than on other physics benchmarks such as CritPt, even though the authors consider their problems harder. A recent study found some theoretical physics benchmarks are graded incorrectly and can badly underestimate model ability.

The problems and test setup are available as a dataset on Hugging Face.

Limits the authors identify

The authors say some current models may already surpass the best human researchers on concrete calculations. They argue the calculation comes at the end of research, after someone builds a theory and picks the right questions. Agents are weak at switching to a simpler, equally interesting problem when one proves too hard, and they lack a shared sense of what makes a problem interesting. When asked to design new problems, they get lost in details a human would find boring. The authors stress these observations are not quantified and not a forecast.

They also report training implications: some problems would have worked as first projects for early-career researchers, but no longer do. They expect resource needs to rise sharply for harder problems, which could make research budgets matter more in a field that has historically needed little funding.

What this means

A 98% score at roughly $116 for the full set puts expert-day-level calculation work within reach of nearly any research group. The 92% offline result for both Astra and Opus matters more for evaluation than the headline number: it points to memorization of the literature, so a high score here does not show novel derivation.

The flawed answer keys are the other caveat. Benchmarks written by experts are now close enough to the ceiling that reference errors are a real source of noise, and the top-model ranking should be read with that in mind. The authors' own argument is that raw calculation is no longer the bottleneck. Problem selection and research direction are, and no current benchmark measures them.