SPU wet-lab benchmark: top model Claude Opus 5.5 scores 0.486 and beats chance on only half of tasks

UniPat AI has released Science Prediction Universe (SPU), a benchmark that asks AI agents to predict real wet-lab results. The best average score in the first evaluation round, 0.486, went to Claude Opus 5.5, which exceeded the random-ranking threshold on only half of its tasks. The details come from a Chinese-language report by QbitAI (量子位) published 2026-11-14.

How SPU works

SPU contains 116 tasks and 4,207 wet-lab experiments across 32 subfields in chemistry, materials science, biology and medicine.

• Public tasks: 100 tasks drawn from 85 recent papers. Domain experts extracted candidates, materials, conditions and measurement methods, and kept the full quantitative results as the scoring basis. The measured results are hidden from the models.
• Private tasks: 16 tasks from four sets of unpublished collaborative experiments: VHH nanobodies, reaction conditions, electrolytes and Magrolimab variants.

Each run lasts eight hours, with 8 CPU cores, 16 GiB of memory and one H20 GPU. Models may search the web and install scientific software, then submit a low-to-high ranking of the comparison groups. Scoring happens in isolated, offline containers. Tool outputs are scanned for source-paper DOIs and titles to limit memorization. Rankings are compared with measured results using Spearman correlation. For each task, 4,000 random rankings are generated, and the 95th percentile serves as the chance threshold.

UniPat AI positions SPU against BrainBench, ScienceAgentBench, Terminal-Bench 4.0 and Long-Horizon-TB, arguing that SPU combines autonomous research with outcome prediction.

First-round results

The first round covered 12 models in 13 configurations (model, reasoning effort and framework).

• Claude Opus 5.5 had the highest average score on all scored tasks: 0.486.
• Across the 13 configurations, the share of tasks above the random threshold ranged from 34% to 52%.
• The configurations split into a leading group of 6 and a trailing group of 7. On shared tasks, the leading group averaged about 0.15 higher.
• About 9% of tasks scored below 0 in the leading group, against as much as 22% for the other seven. A score below 0 means the predicted ranking correlated negatively with the measured one.
• Where a pooled reference ranking was correct, the leading group ordered pairs correctly 86% of the time, against 78% for the rest. Where the reference was wrong, both groups got about 30% right.

By public-task discipline, the highest averages were Opus 5.5 in chemistry, Fable 5.1 in materials science, GPT-6 Astra Medium in biology and Opus 5 in medicine. The report notes that Fable 5.1 and Opus 5.5 showed refusals.

In one solvent task, a polymerization run across 12 solvents (66 pairs), Opus 5.5 scored 0.874 and Qwen3.8 scored -0.126. Opus 5.5 ordered 56 pairs correctly and Qwen3.8 ordered 30. Qwen3.8 ranked mainly by ethylene solubility, while Opus 5.5 focused on feedstock miscibility and catalyst compatibility. The report says Qwen3.8 noticed the relevant issue during computation but did not change its final ranking basis.

On the private tasks, 12 of 13 configurations averaged significantly above random, with no significant differences detected between them. Results varied sharply by measurement target. For ionic conductivity on one electrolyte set, correlations clustered between 0.89 and 0.95. For electrochemical window, the 12 scored configurations ranged from -0.55 to 0.07, and none cleared the threshold.

More compute did not help

Raising GPT-6 Astra from medium to the highest reasoning effort increased median runtime 4.8x and median API cost 5.3x, but scores barely moved. On 79 completed tasks, the highest setting typically produced its answer at a median of 26% of the run, and later work consisted of minor adjustments. GPT-6 Astra Max used a median of 4 method categories per run against 1 for Medium, again with no significant score difference.

GPU computation succeeded only 255 times despite ample GPU supply, fewer than one fifth of runs. Average API cost per run ranged from $0.22 to $65.91 across configurations, which the report describes as a 300x spread.

Availability

The 100 public tasks will be released as complete Harbor task packages. The 16 private tasks stay private for controlled evaluation. The dataset and code are open-sourced. The paper is at arXiv 2611.15049, with a project page at spu.unipat.ai and code at github.com/UniPat-AI/SPU.

Advisors come from Stanford, Cornell, Harvard, UIC, Peking University, HKUST and FutureHouse. Wet-lab partners include 优睿赛思, 爱博泰克, 深度原理 and 恩和生物.

What this means

SPU scores outcome prediction against hidden measurements, with a per-task chance threshold, so a top average of 0.486 alongside 34% to 52% of tasks above chance shows that no tested configuration predicts wet-lab outcomes reliably. The results also show that performance depends on the specific measurement target: the same electrolyte set produced strong ionic conductivity rankings and chance-level electrochemical window rankings. For GPT-6 Astra, extra reasoning effort and tool variety raised cost without raising scores, and the report attributes errors mainly to the step where a model selects its ranking basis. The reported $0.22 to $65.91 per-run cost range makes cost a practical selection factor alongside score.