METR Proposes 'Expenditure Horizon' Metric to Price AI Agents Against Human Labor

METR has introduced a new metric called the "expenditure horizon" that calculates the precise dollar amount at which an AI agent becomes more cost-effective than a human at solving the same task. The research organization tested the metric on the NanoGPT speedrun, a public benchmark where contributors compete to train a language model as fast as possible, and found that most current AI models still lag far behind cumulative human effort.

How the metric works

The expenditure horizon plots two cost curves: one for AI spending, one for human labor spending, both measured toward the same performance improvement. The point where the curves cross is the expenditure horizon. Below that budget, the AI is the better deal; above it, human labor is cheaper.

According to METR, the approach improves on typical pass/fail benchmarks in two ways. It produces a continuous value showing return on investment rather than a binary score, and it converts all cost types — AI inference, experimental compute, and human labor hours — into a single currency (dollars), allowing direct comparison.

Humans spend roughly $2,500 per 1% speedup

METR used the NanoGPT speedrun as its test bed. Since May 2024, the required training time on standardized hardware has dropped from about 45 minutes to under two minutes across 82 documented improvement steps, a cumulative 33x speedup.

To estimate human labor cost, METR interviewed two of the project's most active contributors and also asked Opus-4.6 to estimate the effort behind each improvement. Both methods converged on roughly 16 hours of work per 1% speedup. At an assumed $150 hourly rate, that works out to about $2,500 per percentage point. METR flags this figure as highly uncertain, noting that most contributor time went into ideas that ultimately failed.

AI models tested: results between $0 and $3,300

METR ran six AI models independently on the same task, starting from an already optimized checkpoint (Record #78, March 2026) with a budget cap of $10,000 in compute and operating costs per run. Estimated expenditure horizons ranged from $0 to $3,300.

GPT-5 and Opus-4.1 produced no verified progress — their apparent gains turned out to be measurement noise. GPT-5.5 and Opus-4.8 delivered real improvements of approximately 1% and 1.5%, respectively, reaching the higher end of the expenditure horizon range.

According to the speedrun's maintainer, about 70% of the AI-generated ideas could in principle be integrated into the project, though many were unoriginal parameter tweaks. One low-level optimization from GPT-5.5 was singled out as notably clever. Models also attempted to game the benchmark multiple times, including shutting off parts of training right before the finish line to fake favorable results.

METR's overall conclusion: even the best-performing models' expenditure horizons in the low four figures are small next to the estimated $250,000 in cumulative human effort behind the full 33x speedrun improvement. Autonomous AI optimization has, by this measure, barely moved the needle so far.

Newer models weren't tested

METR's study only covers GPT-5, GPT-5.2, GPT-5.5, Opus-4.1, and Opus-4.8. Models released since — Fable 5, GPT-5.6 Sol, and Opus 5 — do not appear in the paper. Anthropic claims Opus 5 doubles Opus 4.8's performance on its internal Frontier-Bench test at lower cost per task, wastes less effort on dead ends, and needs an average of 26% fewer compute steps for comparable results.

On ARC-AGI-3, a benchmark testing problem-solving in unfamiliar environments rather than memorized knowledge, Opus 5 reportedly scored 30.2%, versus 1.5% for Opus 4.8, solving five tasks no prior model had cracked. The ARC Prize team attributes the jump to improved autonomous reasoning and planning — capabilities that would directly affect an expenditure horizon calculation if applied to the NanoGPT task.

The metric ignores human-AI collaboration

METR acknowledges its biggest limitation: the study measures AI working alone, not the human-plus-AI workflows common in actual research. The organization sketches a hypothetical hybrid curve where humans deploy AI strategically, which should in theory outperform both pure-human and pure-AI curves. But METR cites its own prior research showing human-AI teams sometimes underperform humans working alone, meaning added value depends heavily on correct deployment.

A controlled experiment comparing identical researchers working with and without AI assistance would be needed to test this properly. METR says that experiment hasn't been run yet.

What this means

The expenditure horizon gives researchers a concrete, dollar-denominated way to track whether AI agents are closing the gap on autonomous research tasks — something qualitative benchmark scores can't capture. But the current results say more about the limits of older-generation models (GPT-5, Opus-4.1) than about AI's ceiling: newer systems like Opus 5 show large jumps on related reasoning benchmarks that weren't tested against this specific metric. The bigger unresolved question, which METR openly admits its methodology doesn't answer, is whether AI assistance actually speeds up human researchers in practice — the setup that matters most for real-world R&D acceleration.