benchmark

METR adds open-ended tasks to its AI time horizon plot; models fall short on judgment-heavy work

TL;DR

AI evaluation nonprofit METR has updated its time horizon plot with longer and more open-ended tasks. The report finds some capabilities still improving exponentially, but models fall short of humans on judgment-heavy tasks such as designing AI evaluations.

4 min read
0

METR, the AI evaluation nonprofit, has updated its time horizon plot to include longer tasks and open-ended, judgment-heavy tasks. The new report finds that some AI capabilities continue to improve along an exponential trajectory, but models perform worse on tasks that require judgment than they do on coding. MIT Technology Review reported the update.

How the original plot worked

METR evaluated models by having them attempt a large suite of tasks that humans had also completed. Each task's difficulty was scored by how long human testers took to finish it. Models were then rated by the length of the tasks they could complete. The original plot showed that every four months, the length of tasks frontier models could successfully complete doubled. The plot was widely shared and, according to MIT Technology Review, widely misinterpreted because of the complexity of the measures involved.

What the update adds

The updated plot adds more tasks. Some are estimated to take humans far longer than anything in the original suite, and others are more open-ended. The original plot included no hard-to-verify tasks, because success on them is difficult to measure automatically.

That matters because reinforcement learning works well on verifiable tasks such as coding and math, where a solution can be checked automatically. It is harder to apply to tasks where success is less well defined or takes longer to determine, such as choosing a new research direction.

Results on judgment-laden tasks

  • Designing AI evaluations: Models were asked to design new evaluations METR could use in its research, and METR employees scored the results. Not one AI-designed evaluation was good enough for staff to consider it useful for their research.
  • Scoring philosophical arguments: This task is intended to capture the extent to which models exercise good judgment. METR writes that its employees disagree about how relevant it is to AI R&D automation. Models fell short of human performance here too.
  • Trend over time: Some of the newest models, such as Opus 5.5, tend to reach what METR calls "borderline above the bar" performance on the evaluation design tasks. A year ago, every model scored "clearly below the bar."

Methodological limits

METR includes few of these tasks in the report, which makes it hard to predict how progress will continue. In the original plot, human completion times were measured by having people attempt each task. In the new report they are mostly estimates, because paying people to complete tasks that might take hundreds of hours, or that only a few experts can do, is prohibitively expensive and time-consuming. For the philosophical argument task, one person rated most of the arguments, and agents are judged by how well they agree with that individual.

Matt Putz, a member of technical staff at METR, said in an email: "By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify. Ultimately, I don't think we'll be able to establish trends as clean and robust as for the easy-to-specify [tasks]."

Outside views

Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said: "We're at the point where AI evaluations need to get more creative." He was not surprised by the shortfall on open-ended tasks, citing an August preprint he co-authored reporting that models cannot yet reproduce human computer science research on their own. He thinks research taste will be a major bottleneck to AI R&D automation, while others contend models trained on easy-to-verify tasks will eventually complete open-ended ones too.

Najoung Kim of Boston University said the observed improvement is informative for the current moment, but she is not sure how it would extrapolate.

In a Wednesday blog post, OpenAI reported that its models do worse on real-world tasks its employees ask of them than on the easier, more verifiable tasks in METR's original suite.

The report arrives after a separate, heavily redacted METR report earlier this month that claimed AI might be speeding up AI capabilities progress at Anthropic by a factor of two.

What this means

The time horizon trend that drew attention was measured on tasks with checkable outcomes, and the update shows those gains do not yet transfer cleanly to open-ended work such as designing useful evaluations. The movement of Opus 5.5 from "clearly below the bar" to "borderline above the bar" on that task is a measured change, but METR's few tasks, estimated human times and single-rater scoring limit what it supports. METR itself and the outside researchers quoted say the evidence is not yet enough to answer whether, or how quickly, automated AI R&D might arrive.

Related Articles

benchmark

METR Finds Recent AI Models, Including GPT-6 Astra, Can Complete Tasks Taking Humans a Full Work Week

AI nonprofit METR said recent AI models, including OpenAI's GPT-6 Astra, can complete tasks that would take a human around a full work week. METR said AI capabilities have continued doubling every 3.6 months.

benchmark

OpenAI introduces MedChemBench to test AI on early drug discovery tasks

MedChemBench evaluates how well AI models interpret molecular structures, reason from experimental evidence, and select compounds in early drug discovery. It is published on OpenAI's site under the title "Can AI help scientists identify promising drug candidates?"

benchmark

OCR models now cost under 10 cents per 1,000 pages, down from 51 cents in 18 months

Daniel van Strien of Hugging Face compared eight text-recognition models on 850 historical pages. Cost per 1,000 pages fell from about 51 cents (RolmOCR, April 2025) to just under 10 cents (LightOnOCR-3-0.8B, October 2026), and a million pages now runs between $65 and $130 in GPU rental.

benchmark

InclusionMed launches medical AI evaluation platform, with a 22-model first index across 1,034 tasks

InclusionMed, which describes itself as the world's first independent evaluation and analysis platform for healthcare AI, has gone live with a nine-cell benchmark map (InclusionMed Atlas) and a first composite index (InclusionMed Index). The first round compares 22 models from 11 vendors across five benchmarks and 1,034 tasks.

Comments

Loading...