AI evaluation
4 articles tagged with AI evaluation
METR adds open-ended tasks to its AI time horizon plot; models fall short on judgment-heavy work
AI evaluation nonprofit METR has updated its time horizon plot with longer and more open-ended tasks. The report finds some capabilities still improving exponentially, but models fall short of humans on judgment-heavy tasks such as designing AI evaluations.
METR Finds Recent AI Models, Including GPT-6 Astra, Can Complete Tasks Taking Humans a Full Work Week
AI nonprofit METR said recent AI models, including OpenAI's GPT-6 Astra, can complete tasks that would take a human around a full work week. METR said AI capabilities have continued doubling every 3.6 months.
METR says Anthropic redacted most of the evidence it found most informative in its ~100-page report
METR says it submitted a roughly 100-page full report on its assessment to Anthropic, and that Anthropic redacted a majority of the evidence METR found most informative from the public version. METR says readers must therefore rely on its judgment more than it considers ideal.
Mercor study: AI models beat 12 licensed CPAs on simplified accounting tasks, but top score on full APEX benchmark is 61
A Mercor study found AI models beat 12 licensed CPAs on simplified tasks from the APEX Accounting Benchmark. On the full 160-task benchmark, the top model, Claude Opus 5.5, meets 61.8% of grading criteria, and no model fully solved almost 60% of tasks.