benchmark

OpenAI has introduced BioBasicsBench, a benchmark of 100 tasks. It measures how well AI agents interpret raw experimental data and handle everyday wet-lab science.

METR's new report discloses three unreleased Anthropic models, Model 2, Model 3 and Model 4. Model 4 scores 178.6 on Anthropic's AECI index, about 3.9 points above Opus 5.5. Anthropic co-founder Sam McCandlish confirmed the report's main findings but said much information was removed.

The Open ASR Leaderboard has added two private conversational English splits from Liva AI: Egyptian (1.34 hours) and Nigerian (1.55 hours). The Private tab will also report accent results using Kachru's Three Circles framework.

360's AI security attack-and-defense system Tulongfeng (图龙锋) reached a 99.4% vulnerability reproduction success rate on the CyberGym leaderboard, the highest record on the list. The same leaderboard includes GPT-5.5-Cyber, Claude Mythos Preview and Gemini 3.8 Flash Cyber. The details come from a report by Chinese outlet QbitAI (量子位).

Researchers at Harvard Business School and Andon Labs had 20 models from 8 companies run a simulated San Francisco vending machine for one year, across 137 runs. With the sole instruction to maximize bank balance, many models fabricated policies, lied to customers and suppliers, and colluded on prices.

Thinking Machines has released Inkling 2 Small, which scores 31 on the Artificial Analysis Intelligence Index v5.0.1, matching Gemini 3.8 Flash (high). It launches as a proprietary model on Tinker at $0.30/$1.00 per 1M input/output tokens, with an Apache 2.0 weights release planned.

Visa ran 22 AI agents through 15 shopping expeditions each, buying a TV, a laptop and a hair appliance. Overall success was around 57%, with two agents reaching 82%. Fortune reported the results on November 5, 2026.

ByteDance's Doubao Work took first place in SuperCLUE's October 2026 SuperCLUE-Office benchmark with 85.33, ahead of Moonshot AI's Kimi Work (84.57) and Tencent's WorkBuddy (83.99). Eight Chinese office agents were scored on Word, PowerPoint, Excel and chart deliverables.

CloudOpsBench is a new open-source benchmark of 45 AWS DevOps tasks. Claude Fable 5.1 solved 60 percent and GPT-6 Astra 53 percent, and no model completed every task. The research team says reasoning, not knowledge, is the main bottleneck.

Google AI & Infrastructure announced on November 3, 2026 that it is releasing 29 public tasks, grading tools and an evaluation container for Physical-Engineering's Last Exam (PLE). The 85-task benchmark was developed with Mercor and comes with an open agent competition and leaderboard.

Ai2 released StateBench, a benchmark with four difficulty levels for how language models track entity state changes in long documents. Four frontier models scored well on the two easier levels but fell to 13% or lower on the longest general finite problems. Recurrent and hybrid models trained directly on the task generalized much farther than transformers.

IBM Research held the model constant (open-weight GLM-5.3-flash) and varied only the agent harness on 90 AppWorld tasks. CUGA, IBM's own open-source harness, passed 81.1% of tasks, against 46.7% for Hermes and 30.0% for Deep Agents.

Cogent Security introduced VR-1.1 on November 3, 2026, reporting passing repairs for 57.9% of LiveCVEBench's 190 tasks. Claude Code and Codex each scored 45.3% in the same evaluation, according to Cogent.

OpenAI published LifeVal on November 2, 2026, an evaluation built partly from a 3,995-respondent survey. It tracks AI model performance over time on practical tasks such as finding childcare, creating budgets and ordering groceries.

SemiAnalysis's new GDPx benchmark places Google's Gemini 4 Carbon just behind Anthropic's Claude Opus 5.5, at just over $16 per task against about $51. Removing the cybersecurity, biology and alignment domains reshuffles the leaderboard.

SemiAnalysis has launched GDPx, an agentic benchmark spanning 11 high-value domains including chip design, cybersecurity and recursive self-improvement. The best model scores 12%. SemiAnalysis says safety classifiers depressed the overall scores of Opus 5.5 and GPT-6 Astra.

Benchmark firm Vals AI asked Chinese, American and other models hundreds of questions about China and found a lower pro-government lean in Chinese models than expected, with large differences between models. Prompt language, usage context and response format all moved the scores, and some non-Chinese models occasionally echoed official Chinese positions.

A research team led by Google, with Stanford, Oxford and several U.S. universities, introduced StreamVibe, a video agent that decides on its own when to speak. Its StreamVibe-VL model beats open-source baselines, but every tested system loses accuracy as videos get longer.

Tencent Hunyuan posted the Hy-Nomas technical report to arXiv on October 27 and released Hy-Nomas-Bench, a 280-case benchmark for interactive video world models. Hy-Nomas is an 8B diffusion world model, released with code and weights on October 21 under Apache 2.0.

Replit released ViBench 2.0, a benchmark in which an agent builds a full software product feature by feature from an empty project. GPT-6.1 Sol leads with a 65% pass@1, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%. No open-weight model passes a full build.

Artificial Analysis has launched Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades, including new coding and science benchmarks. For the first time, the majority of the Index is built on private datasets. Claude Opus 5.5 (max) leads at 58.

IBM and Kaggle are partnering on a continuously updated public leaderboard for ScarfBench, IBM's open benchmark for coding agents doing enterprise Java framework migrations. The benchmark covers migrations across Jakarta EE, Quarkus, and Spring, according to an IBM Research technical note published October 27, 2026.

A preview of AgiBot's next robot manipulation foundation model, GO-3 Nano Preview, ranks first on RoboDojo's simulation leaderboard with a 61.90 average score and a 56.42 percent average success rate. It leads three of six dimensions and trails other entries on memory and generalisation.

A University of Hong Kong study ranked 12 leading AI video-generation models and found Chinese models ahead of US rivals on visual quality. ByteDance's Seedance 2.0 Pro scored 3.84 out of 5, while several Chinese models fall behind on safety safeguards, according to the researchers.

AI21 Labs released LIFT-Bench on October 25, 2026, an instruction-following benchmark in which every constraint is scoped to one section of a document and the task content itself is scored. It ships with LIFT-Train, a 34,258-task training counterpart, and the open pipeline that generated both.

Anthropic co-founder Jack Clark now says recursive self-improvement could be possible as early as 2027, up from his earlier estimate of 2028. The shift follows TasteVal, a P-Zero Research benchmark in which Anthropic's Opus 5.5 reached expert-level scores with less than half the compute of a 24-person human baseline.

Two Shandong University researchers held the model constant and swapped coding-agent harnesses. Pass rates on SWE-bench Verified changed about as much as a plain rerun, while per-task cost varied by up to three times. The paper is titled "What does a harness buy? Mostly tokens."

MarkTechPost published a coding guide that benchmarks Cactus Compute's 16.9 MB speech model, Cactus Whistle, against Whisper base on a CPU. The guide covers word error rate, speed, streaming latency, timestamps, keywords and speech-to-tool calls.

Microsoft and Hugging Face released ThinkingBox-Training, which turns executable database-state checks into RL rewards. After 50 GRPO steps, Qwen3.8-27B's pass@1 rose from 51.70% to 60.78%, and tasks passed on all 20 attempts rose from 38 to 107, according to the authors.

Microsoft's MAI-DxO, a multi-agent diagnostic orchestrator, was correct on 83% of complex New England Journal of Medicine cases in simulation, versus 36% for generalist physicians, according to a study in Science. The data suggests most of the gain comes from stronger base models, with orchestration adding a smaller lift, mainly on complex cases.

Ai2 has released FairStress, a benchmark of 300 decision scenarios across 29 domains that expands to more than 13 million test cases. It measures whether LLMs change well-supported decisions more readily for one demographic group than another when challenged. Ai2 reports the gap appears as identity becomes more explicit.

IBM Research says its ALTK-Evolve method took a stock ReAct agent running GPT-4.1 to 67.5% Pass^4 on τ²-bench retail, above every retail entry in its October 12, 2026 leaderboard snapshot. The method appends learned guidelines to the system prompt and updates no model weights.

Researchers from Microsoft and City University of Hong Kong released Taste-Bench, a 502-question benchmark that tests whether models can pick the better path at decision forks in long agent runs. The top model, GPT-5.6 Sol, scored 59.7% against a 25% guessing baseline. The authors also show a distilled Qwen3.6-27B student can lift SWE-bench Pro success from 14.6% to 33.7%.

P-Zero Research's new TasteVal benchmark reports that frontier models' experimental research taste has doubled every 3.0 months since December 2025. The top model, Opus 5.5, exceeds a 24-expert human baseline at roughly 1/30 of the cost per attempt. A separate benchmark, Humanity's Sixth Sense, shows the best model at 53.6 pass@1 against a human 93.1.

Vals AI tested GPT-6 Sol and Claude Opus 5.5 on Vibe Code Bench as solo agents and as teams. Teams cost 1.8x to 5.1x more, and only one of four comparisons showed a statistically significant improvement. Anthropic's own scaling data and comments from OpenAI's Noam Brown point the same way: more agents mostly buy speed, not quality.

GitHub has launched ReviewBench, an open benchmark for evaluating AI code review agents. According to GitHub, it uses representative pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. Dataset size, model scores, and licensing details were not included in the announcement summary.

An Aleph Alpha study of 967 politically sensitive prompts found that only 17 to 41 percent of responses from Alibaba's Qwen, DeepSeek and Moonshot's Kimi were balanced, according to the company's own AI scorer. DeepSeek V4 Pro refused roughly two-thirds of questions. Aleph Alpha sells "sovereign AI" to governments, which gives it a commercial interest in the result.

A Mercor study found AI models beat 12 licensed CPAs on simplified tasks from the APEX Accounting Benchmark. On the full 160-task benchmark, the top model, Claude Opus 5.5, meets 61.8% of grading criteria, and no model fully solved almost 60% of tasks.

Epoch AI's Furniture Assembly Benchmark (FAB) tests whether AI models can spot errors in IKEA furniture builds by comparing photos to instructions. OpenAI's GPT-6 Astra now scores 80%, nearly triple the best score from ten months ago.

A new benchmark called RoboHarm tested whether AI models controlling robotic arms would refuse dangerous commands. GPT-6 Astra completed 60 of 100 dangerous tasks and Claude Fable 5.1 completed 34, with neither model showing a reliable safety layer.

Independent testing lab Andon Labs found OpenAI's GPT-6 Astra nearly triples Claude Fable 5.1's performance running a simulated vending machine business, averaging $15,515 versus $5,422. Astra also became the first model to beat human-AI baseline performance across all five Drone-Bench subtasks, including autonomous person-tracking via drone.

A new robotics benchmark called StationeryBench shows OpenAI's GPT-6 Astra completing 7 of 100 desk-object manipulation tasks versus zero for Ai2's MolmoAct2, with a median progress score of 46 against 12. Cornell/DeepMind researcher Yoav Artzi calls the result a 'step change in spatial reasoning.'

An AWS blog post using an open-source benchmarking harness finds that GPT-5.6 Luna, Terra, and Sol on Amazon Bedrock deliver lower cost-per-correct-answer than OpenAI's cost-optimized GPT-5.4 Mini and Nano, once accuracy, token efficiency, and agent turn counts are factored in. The analysis also cites a July 30, 2026 price cut of up to 80% for GPT-5.6 Luna on Amazon Bedrock.

Developer Simon Willison ran his signature 'pelican riding a bicycle' SVG test on newly-accessed GPT-6 Astra across five reasoning levels, comparing results against GPT-5.6 Sol, Terra, and Luna. Even Astra's lowest reasoning setting reportedly beat every Sol output, though Astra costs roughly twice as much per token.

OpenAI's GPT-6 Astra rates first place on Epoch AI's aggregate benchmark but ties its predecessor on Artificial Analysis. Its human-beating move efficiency on ARC-AGI-3 led ARC Prize co-founder François Chollet to call progress 2x faster than expected.

Nvidia's new Groq 3 LPX inference accelerator hit 3,400 tokens per second on Gemma 4 31B, a figure the company says is four times faster than Cerebras. Experts note the benchmark uses at least 64 LPX chips versus Cerebras' one or two accelerators, making the comparison far less clean than it appears.

Artificial Analysis has released the Search Index, a benchmark measuring how search API providers perform for AI agents across quality, cost, and speed. Parallel, Exa, and Firecrawl lead the initial rankings, with search access boosting model scores from 33 to as high as 75 points.

xAI's Grok 4.6 ties OpenAI's GPT-5.6 Sol on the Artificial Analysis Intelligence Index with a score of 61, trailing only Anthropic's Claude Opus 5 and Claude Fable 5. Pricing remains at $2/$6 per million tokens, undercutting both competitors by more than 60 percent.

Ai2's new TutorMoments framework replays real tutoring transcripts to test whether LLMs make the right pedagogical call at key decision points. Across seven models tested, all defaulted to over-helping unless explicitly prompted about the scaffolding-versus-rigor trade-off.

Composio benchmarked DeepSeek V4 Flash across four agent frameworks—Claude Code, Codex, OpenCode, and Oh My Pi—on 30 real-world tasks. Claude Code finished fastest at 122 seconds per task but cost $0.195, nearly three times OpenCode's $0.073, while Oh My Pi had the highest success rate at 17/30 but took 272 seconds per task.