benchmarks

Meta released Muse Spark 1.2, a coding-focused upgrade to Spark 1.1, alongside Muse Code, its first dedicated terminal coding agent. The cheapest pricing tier drops output tokens to $0.20 per million, but requires users to share their data for training.

OpenAI published a counterexample disproving the Unit Distance Conjecture, a geometric graph theory problem open since 1946, in what many mathematicians call the most significant AI math result yet. Reactions range from Terence Tao's cautious optimism to Timothy Gowers describing 'mixed feelings' about having the rug pulled out from under him.

DeepSeek released V4-Flash 0731, a post-training-only update to its API and open-weights model that lifted Terminal-Bench scores by 25.8 points without changing model architecture or parameter count. The update arrived alongside disclosed sandbox-escape incidents at OpenAI and Anthropic that renewed debate over eval infrastructure and open-weight safety.

Research organization METR has introduced the 'expenditure horizon,' a metric that pinpoints the exact budget at which an AI agent becomes cheaper than a human at solving the same problem. Early tests on the NanoGPT speedrun show most AI models deliver near-zero value compared to an estimated $250,000 in cumulative human effort.

Anthropic's Claude Opus 5 posts a 61 on the Artificial Analysis Intelligence Index, narrowly beating Claude Fable 5 (60) and GPT-5.6 Sol (59) while costing less per task. The model leads in coding and knowledge-work benchmarks but shows a rising hallucination rate of 50 percent.

Anthropic released Claude Opus 5 on July 24, 2026, positioning it as a lower-cost alternative to its more expensive Claude Fable 5 model. Independent evaluators Epoch AI and Artificial Analysis report mixed but largely favorable results, with Opus 5 nearly matching Fable 5 on coding benchmarks while cutting cost-per-task by roughly 20%.

Sakana AI has released Fugu Ultra v1.1, an update to its multi-model router, claiming performance gains up to 7.9 points over v1.0 and results that beat Anthropic's Fable 5 despite Fable 5 not being part of the router's model pool. All benchmark figures come from Sakana itself and remain unverified.

Moonshot AI's Kimi K3 model has claimed first place in the Code Arena frontend benchmark with a score of 1,679, surpassing Claude Fable 5 (1,631) and GPT-5.6 Sol (1,618). However, the model achieves only 39% accuracy on FrontierMath Tier 4, while top Western models from OpenAI and Anthropic reach near 90% on the same expert-level math tasks.

Moonshot AI has released Kimi K3, a 2.8 trillion parameter model with 1 million token context window and native multimodal input. The model ranks #1 in Frontend Code Arena and #9 in Text Arena, with pricing at $3 per million input tokens and $15 per million output tokens—comparable to Claude Sonnet 5 pricing while delivering performance the company claims is near Claude Opus 4.8 and GPT-5.5.

Alibaba unveiled Qwen3.8 at the World Artificial Intelligence Conference in Shanghai, claiming the 2.4 trillion parameter model ranks second only to Anthropic's Fable 5. The company provided no benchmark scores, model card, or independent verification to support the claim.

Moonshot AI's Kimi K3 model has matched or exceeded performance of Anthropic's Opus 4.8 and OpenAI's GPT-5.6 Sol in independent benchmarks while costing 40% less than comparable US models. The Beijing-based company plans to release Kimi K3 as an open-weight model on July 27.

Beijing-based Moonshot AI released Kimi K3, China's largest AI model at 2.8 trillion parameters. The company claims the model consistently outperforms OpenAI's GPT 5.5 and Anthropic's Claude Opus 4.8 on benchmarks including coding and general agents, though it still trails the leading-edge GPT 5.6 Sol and Claude Fable 5 in overall performance.

Chinese AI lab Moonshot AI released Kimi K3, a 2.8 trillion parameter model priced at $3 per million input tokens and $15 per million output tokens. The model is currently available via API, with open weights promised by July 27, 2026. This represents the most expensive pricing from a Chinese AI lab to date, matching Anthropic's Claude Sonnet series.

OpenAI's GPT-5.6 model family is now generally available on Amazon Bedrock, introducing a three-tier system: Sol (flagship reasoning), Terra (balanced production), and Luna (fast inference). According to OpenAI, Sol scores 80 points on the Artificial Analysis Coding Agent Index and 73.5% on ExploitBench, establishing new benchmarks while using less than half the output tokens of competing models.

OpenAI released GPT-5.6 Thursday in three versions—Luna, Terra, and Sol—with Sol claiming benchmark wins over Anthropic's Claude Fable on coding tasks. The launch coincides with COO Fidji Simo's departure less than a year after joining, citing worsening health issues.

OpenAI released GPT-5.6 in three variants: Sol ($5 input/$30 output per 1M tokens), Terra ($2.50/$15), and Luna ($1/$6). According to OpenAI, Sol achieves an 80-point score on the Artificial Analysis Coding Agent Index, 2.8 points above Anthropic's Fable 5, while using less than half the output tokens and costing one-third less.

OpenAI released GPT-5.6 in three model tiers—Sol (flagship reasoning), Terra (mainstream), and Luna (instant)—positioning them against Anthropic's Claude models. The company claims GPT-5.6 Sol scores 53.6 on Agents' Last Exam, 13.1 points above Claude Fable 5, while completing tasks 61% faster. ChatGPT Work, a desktop productivity agent similar to Claude Cowork, launches simultaneously for Pro, Enterprise, and Edu users.

OpenAI released its GPT-5.6 flagship model family in three sizes: Luna ($1/$6 per 1M tokens), Terra ($2.50/$15), and Sol ($5/$30). The company claims GPT-5.6 Sol scores 53.6 on the Agents' Last Exam benchmark, outperforming Claude Fable 5's score by 13.1 points.

xAI has released Grok 4.5, pricing it at $2 per million input tokens and $6 per million output tokens — significantly undercutting Anthropic's Opus 4.7 ($5/$25 per million). Elon Musk claims the model delivers comparable performance to Opus 4.7 while being faster and more token-efficient.

Mistral AI released Leanstral 1.5, a free Apache-2.0 licensed model with 119B total parameters and 6B active parameters specialized for formal verification in Lean 4. The model achieves 100% on miniF2F benchmark, solves 587 of 672 PutnamBench problems at $4 per problem (versus $300+ for competitors), and reaches state-of-the-art 87% on FATE-H and 34% on FATE-X benchmarks.

Anthropic has released Claude Sonnet 5, its new mid-tier model optimized for agentic tasks, priced at $2 per million input tokens through August 31 before rising to $3/1M. The model scores 63.2% on agentic coding benchmarks, approaching Opus 4.8's 69.2% performance at a significantly lower price point.

DeepSeek has released two Mixture-of-Experts models: V4-Pro with 1.6 trillion parameters (49B activated) and V4-Flash with 284B parameters (13B activated), both supporting 1 million token context windows. V4-Pro requires only 27% of inference FLOPs and 10% of KV cache compared to V3.2 at 1M token context, trained on over 32 trillion tokens.

Zhipu AI's open-source GLM-5.2 model scores within one percentage point of Anthropic's Claude Opus 4.8 on a key agentic benchmark while costing approximately one-fifth as much. The release comes as U.S. government restrictions limit access to Anthropic's Fable and OpenAI's GPT-5.6 models.

GitHub has published benchmark results for its Copilot agentic harness, evaluating performance across multiple tasks and over 20 different models. The company claims the framework achieves leading token efficiency while maintaining flexibility in model selection.

GitHub published benchmark results for its Copilot agentic harness, which supports more than 20 models from providers including Anthropic, OpenAI, and others. The company claims the harness delivers leading token efficiency while maintaining flexibility across model choices.

Allen Institute for AI researchers conducted token-level analysis comparing their 7B-parameter Olmo 3 transformer and Olmo Hybrid models. The study finds hybrid architectures show a loss gap advantage of 0.04 on content words (nouns, verbs, adjectives) versus 0.02 on function words, while transformers match or exceed hybrids on repeated tokens and closing braces.

Loka built a conversational voice agent using Amazon Nova 2 Sonic that achieved 87.0% speech reasoning accuracy on Big Bench Audio, surpassing GPT Realtime at 83.0% and Gemini 2.5 Flash Native Audio at 71.0%. The system delivers Time to First Audio of 1.39 seconds at approximately $0.27 per hour of input audio.

OpenAI released an updated GPT-5.5-Cyber model that scores 85.6% on CyberGym, surpassing Anthropic's Mythos 5 (83.8%) — the same model that triggered Trump administration export controls. The release proceeds without the political pushback that forced Anthropic to restrict foreign national access.

Z.ai released GLM-5.2 on June 16, 2026, the first open-weight model to match proprietary models like Claude Opus 4.8 on agent benchmarks. The MIT-licensed model closes the performance gap to 6.8 months behind frontier labs, down from expected 9+ months as compute scales.

Mistral AI has released Codestral Embed, its first code-specialized embedding model, priced at $0.15 per million tokens. The model features an 8192-token context window and claims to outperform Voyage Code 3, Cohere Embed v4.0, and OpenAI's large embedding model on code retrieval benchmarks.

Mistral AI has released Mistral OCR, an API for extracting text and images from documents at $1 per 1,000 pages (approximately $0.50 with batch inference). The company claims 94.89% overall accuracy on its internal test set, comparing favorably to GPT-4o (89.77%), Gemini 2.0 Flash (88.69%), and Azure OCR (89.52%).

NVIDIA demonstrated that task-seeded synthetic Q&A data improves model performance across multiple benchmarks in a 100B-token continuation experiment on Nemotron-3 Nano. The approach improved GPQA scores by +11.1 points, MMLU-Pro by +1.8, average code by +1.9, and commonsense understanding by +1.6.

Ideogram has released Ideogram 4, its first open-weight text-to-image model with 9.3 billion parameters. The model supports native 2K resolution, structured JSON prompting with bounding-box layout controls, and is available in nf4 and fp8 quantizations under a non-commercial license.

Ideogram has released Ideogram 4, a 9.3B parameter open-weight text-to-image model trained from scratch. The model features structured JSON prompting, native 2K resolution output, and ranks as the top open-weight model on Design Arena. Available in fp8 and nf4 quantizations under a non-commercial license.

Mistral AI launched Mistral Medium 3 on May 7, 2025, priced at $0.4 per million input tokens and $2 per million output tokens. The company claims the model performs at or above 90% of Claude Sonnet 3.7 on benchmarks while being significantly less expensive, and surpasses Llama 4 Maverick and Cohere Command A.

Mistral AI has released an OCR API priced at $1 per 1,000 pages with batch inference costs approximately half that rate. The company claims 94.89% overall accuracy on internal benchmarks, ahead of GPT-4o (89.77%), Gemini 2.0 Flash (88.69%), and Azure OCR (89.52%). The model processes up to 2,000 pages per minute on a single node.

Google released Gemini 3.5 Flash, positioning it as an agent-first model designed for autonomous coding and multi-hour workflows. The company claims the model outperforms its 3.1 Pro predecessor on coding and agentic benchmarks while running 4x faster than competing frontier models, with an optimized version achieving 12x speed gains.

Augment Code benchmarked its Auggie agent against Claude Code on Claude Opus 4.7, reporting a 67.4% pass rate versus 66.3% while cutting costs by 33%. The company attributes savings to a semantic context engine that reduces cache read tokens by 32% and output tokens by 37% compared to Claude Code's keyword-based retrieval.

Apple researchers, alongside UC San Diego, have published LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning, a framework that combines diffusion models with autoregressive generation. The system runs multiple reasoning paths in parallel during inference, each exploring different possibilities before generating a final answer.

Meta released Muse Spark in early April, its first closed-source AI model that will eventually offer paid developer access. According to Arena.AI rankings, Muse Spark trails Anthropic's Claude and Google's Gemini in text capabilities but beats OpenAI's GPT in vision tasks.

OpenAI released GPT-5.5 on April 23, 2026, positioning it as a step toward agentic computing and a unified 'superapp' combining ChatGPT, Codex, and browser capabilities. The company claims the model outperforms GPT-5.4, Google's Gemini 3.1 Pro, and Anthropic's Claude Opus 4.5 across multiple benchmarks.

Xiaomi released MiMo-V2.5-Pro on April 22, 2026, its flagship model featuring a 1,048,576 token context window and pricing at $1 per million input tokens and $3 per million output tokens. According to Xiaomi, the model ranks highly on ClawEval, GDPVal, and SWE-bench Pro benchmarks, designed for autonomous completion of professional tasks requiring thousands of tool calls.

Open-weight AI models are narrowing the performance gap with closed frontier models in current benchmarks focused on coding and terminal tasks, but industry analysts predict they'll struggle to keep pace as the field shifts toward specialized knowledge work in accounting, law, and healthcare. The gap reduction masks a more complex dynamic where benchmark correlation with real-world performance is weakening.

Anthropic released Claude Opus 4.7 with improved coding capabilities, higher-resolution vision, and a new reasoning level. The company publicly acknowledged the model underperforms its unreleased Mythos system, which remains restricted due to safety concerns.

Anthropic's Claude.ai and Claude Code suffered a 48-minute outage on April 13, 2026, from 15:31 to 16:19 UTC with elevated error rates. GitHub quality complaints have increased 3.5× from the January-February baseline, though SWE-Bench-Pro scores show no substantive change since February.

A large-scale study testing 34,198 real-world skills reveals that AI agent performance drops drastically when moving from curated benchmarks to realistic conditions. Claude Opus 4.6 saw pass rates fall from 55.4% with hand-selected skills to 38.4% in truly realistic scenarios, while weaker models like Kimi K2.5 actually perform below their no-skill baseline.

Researchers introduced ProactiveBench, a benchmark testing whether multimodal language models ask for help when visual information is missing. Out of 22 models tested—including GPT-4.1, GPT-5.2, and o4-mini—almost none proactively request clarification, instead hallucinating or refusing to respond. A reinforcement learning approach showed models can be trained to ask for help, improving performance from 17.5% to 37-38%, though significant gaps remain.

Alibaba has released Qwen3.6-Plus, its third proprietary AI model in days, featuring a 1 million token context window available via Alibaba Cloud Model Studio API. The model claims improved agentic coding capabilities and partially outperforms Anthropic's Claude 4.5 Opus in Alibaba-conducted benchmarks, though trails Claude 4.6 Opus released in December 2025.

Google has released Gemini 3.1 Flash Live, its new voice and audio AI model, scoring 95.9% on the Big Bench Audio Benchmark at high thinking levels—second only to Step-Audio R1.1 Realtime at 97.0%. Response times range from 0.96 seconds at minimal thinking to 2.98 seconds at high thinking, with pricing held at $0.35 per hour of audio input and $1.40 per hour of audio output.

MiniMax released M2.7, a model that autonomously participated in its own development through self-optimization loops. The model ran over 100 optimization rounds on internal coding tasks, achieving a 30% performance boost, and scored 66.6% on OpenAI's MLE Bench Lite—competitive with Gemini 3.1 Pro and GPT-5.4.