benchmarks

Sage Sci Perception (书生·元征) connects scientific encoders to a language model through trainable projectors. Published results include 97.35% accuracy on BEACON ncRNA RNA family classification and 49.75% Acc@1 on QM9S joint-spectrum structure identification.

Tencent says it has expanded the inference cluster serving Hy4 tenfold after many WorkBuddy users hit their quotas on the model's first day. Hy4 scores 35 on the Artificial Analysis Intelligence Index, second among Chinese models, and ranks 13th on Agent Arena.

Pangram has released Pangram Image 1.0, an AI image detector that it says reaches 99.96% accuracy on a 1,200-image internal test, and Pangram Image Flash 1.0, a smaller model for AI video detection that it says scores 98.7% on the DF26 benchmark. Both are available through the API, and the image model is integrated into Pangram's Chrome extension.

Google's Gemini 4 Carbon (high) scores 59 on Artificial Analysis Intelligence Index V5, one point above Claude Opus 5.5, and leads the AA-Omniscience hallucination test with 54. Access is limited at first to cyber defenders through Google's Fairwind program.

SenseTime has released SenseNova 6.8 Flash (日日新), a multimodal agent model built around "delegated intelligence": the user sets a goal, the AI runs the full workflow, and the human only reviews. According to SenseTime's results, it ranks first on TeamBench and DataAnalysis2.0 and is free for a limited time.

Indian startup NeuroDX says its 240-million-parameter language model Sphota averaged 52.4% across about 11 benchmarks, against 52.1% for a comparable Liquid AI model. The company also claims 50x less training data and 4.5x better memory efficiency. The results are preliminary and have not been independently evaluated.

Artificial Analysis released Intelligence Index v5 on Tuesday, with 60% of the test built on private, held-out questions. Moonshot AI's Kimi K3.1 Pro fell from 4th to 9th, and Google's Gemini 4 Carbon debuted at #1 with a score of 59.

Google announced Gemini 4 Carbon, which it says ranks #1 on v5.0 of the Artificial Analysis Intelligence Index with a score of 59. Access begins this week with a subset of cyber defenders through the Fairwind Program, at a stated price of $2 per million input tokens and $10 per million output tokens.

Moonshot's Kimi K3.1 Pro debut is drawing scrutiny over allegations that influencers were asked to promote the model without disclosing a commercial relationship with the company. VentureBeat's Carl Franzen reports that the model scores highly on third-party benchmarks, according to both Moonshot and third-party AI model evaluators.

Moonshot AI has released Kimi K3.1 Pro, which scores 55 on the Artificial Analysis Intelligence Index, an 11-point gain over Kimi K3 and fourth place overall. Parameter count is unchanged at 2.8 trillion, while API prices fall by 33 percent. Moonshot plans to release the weights by November 4, 2026.

OpenAI says its models are now better at legacy languages including COBOL, Fortran and Visual Basic. On its own MultiLangBench suite, GPT-6 Astra averaged an 87.4% pass rate across 20 languages versus 26.8% for GPT-5.6 Sol, and GPT-6.1 Sol reached 97.9% accuracy on COBOL migration tasks.

Google DeepMind's Gemini 4 Argon scored 97.5% on ARC-AGI-3 at a cost of $4,800 using the Provider Adapter Harness, according to ARC Prize. Under the Standard Harness it falls to 49.8%, and Epoch AI places it eighth on its Math ECI and sixth on its SWE ECI.

Artemis Security has launched Orion-1, a model that investigates and responds to corporate network attacks, in private preview. The company claims it beats Claude Opus 5.5 and GPT-6 Sol on its own Decision-Grade Readiness benchmark, with the largest gap in attack reconstruction (80.8 vs 58.3). Pricing, context window and parameter count have not been disclosed.

OpenAI's GPT-6 Sol and Luna cut input/output token prices in half versus GPT-5.6, with Sol now at $2/$10 per million tokens and Luna at $0.10/$0.50. Independent testing from Artificial Analysis shows intelligence scores barely moved, with regressions on some knowledge-work benchmarks.

GPT-6 Astra completed Pokémon FireRed in 18 hours 12 minutes, five times faster than its predecessor, and scored 62.7% on ARC-AGI-3 versus 7.78% for GPT-5.6 Sol. The model also ran a 141-hour Minecraft session and finished Fallout 3 in roughly 59 hours, according to independent testers.

Chinese lab AllSpark has released Iris-mini and Iris-pro, two open-weight search agents built on Qwen3 models that claim the top spot among open-weight systems in their size classes on four research benchmarks. The release includes model weights, an agent harness, and evaluation code, with training pipelines to follow.

Artificial Analysis released version 4.2 of its Intelligence Index after its original scoring showed GPT-6 Astra barely improving on its predecessor, contradicting Epoch AI's ranking of Astra as the top model out of 267 tested. The update adds two benchmarks, drops the saturated GPQA-Diamond, and increases private test weighting to 40 percent.

Meta has released Muse Spark 1.3, its fourth model in five months, with an xhigh tier available now and a more powerful max tier in limited preview. The model improves sharply on agentic benchmarks and costs $0.55 per index task—cheaper than any rival at the same performance level—but still trails Claude Fable 5.1 on most tests.

Meta Superintelligence Labs shipped Muse Spark 1.3, which the company claims ranks #3 globally on the Artificial Analysis Intelligence Index and matches OpenAI's GPT-5.6-Sol on coding and agentic benchmarks. The model is available now via Muse Code and Meta's API, with open weights and a follow-up model promised soon.

Nvidia researchers report that a custom agent harness with a supervisory component drove Claude Opus 5 from a 30% baseline score to a perfect 100% on the ARC-AGI-3 benchmark. The findings add to mounting evidence that scaffolding, not the underlying model, is the dominant factor in long-horizon agentic performance.

A new industry analysis argues the performance gap between Chinese and Western AI models has narrowed to single-digit differences on broad benchmarks. Moonshot's Kimi K3 and Zhipu's GLM-5.3 now trail OpenAI and Anthropic's top models by only a few points on the Artificial Analysis Intelligence Index, with a clear Western edge remaining only in abstract reasoning, output reliability, and offensive cybersecurity capability.

Z.ai's GLM-5.3 ties Kimi K3 for the top spot among open models on the Artificial Analysis Intelligence Index, driven by a major leap in agentic task performance. The company is delaying the open-weight release by about two weeks, citing the model's unusually strong vulnerability-detection capabilities.

Artificial Analysis has launched Optima, a platform that lets users build custom AI benchmarks using their own data, workflows, or use-case descriptions. Unlike public benchmarks, Optima compares models on cost per task and time per task in addition to quality.

Microsoft's MAI Code 1.1 Flash beats its predecessor and mini-models from Anthropic and OpenAI on SWE-bench Verified, but DeepSeek-V4-Flash-0731 outperforms it on Terminal Bench 2.1 (82.7% vs 62.9%) while costing roughly a third as much per token.

OpenAI published a counterexample disproving the Unit Distance Conjecture, a geometric graph theory problem open since 1946, in what many mathematicians call the most significant AI math result yet. Reactions range from Terence Tao's cautious optimism to Timothy Gowers describing 'mixed feelings' about having the rug pulled out from under him.

Research organization METR has introduced the 'expenditure horizon,' a metric that pinpoints the exact budget at which an AI agent becomes cheaper than a human at solving the same problem. Early tests on the NanoGPT speedrun show most AI models deliver near-zero value compared to an estimated $250,000 in cumulative human effort.

Anthropic's Claude Opus 5 posts a 61 on the Artificial Analysis Intelligence Index, narrowly beating Claude Fable 5 (60) and GPT-5.6 Sol (59) while costing less per task. The model leads in coding and knowledge-work benchmarks but shows a rising hallucination rate of 50 percent.

Sakana AI has released Fugu Ultra v1.1, an update to its multi-model router, claiming performance gains up to 7.9 points over v1.0 and results that beat Anthropic's Fable 5 despite Fable 5 not being part of the router's model pool. All benchmark figures come from Sakana itself and remain unverified.

Moonshot AI's Kimi K3 model has claimed first place in the Code Arena frontend benchmark with a score of 1,679, surpassing Claude Fable 5 (1,631) and GPT-5.6 Sol (1,618). However, the model achieves only 39% accuracy on FrontierMath Tier 4, while top Western models from OpenAI and Anthropic reach near 90% on the same expert-level math tasks.

Alibaba unveiled Qwen3.8 at the World Artificial Intelligence Conference in Shanghai, claiming the 2.4 trillion parameter model ranks second only to Anthropic's Fable 5. The company provided no benchmark scores, model card, or independent verification to support the claim.

Moonshot AI's Kimi K3 model has matched or exceeded performance of Anthropic's Opus 4.8 and OpenAI's GPT-5.6 Sol in independent benchmarks while costing 40% less than comparable US models. The Beijing-based company plans to release Kimi K3 as an open-weight model on July 27.

OpenAI's GPT-5.6 model family is now generally available on Amazon Bedrock, introducing a three-tier system: Sol (flagship reasoning), Terra (balanced production), and Luna (fast inference). According to OpenAI, Sol scores 80 points on the Artificial Analysis Coding Agent Index and 73.5% on ExploitBench, establishing new benchmarks while using less than half the output tokens of competing models.

OpenAI released its GPT-5.6 flagship model family in three sizes: Luna ($1/$6 per 1M tokens), Terra ($2.50/$15), and Sol ($5/$30). The company claims GPT-5.6 Sol scores 53.6 on the Agents' Last Exam benchmark, outperforming Claude Fable 5's score by 13.1 points.

Mistral AI released Leanstral 1.5, a free Apache-2.0 licensed model with 119B total parameters and 6B active parameters specialized for formal verification in Lean 4. The model achieves 100% on miniF2F benchmark, solves 587 of 672 PutnamBench problems at $4 per problem (versus $300+ for competitors), and reaches state-of-the-art 87% on FATE-H and 34% on FATE-X benchmarks.

Anthropic has released Claude Sonnet 5, its new mid-tier model optimized for agentic tasks, priced at $2 per million input tokens through August 31 before rising to $3/1M. The model scores 63.2% on agentic coding benchmarks, approaching Opus 4.8's 69.2% performance at a significantly lower price point.

DeepSeek has released two Mixture-of-Experts models: V4-Pro with 1.6 trillion parameters (49B activated) and V4-Flash with 284B parameters (13B activated), both supporting 1 million token context windows. V4-Pro requires only 27% of inference FLOPs and 10% of KV cache compared to V3.2 at 1M token context, trained on over 32 trillion tokens.

Zhipu AI's open-source GLM-5.2 model scores within one percentage point of Anthropic's Claude Opus 4.8 on a key agentic benchmark while costing approximately one-fifth as much. The release comes as U.S. government restrictions limit access to Anthropic's Fable and OpenAI's GPT-5.6 models.

GitHub has published benchmark results for its Copilot agentic harness, evaluating performance across multiple tasks and over 20 different models. The company claims the framework achieves leading token efficiency while maintaining flexibility in model selection.

GitHub published benchmark results for its Copilot agentic harness, which supports more than 20 models from providers including Anthropic, OpenAI, and others. The company claims the harness delivers leading token efficiency while maintaining flexibility across model choices.

Allen Institute for AI researchers conducted token-level analysis comparing their 7B-parameter Olmo 3 transformer and Olmo Hybrid models. The study finds hybrid architectures show a loss gap advantage of 0.04 on content words (nouns, verbs, adjectives) versus 0.02 on function words, while transformers match or exceed hybrids on repeated tokens and closing braces.

Loka built a conversational voice agent using Amazon Nova 2 Sonic that achieved 87.0% speech reasoning accuracy on Big Bench Audio, surpassing GPT Realtime at 83.0% and Gemini 2.5 Flash Native Audio at 71.0%. The system delivers Time to First Audio of 1.39 seconds at approximately $0.27 per hour of input audio.

OpenAI released an updated GPT-5.5-Cyber model that scores 85.6% on CyberGym, surpassing Anthropic's Mythos 5 (83.8%) — the same model that triggered Trump administration export controls. The release proceeds without the political pushback that forced Anthropic to restrict foreign national access.

Z.ai released GLM-5.2 on June 16, 2026, the first open-weight model to match proprietary models like Claude Opus 4.8 on agent benchmarks. The MIT-licensed model closes the performance gap to 6.8 months behind frontier labs, down from expected 9+ months as compute scales.

Mistral AI has released Codestral Embed, its first code-specialized embedding model, priced at $0.15 per million tokens. The model features an 8192-token context window and claims to outperform Voyage Code 3, Cohere Embed v4.0, and OpenAI's large embedding model on code retrieval benchmarks.

Mistral AI has released Mistral OCR, an API for extracting text and images from documents at $1 per 1,000 pages (approximately $0.50 with batch inference). The company claims 94.89% overall accuracy on its internal test set, comparing favorably to GPT-4o (89.77%), Gemini 2.0 Flash (88.69%), and Azure OCR (89.52%).

NVIDIA demonstrated that task-seeded synthetic Q&A data improves model performance across multiple benchmarks in a 100B-token continuation experiment on Nemotron-3 Nano. The approach improved GPQA scores by +11.1 points, MMLU-Pro by +1.8, average code by +1.9, and commonsense understanding by +1.6.

Ideogram has released Ideogram 4, its first open-weight text-to-image model with 9.3 billion parameters. The model supports native 2K resolution, structured JSON prompting with bounding-box layout controls, and is available in nf4 and fp8 quantizations under a non-commercial license.

Ideogram has released Ideogram 4, a 9.3B parameter open-weight text-to-image model trained from scratch. The model features structured JSON prompting, native 2K resolution output, and ranks as the top open-weight model on Design Arena. Available in fp8 and nf4 quantizations under a non-commercial license.

Mistral AI launched Mistral Medium 3 on May 7, 2025, priced at $0.4 per million input tokens and $2 per million output tokens. The company claims the model performs at or above 90% of Claude Sonnet 3.7 on benchmarks while being significantly less expensive, and surpasses Llama 4 Maverick and Cohere Command A.

Mistral AI has released an OCR API priced at $1 per 1,000 pages with batch inference costs approximately half that rate. The company claims 94.89% overall accuracy on internal benchmarks, ahead of GPT-4o (89.77%), Gemini 2.0 Flash (88.69%), and Azure OCR (89.52%). The model processes up to 2,000 pages per minute on a single node.