IBM Research launches Open Agent Leaderboard, showing same models achieve different results based on agent architecture
IBM Research has launched the Open Agent Leaderboard, the first open benchmark that evaluates complete AI agent systems rather than just underlying models. The leaderboard reveals that agents using identical models can achieve significantly different success rates and costs depending on system architecture, with failed runs costing 20-54% more than successful ones.
IBM Research launches Open Agent Leaderboard for complete AI agent systems
IBM Research has released the Open Agent Leaderboard, the first open benchmark designed to evaluate full AI agent systems rather than just the models that power them. The leaderboard reports both quality and cost metrics across six diverse benchmarks, revealing that agent architecture significantly impacts performance even when using identical models.
Testing generality, not specialization
The benchmark evaluates agents across six established tasks spanning different domains: SWE-Bench Verified (code bug fixes), BrowseComp+ (web research), AppWorld (personal task completion), and three tau2-Bench variants (customer service and technical support). According to IBM Research, agents are tested as general-purpose systems without benchmark-specific tuning.
The leaderboard shows that the top three configurations all use the same underlying model but achieve different success rates and costs due to variations in agent architecture. "Same model, different agents, different results — the agent matters," the researchers write.
Failed runs cost 20-54% more
One of the most significant findings concerns failure behavior. IBM Research reports that failed agent runs cost 20-54% more than successful ones, with some agents failing fast and cheaply while others burn through expensive runs before terminating. This cost differential matters for production deployments where failure patterns directly impact operational expenses.
Tool shortlisting improves all models
The research identifies specific architectural components that improve performance. Tool shortlisting, which helps agents focus on relevant tools rather than searching through all available options, improved performance across every model tested. The technique "turned otherwise failing configurations into viable ones," according to the paper.
General agents match specialized systems
Contrary to expectations, IBM Research found that general-purpose agents are already competitive with specialized ones. "Across most benchmarks, general agents match or even outperform the best specialized systems," the researchers report. This suggests that single agents can increasingly handle diverse tasks without job-specific customization.
Unified protocol enables comparison
The technical foundation is Exgentic, an open evaluation framework that implements a unified protocol. The protocol standardizes how different agent systems interact with benchmarks by providing a consistent structure: a task description, context information, and available actions. This standardization allows fair comparison across agent architectures while preserving each benchmark's original design.
What this means
By evaluating complete agent systems rather than isolated models, the leaderboard makes visible what drives real-world performance: planning strategies, memory management, tool selection, and error recovery. The finding that failed runs cost significantly more than successful ones highlights a critical operational consideration typically absent from model benchmarks. Most importantly, the open release of methodology, framework, and results enables the research community to reproduce evaluations and submit new agent configurations. This transparency is essential for understanding which architectural choices generalize across tasks and which improvements come from the model versus the agent wrapper. For teams deploying AI agents, the leaderboard provides the first standardized way to compare full system costs and capabilities rather than relying solely on model performance claims.
Related Articles
InclusionMed launches medical AI evaluation platform, with a 22-model first index across 1,034 tasks
InclusionMed, which describes itself as the world's first independent evaluation and analysis platform for healthcare AI, has gone live with a nine-cell benchmark map (InclusionMed Atlas) and a first composite index (InclusionMed Index). The first round compares 22 models from 11 vendors across five benchmarks and 1,034 tasks.
Tencent Yuanbao and KCL release GroupAssistBench: 2,395 questions from 80,299 group chat messages
Tencent Yuanbao and the King's College London NLP team released GroupAssistBench, an agent memory benchmark of 2,395 questions built from 50 real group chats and 80,299 messages. Across 14 configurations, better recall did not guarantee better service: EverMemOS scored 0.621 on L1 recall but 0.247 on L2 open-ended service.
GPT-6.2 Astra scores 41.2% on XDOF's Ember-0 robot benchmark, up from 19.6% for GPT-6 Astra
Robotics infrastructure company XDOF reports that GPT-6.2 Astra reached a 41.2% average success rate across the 16 tasks of its Ember-0 benchmark, versus 19.6% for GPT-6 Astra. It is the first model to complete rope knot-tying and T-shirt unfolding on the benchmark. Average prediction latency fell from 15 seconds to 10.1 seconds.
OpenAI introduces MedChemBench to test AI on early drug discovery tasks
MedChemBench evaluates how well AI models interpret molecular structures, reason from experimental evidence, and select compounds in early drug discovery. It is published on OpenAI's site under the title "Can AI help scientists identify promising drug candidates?"
Comments
Loading...