IBM Research launches Open Agent Leaderboard for complete AI agent systems

IBM Research has released the Open Agent Leaderboard, the first open benchmark designed to evaluate full AI agent systems rather than just the models that power them. The leaderboard reports both quality and cost metrics across six diverse benchmarks, revealing that agent architecture significantly impacts performance even when using identical models.

Testing generality, not specialization

The benchmark evaluates agents across six established tasks spanning different domains: SWE-Bench Verified (code bug fixes), BrowseComp+ (web research), AppWorld (personal task completion), and three tau2-Bench variants (customer service and technical support). According to IBM Research, agents are tested as general-purpose systems without benchmark-specific tuning.

The leaderboard shows that the top three configurations all use the same underlying model but achieve different success rates and costs due to variations in agent architecture. "Same model, different agents, different results — the agent matters," the researchers write.

Failed runs cost 20-54% more

One of the most significant findings concerns failure behavior. IBM Research reports that failed agent runs cost 20-54% more than successful ones, with some agents failing fast and cheaply while others burn through expensive runs before terminating. This cost differential matters for production deployments where failure patterns directly impact operational expenses.

Tool shortlisting improves all models

The research identifies specific architectural components that improve performance. Tool shortlisting, which helps agents focus on relevant tools rather than searching through all available options, improved performance across every model tested. The technique "turned otherwise failing configurations into viable ones," according to the paper.

General agents match specialized systems

Contrary to expectations, IBM Research found that general-purpose agents are already competitive with specialized ones. "Across most benchmarks, general agents match or even outperform the best specialized systems," the researchers report. This suggests that single agents can increasingly handle diverse tasks without job-specific customization.

Unified protocol enables comparison

The technical foundation is Exgentic, an open evaluation framework that implements a unified protocol. The protocol standardizes how different agent systems interact with benchmarks by providing a consistent structure: a task description, context information, and available actions. This standardization allows fair comparison across agent architectures while preserving each benchmark's original design.

What this means

By evaluating complete agent systems rather than isolated models, the leaderboard makes visible what drives real-world performance: planning strategies, memory management, tool selection, and error recovery. The finding that failed runs cost significantly more than successful ones highlights a critical operational consideration typically absent from model benchmarks. Most importantly, the open release of methodology, framework, and results enables the research community to reproduce evaluations and submit new agent configurations. This transparency is essential for understanding which architectural choices generalize across tasks and which improvements come from the model versus the agent wrapper. For teams deploying AI agents, the leaderboard provides the first standardized way to compare full system costs and capabilities rather than relying solely on model performance claims.