Composio Benchmark: Claude Code Fastest Agent Framework, But Costs Nearly 3x More Than OpenCode

AI tooling company Composio has published a benchmark showing that the choice of agent framework—not just the underlying model—significantly changes what developers pay and how fast tasks complete, even when the model stays constant.

Composio ran the same model, DeepSeek V4 Flash, across four agent frameworks: Claude Code, Codex, OpenCode, and Oh My Pi. Each framework tackled the same 30 real-world tasks involving tools like Gmail, GitHub, Slack, and Notion.

Key results

No single framework won across every category tested. According to Composio, the results broke down as follows:

• Oh My Pi: Highest success rate at 17/30 tasks completed, but slowest at 272 seconds per task on average.
• Claude Code: Fastest at 122 seconds per task, but most expensive at $0.195 per successful task—despite using the fewest tool calls and generating the least output tokens of any framework tested.
• OpenCode: Cheapest at $0.073 per successful task, but had the lowest success rate at 14/30.
• Codex: Fell between the other three on the metrics reported, though Composio did not disclose its exact success rate, cost, or speed figures.

Overall success rates across frameworks stayed fairly close, with OpenCode trailing only slightly behind the pack. Composio noted that seven of the 30 tasks passed or failed depending solely on which framework executed them—suggesting framework-specific behavior, not just model capability, determines outcomes on certain task types.

The headline gaps were in cost and speed: a nearly 3x price difference and a 2.2x speed difference separated the frameworks, even though all four ran identical underlying model weights.

Why the framework matters

Agent frameworks act as the orchestration layer between a language model and the tools it calls—deciding how many tool calls to make, how much context to retain, and how output gets generated. Composio's test isolates this layer by holding the model fixed across all four frameworks, showing that orchestration design alone can swing cost by a factor of three and latency by more than double.

Claude Code's combination of speed and low tool-call count suggests it is more aggressive at completing tasks in fewer steps, but each step appears to carry a higher per-call cost that pushes total spend above rivals. OpenCode's low cost paired with its lower success rate points to a possible tradeoff between thrift and reliability, though Composio's published results do not detail the failure modes.

What this means

For teams building on agent frameworks, model choice is only half the cost-and-latency equation—the orchestration layer matters just as much. Composio's benchmark, while limited to 30 tasks and one model, indicates that framework selection deserves the same scrutiny as model selection when optimizing for either speed or budget. The results have not been independently replicated, and Composio has not disclosed the exact prompts, task difficulty distribution, or Codex's full metrics, so the specific multipliers cited (3x cost, 2.2x speed) should be treated as directional rather than definitive until broader testing confirms them.