Artificial Analysis Launches Search Index Benchmark for AI Agent Search APIs

Artificial Analysis has released the Search Index, a benchmark designed to measure how well search API providers perform when used by AI agents. The index evaluates providers on three dimensions: answer quality, total cost, and speed.

How the benchmark works

The initial lineup includes seven search providers: Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave. To isolate the effect of the search layer itself, Artificial Analysis tests each provider with the same underlying model — GPT-5.6 Luna — in an identical agent setup. Only the search API changes between test runs.

The agent runs on Stirrup, an open-source agent framework built by Artificial Analysis, and each task is repeated 25 times per provider to account for variance in search and retrieval outcomes.

The Search Index score combines three equally weighted benchmarks:

• DeepSearchQA: 900 research questions, each requiring multiple search queries to answer
• BrowseComp subset: 200 hard-to-find facts requiring multi-step browsing
• AA-Omniscience: 600 questions spanning six knowledge domains

A tool-free baseline — where the model answers without any search access — serves as the comparison point. According to Artificial Analysis, the model scores just 33 points without search. With search enabled, scores range from 65 to 75 across the tested providers.

Results: quality, cost, and speed tradeoffs

Parallel, Exa, and Firecrawl lead the quality rankings with scores of 75, 74, and 73, respectively.

Artificial Analysis found that higher search quality tends to reduce total costs, because a model that gets good results on its first pass needs fewer follow-up queries and burns fewer tokens overall. In one comparison, Parallel Search's "advanced" tier cut token usage by more than 40 percent relative to its "Basic" tier. Even though per-task search costs rose under the advanced tier, total task cost came in lower — $0.084 versus $0.11 for Basic.

Speed showed a similar tradeoff. Parallel Search's "turbo" mode posted the fastest per-query response time at 0.51 seconds, compared to 1.03 seconds for Basic. But turbo's lower quality score (67 versus 73 for Basic) forced the agent to run additional search passes, so total time per task ended up roughly the same across both modes.

Based on these results, Artificial Analysis says Parallel, Firecrawl, and Parallel (turbo) offer the best combined mix of cost and performance among the providers tested.

Open methodology, open participation

Artificial Analysis has published its full benchmark methodology and says additional search providers can apply to be included in future rounds of testing.

What this means

As AI agents increasingly rely on live web search to answer questions and complete tasks, the search API layer has become as consequential to agent performance as the underlying language model itself. This benchmark makes an argument that's easy to miss when providers publish speed or price alone: a slower, pricier query can still produce a cheaper, faster task if it returns better results the first time. For teams building search-augmented agents, that reframes the buying decision — total cost and end-to-end latency matter more than any single per-query metric. It's also notable that this comes from Artificial Analysis, which has built credibility as a neutral evaluator of LLM providers; extending that model to the search-API market suggests this layer of the agent stack is now mature — and commercially important — enough to warrant independent, standardized comparison.