benchmark

Artificial Analysis Launches Search Index Benchmark for AI Agent Search APIs

TL;DR

Artificial Analysis has released the Search Index, a benchmark measuring how search API providers perform for AI agents across quality, cost, and speed. Parallel, Exa, and Firecrawl lead the initial rankings, with search access boosting model scores from 33 to as high as 75 points.

3 min read
0

Artificial Analysis has released the Search Index, a benchmark designed to measure how well search API providers perform when used by AI agents. The index evaluates providers on three dimensions: answer quality, total cost, and speed.

How the benchmark works

The initial lineup includes seven search providers: Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave. To isolate the effect of the search layer itself, Artificial Analysis tests each provider with the same underlying model — GPT-5.6 Luna — in an identical agent setup. Only the search API changes between test runs.

The agent runs on Stirrup, an open-source agent framework built by Artificial Analysis, and each task is repeated 25 times per provider to account for variance in search and retrieval outcomes.

The Search Index score combines three equally weighted benchmarks:

  • DeepSearchQA: 900 research questions, each requiring multiple search queries to answer
  • BrowseComp subset: 200 hard-to-find facts requiring multi-step browsing
  • AA-Omniscience: 600 questions spanning six knowledge domains

A tool-free baseline — where the model answers without any search access — serves as the comparison point. According to Artificial Analysis, the model scores just 33 points without search. With search enabled, scores range from 65 to 75 across the tested providers.

Results: quality, cost, and speed tradeoffs

Parallel, Exa, and Firecrawl lead the quality rankings with scores of 75, 74, and 73, respectively.

Artificial Analysis found that higher search quality tends to reduce total costs, because a model that gets good results on its first pass needs fewer follow-up queries and burns fewer tokens overall. In one comparison, Parallel Search's "advanced" tier cut token usage by more than 40 percent relative to its "Basic" tier. Even though per-task search costs rose under the advanced tier, total task cost came in lower — $0.084 versus $0.11 for Basic.

Speed showed a similar tradeoff. Parallel Search's "turbo" mode posted the fastest per-query response time at 0.51 seconds, compared to 1.03 seconds for Basic. But turbo's lower quality score (67 versus 73 for Basic) forced the agent to run additional search passes, so total time per task ended up roughly the same across both modes.

Based on these results, Artificial Analysis says Parallel, Firecrawl, and Parallel (turbo) offer the best combined mix of cost and performance among the providers tested.

Open methodology, open participation

Artificial Analysis has published its full benchmark methodology and says additional search providers can apply to be included in future rounds of testing.

What this means

As AI agents increasingly rely on live web search to answer questions and complete tasks, the search API layer has become as consequential to agent performance as the underlying language model itself. This benchmark makes an argument that's easy to miss when providers publish speed or price alone: a slower, pricier query can still produce a cheaper, faster task if it returns better results the first time. For teams building search-augmented agents, that reframes the buying decision — total cost and end-to-end latency matter more than any single per-query metric. It's also notable that this comes from Artificial Analysis, which has built credibility as a neutral evaluator of LLM providers; extending that model to the search-API market suggests this layer of the agent stack is now mature — and commercially important — enough to warrant independent, standardized comparison.

Related Articles

benchmark

Qwen4 Max scores 43 on Artificial Analysis Intelligence Index, 9 points above Qwen3.8 Max

Artificial Analysis reports that Alibaba's Qwen4 Max scores 43 on its Intelligence Index using xhigh reasoning effort. That is a gain of 9 points over Qwen3.8 Max, and 3 points behind Kimi K3.1 Pro.

benchmark

Gemini 4 Hydrogen scores 54 on Artificial Analysis Intelligence Index, just below Claude Sonnet 5.5 and GPT-6.1 Sol

Artificial Analysis reports that Google's Gemini 4 Hydrogen scores 54 on its Intelligence Index, just below Claude Sonnet 5.5 and GPT-6.1 Sol. It costs a third of Sonnet's cost but about 2.5x that of GPT-6.1 Sol, and adds an xhigh reasoning level.

benchmark

InclusionMed launches medical AI evaluation platform, with a 22-model first index across 1,034 tasks

InclusionMed, which describes itself as the world's first independent evaluation and analysis platform for healthcare AI, has gone live with a nine-cell benchmark map (InclusionMed Atlas) and a first composite index (InclusionMed Index). The first round compares 22 models from 11 vendors across five benchmarks and 1,034 tasks.

benchmark

Databricks launches public AI Index: open-weight models reach 30.9% of enterprise tokens, up from 7.3% in July

Databricks has launched the AI Index, a public resource combining usage data from 10,000+ companies on Unity Gateway with weekly coding-agent evaluations. Open-weight models grew from 7.3% of enterprise token traffic in July to 30.9% in November, and cost per task varies far more than token prices suggest.

Comments

Loading...