Artificial Analysis Launches Optima, a Platform to Build Custom AI Benchmarks on Your Own Data

Artificial Analysis has released Optima, a platform that lets users build custom AI benchmarks tailored to their own data and workflows rather than relying on generic public benchmarks. The platform is available now.

Artificial Analysis is known for independent LLM evaluations and benchmark implementations like GDPval-AA and AA-Briefcase. According to the company, Optima addresses a core limitation of public benchmarks: they compare models on predefined tasks and criteria that don't necessarily reflect which model performs best for a specific real-world use case.

How Optima works

Users can build benchmarks from several source types. Optima accepts existing evaluation datasets uploaded directly or pulled from Hugging Face, as well as AI agent traces from platforms like Arize, Braintrust, or Langfuse. Developers can also install a skill that collects data from their coding environment and past sessions, according to Artificial Analysis.

Users without existing datasets can instead describe their use case along with sample inputs and outputs. Optima then generates suggested test inputs, evaluation criteria, and example tasks, which users can review and refine before running the benchmark.

Two scoring methods are available: rubric-based evaluation against defined criteria, and pairwise comparison, the same method Artificial Analysis uses in GDPval-AA and AA-Briefcase. In the pairwise approach, users evaluate a sample of response pairs to indicate preference, and Optima derives a full ranking across the dataset from those judgments.

Cost and time as core metrics

Optima tracks cost per task and time per task as standalone comparison dimensions alongside quality. According to Artificial Analysis, this lets users determine whether a model's performance gain justifies its higher cost or longer processing time — a distinction raw token pricing doesn't capture, since a cheaper model can cost more overall if it requires more attempts or produces more failures.

Artificial Analysis says early testers used Optima to find finance and accounting agent models that cut costs by a factor of ten without significant quality loss, to match legal writing styles, and to evaluate accuracy on proprietary image datasets.

Pricing

Optima charges only the actual token costs of the models tested, with no markup, according to Artificial Analysis. Rubric-based evaluations cost $0.125 per criterion per model, and pairwise evaluations cost $0.375 per comparison. The platform holds a balance based on a cost estimate at benchmark creation, each run, and each evaluation round, then bills based on actual usage.

The benchmarking problem it targets

Public benchmarks have well-documented reliability issues. An Epoch AI analysis found that undisclosed differences in prompt wording and temperature settings caused identical models to score differently across implementations. For agentic benchmarks like SWE-bench, simply changing the scaffold — the agent's control software and tool environment — produced up to 15 percentage points of score variation.

A broader study of 445 benchmark papers from leading AI conferences found that nearly all had methodological weaknesses in at least one area, including unclear definitions, unrepresentative samples, or missing statistical validation. Only about 10 percent used complete real-world tasks reflecting actual application scenarios.

What this means

Optima solves a real and specific problem: generic benchmarks don't tell you which model is best for your exact workflow. Letting teams test against their own data, with cost and time tracked as first-class metrics, is a genuine improvement over comparing raw benchmark leaderboard scores.

But Optima doesn't fix benchmarking's deeper methodological issues — it relocates them. A custom benchmark is only as good as its design: how representative the test cases are, how clearly the target capability is defined, and how rigorously the evaluation is documented. Users still need to do that work themselves, and Artificial Analysis's own research shows most teams historically haven't. Cost and time per task also don't capture downstream value — a fast, cheap output that requires heavy rework may still be a poor deal. Optima is a tool for better benchmarking, not a guarantee of it.