Ai2 Introduces BenchMIRT, a Method to Reveal What LLM Benchmarks Actually Measure

Ai2 has published BenchMIRT, a method for auditing large language model benchmarks at the individual question level, revealing that many evaluations marketed as measuring one capability actually reflect a mix of different underlying skills.

The research, detailed in a blog post and accompanying technical report, applies multidimensional item response theory (MIRT) — a technique adapted from psychometrics — to determine which capabilities are actually driving performance on benchmark questions, rather than relying on the benchmark's stated purpose.

How it works

BenchMIRT was trained on benchmarking results from 100 LLMs across 16 benchmarks and more than 34,000 questions. Six benchmarks measured general reasoning, including MMLU-Pro, GPQA, MATH, and BBH. The other 10 came from Ai2's Olmo 3 safety suite, including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest.

According to Ai2, the system was not told in advance which benchmarks corresponded to which capability. It independently recovered two dominant dimensions — safety and general reasoning — and produced the same two dimensions when the analysis was repeated from scratch, which the researchers say indicates the result is stable.

What it found

For most benchmarks, BenchMIRT confirmed their intended focus: reasoning benchmarks tracked with reasoning ability, and jailbreak/harmful-content benchmarks tracked with safety. But several evaluations showed more complicated patterns, according to Ai2:

• BBQ, a bias benchmark typically grouped with safety evaluations, aligned much more strongly with general reasoning in BenchMIRT's analysis — suggesting low scores may partly reflect difficulty reasoning through questions rather than safety behavior.
• WMDP, which tests dangerous dual-use knowledge in biology, chemistry, and cybersecurity, was found to be more strongly associated with general reasoning than safety. Stronger reasoning correlated with lower WMDP scores, since the benchmark treats refusal to provide dangerous information as the correct answer.
• HarmBench showed internal splits: standard and contextual harmful-request questions aligned with safety, while its copyright-related questions (e.g., reproducing song lyrics) aligned more closely with general reasoning.

Ai2 says these findings don't necessarily mean the benchmarks are flawed, but that a single aggregate score can combine multiple distinct signals that BenchMIRT can help separate.

Efficiency gains

Using BenchMIRT's question-level difficulty and discrimination estimates, Ai2 researchers ranked questions across the 16 benchmarks and found that keeping only 10% of the most informative questions largely preserved the same ranking of model strength on safety or reasoning as using the full question set. Keeping 50% of questions matched the full benchmark's measurements even more closely, according to the company.

The method could also predict how a model would perform on a benchmark question it had not been evaluated on, correctly predicting outcomes 79% of the time — compared to 70% for a baseline approach that assumes uniform performance across questions.

Ai2 has released a tech report, a Hugging Face data collection, and code on GitHub for BenchMIRT.

What this means

BenchMIRT is not a new model — it's a diagnostic tool for the benchmarks used to evaluate models, and its implications are significant for how the field interprets leaderboard scores. If widely adopted, it could change how researchers report results, potentially forcing benchmark aggregate scores to be broken into sub-scores by underlying capability rather than a single number. It also raises questions about specific benchmarks already embedded in industry practice: if WMDP and BBQ scores are driven substantially by reasoning ability rather than safety behavior, then models optimized for reasoning could appear artificially safer or less safe on these tests independent of actual safety improvements. The efficiency findings — that 10% of well-chosen questions can approximate a full benchmark — are also notable for reducing the compute cost of evaluation, though this remains an Ai2 research claim pending independent verification across other benchmark suites.