item response theory

Ai2 has released BenchMIRT, a technique that uses multidimensional item response theory to analyze which underlying capabilities drive scores on individual benchmark questions. Trained on 100 LLMs across 16 benchmarks and 34,000+ questions, it found that benchmarks like BBQ and WMDP measure general reasoning more than safety, despite being marketed as safety evaluations.