LLM evaluation
4 articles tagged with LLM evaluation
Ai2 releases FairStress: 13M-case benchmark finds LLM fairness gaps hidden by final answers
Ai2 has released FairStress, a benchmark of 300 decision scenarios across 29 domains that expands to more than 13 million test cases. It measures whether LLMs change well-supported decisions more readily for one demographic group than another when challenged. Ai2 reports the gap appears as identity becomes more explicit.
Simon Willison's Pelican Benchmark Shows GPT-6 Astra Outperforming GPT-5.6 Sol at Every Reasoning Level
Developer Simon Willison ran his signature 'pelican riding a bicycle' SVG test on newly-accessed GPT-6 Astra across five reasoning levels, comparing results against GPT-5.6 Sol, Terra, and Luna. Even Astra's lowest reasoning setting reportedly beat every Sol output, though Astra costs roughly twice as much per token.
Artificial Analysis Launches Optima, a Platform to Build Custom AI Benchmarks on Your Own Data
Artificial Analysis has launched Optima, a platform that lets users build custom AI benchmarks using their own data, workflows, or use-case descriptions. Unlike public benchmarks, Optima compares models on cost per task and time per task in addition to quality.
Ai2's TutorMoments Benchmark Finds LLMs Over-Help Students, Rarely Push for Rigor
Ai2's new TutorMoments framework replays real tutoring transcripts to test whether LLMs make the right pedagogical call at key decision points. Across seven models tested, all defaulted to over-helping unless explicitly prompted about the scaffolding-versus-rigor trade-off.