LLM evaluation
3 articles tagged with LLM evaluation
Simon Willison's Pelican Benchmark Shows GPT-6 Astra Outperforming GPT-5.6 Sol at Every Reasoning Level
Developer Simon Willison ran his signature 'pelican riding a bicycle' SVG test on newly-accessed GPT-6 Astra across five reasoning levels, comparing results against GPT-5.6 Sol, Terra, and Luna. Even Astra's lowest reasoning setting reportedly beat every Sol output, though Astra costs roughly twice as much per token.
Artificial Analysis Launches Optima, a Platform to Build Custom AI Benchmarks on Your Own Data
Artificial Analysis has launched Optima, a platform that lets users build custom AI benchmarks using their own data, workflows, or use-case descriptions. Unlike public benchmarks, Optima compares models on cost per task and time per task in addition to quality.
Ai2's TutorMoments Benchmark Finds LLMs Over-Help Students, Rarely Push for Rigor
Ai2's new TutorMoments framework replays real tutoring transcripts to test whether LLMs make the right pedagogical call at key decision points. Across seven models tested, all defaulted to over-helping unless explicitly prompted about the scaffolding-versus-rigor trade-off.