XYBench

Ai2 released XYBench, a benchmark of 8,115 real-world questions that tests whether AI assistants recognize when a user's request rests on a misconception. Five evaluated models, including Claude Sonnet 4.6 and GPT-5, were less likely than human reference answers to steer users toward their actual goal. The paper won the Best Resource Paper Award at EMNLP 2026.