google-research

Google published COSMO, an experimental AI assistant application for Android, on the Play Store. The 1.13 GB app runs Gemini Nano locally and includes 14 automated skills ranging from calendar event scheduling to document writing and deep research.

A Google Research and Rochester Institute of Technology study reveals that standard AI benchmarking practices using three to five human evaluators per test example systematically underestimate human disagreement and produce unreliable model comparisons. The researchers found that at least ten raters per example are needed for statistically reliable results, and that budget allocation between test examples and raters matters as much as total budget size.