Google releases 29 public tasks from 85-task engineering benchmark PLE and opens agent competition
Google AI & Infrastructure announced on November 3, 2026 that it is releasing 29 public tasks, grading tools and an evaluation container for Physical-Engineering's Last Exam (PLE). The 85-task benchmark was developed with Mercor and comes with an open agent competition and leaderboard.
Google AI & Infrastructure announced on November 3, 2026 that it is releasing 29 public tasks, grading tools and an evaluation container for Physical-Engineering's Last Exam (PLE), an 85-task engineering benchmark developed with Mercor. The release comes with an open agent competition and a leaderboard, according to a report from Unite.AI.
What PLE tests
PLE asks an agent to complete professional assignments in mechanical, electrical, civil and structural engineering. According to the report, this differs from textbook-style questions and from isolated tool calls.
What is being released
- 29 public tasks
- Grading tools
- An evaluation container
- An open agent competition
- A leaderboard
The full benchmark contains 85 tasks. Further details on scoring, participation rules and results were not available in the summary of the report.
What this means
PLE frames engineering evaluation around complete professional assignments across four disciplines, not question answering or single tool calls. Publishing grading tools and an evaluation container alongside the 29 public tasks gives outside developers materials to run agents against those tasks, and the open competition and leaderboard provide a venue for comparing agents.
Related Articles
OpenAI publishes LifeVal, an evaluation built partly from a 3,995-respondent survey to test AI on everyday tasks
OpenAI published LifeVal on November 2, 2026, an evaluation built partly from a 3,995-respondent survey. It tracks AI model performance over time on practical tasks such as finding childcare, creating budgets and ordering groceries.
Gemini 4 Carbon ranks second on SemiAnalysis GDPx at just over $16 per task, behind Claude Opus 5.5
SemiAnalysis's new GDPx benchmark places Google's Gemini 4 Carbon just behind Anthropic's Claude Opus 5.5, at just over $16 per task against about $51. Removing the cybersecurity, biology and alignment domains reshuffles the leaderboard.
Google-led StreamVibe video agent decides when to speak up, but all tested models degrade over long videos
A research team led by Google, with Stanford, Oxford and several U.S. universities, introduced StreamVibe, a video agent that decides on its own when to speak. Its StreamVibe-VL model beats open-source baselines, but every tested system loses accuracy as videos get longer.
Ai2's StateBench: Frontier models fall to 13% or lower on hardest long-input state tracking
Ai2 released StateBench, a benchmark with four difficulty levels for how language models track entity state changes in long documents. Four frontier models scored well on the two easier levels but fell to 13% or lower on the longest general finite problems. Recurrent and hybrid models trained directly on the task generalized much farther than transformers.
Comments
Loading...