benchmarkOpenAI

GPT-6.2 Astra scores 41.2% on XDOF's Ember-0 robot benchmark, up from 19.6% for GPT-6 Astra

TL;DR

Robotics infrastructure company XDOF reports that GPT-6.2 Astra reached a 41.2% average success rate across the 16 tasks of its Ember-0 benchmark, versus 19.6% for GPT-6 Astra. It is the first model to complete rope knot-tying and T-shirt unfolding on the benchmark. Average prediction latency fell from 15 seconds to 10.1 seconds.

3 min read
0

GPT-6.2 Astra reached a 41.2% combined success rate on XDOF's Ember-0 robot manipulation benchmark, up from 19.6% for GPT-6 Astra two months earlier, according to results published by XDOF and reported by Chinese outlet QbitAI (量子位) in an article written in Chinese.

The results

XDOF, a robotics infrastructure company, ran both models on Ember-0, a benchmark for real physical environments in which a bimanual (dual-arm) robot performs 16 manipulation tasks. The 41.2% figure is an equal-weighted average across the 16 shared tasks. The benchmark covers spatial reasoning, object grasping, two-arm coordination, tool use, fine manipulation and handling of flexible objects. QbitAI reports the gain as 21.6 percentage points.

Two tasks that no AI model had completed before were solved: tying a knot in rope and unfolding a T-shirt. Ember-0 had previously logged 143 valid T-shirt unfolding trials without a single success from any model.

The largest single-task gain was picking up bread with tongs, where success rose from 10% for GPT-6 Astra to 90% for GPT-6.2 Astra. QbitAI also lists improvements on:

  • Physical reasoning: pulling a towel to retrieve a distant plush toy, and tilting a tray to roll a ball into a bowl.
  • Memory and perception: finding an occluded marker, and following hidden instructions on a red sticky note.
  • Fine force control: moving a chip without crushing it, and carrying cereal onto a tray.

Test protocol

XDOF ran trials on six YAM workstations in parallel. Each task was repeated 30 times, allowing 480 trials per day. Human annotators labeled sub-steps, grasp failures and collisions to filter out hardware bias and random noise. XDOF said in a post that the Ember-0 benchmark may soon be saturated by model capability.

Latency

Average prediction latency fell from 15 seconds for GPT-6 Astra to 10.1 seconds for GPT-6.2 Astra. QbitAI notes that this inference approach remains less efficient than other model architectures, and that roughly 10 seconds is still long for robots that need continuous perception and fast response. The article also describes the test hardware as lacking tactile feedback, with the arm unable to adjust in real time while waiting for the next action chunk.

Reactions and other results

OpenAI Developers' official account reposted XDOF's results. OpenAI co-founder and president Greg Brockman wrote that GPT-6.2 Astra performs very well in robotics.

Separately, Jiafei Duan, a presidential young professor at the National University of Singapore, said that in his team's real-robot comparison tests GPT-6.2 Astra reached first place in the RoboColosseum SO-101 Elo ranking within a day.

QbitAI places the result in a broader trend, citing Nvidia researchers testing GPT-6 Astra directly as a robot policy, Claude showing complex manipulation in robot simulation environments, and Google DeepMind extending Gemini's multimodal understanding to robot control.

What this means

The gain on Ember-0 is spread across the 16-task suite rather than concentrated in one task, and it includes deformable-object tasks that no model had completed in the benchmark's earlier trials. Those tasks require tracking changing object shape, coordinating two arms and planning over long horizons.

The results also carry limits. A 41.2% average means most attempts across the suite still fail, and prediction latency of 10.1 seconds is still a constraint for real-time manipulation. XDOF's own comment that Ember-0 may soon saturate suggests the benchmark may need harder tasks to keep distinguishing models.

Comments

Loading...