Anthropic's Fable 5.2 tops Remote Labor Index at 41.3%, completing real freelance jobs at professional quality
Anthropic's Fable 5.2 scores 41.3 percent on the Remote Labor Index from Scale Labs and the Center for AI Safety, which measures how often AI agents complete real freelance jobs well enough for a client to accept. The next-best systems, Opus 5.5 and GPT 6 Astra, score 21.25 and 20.83 percent. The top score at the benchmark's launch about a year ago was 2.5 percent.
Anthropic's Fable 5.2 leads the Remote Labor Index (RLI) with 41.3 percent, according to results reported by The Decoder. The benchmark, built by Scale Labs and the Center for AI Safety (CAIS), measures how often AI agents complete real freelance jobs at a quality a paying client would accept.
Leaderboard
- Fable 5.2 (Anthropic): 41.3%
- Opus 5.5 (Anthropic): 21.25%
- GPT 6 Astra (OpenAI): 20.83%
- Gemini Argon 4 (Google), fifth place: 17.08%
- Fable 5.1 (Anthropic): just under 18%
- Fable 5 (Anthropic), July measurement: about 16%
- GPT-5.2 and Gemini 3 Pro: below 3%
When the benchmark launched about a year ago, the top system, Manus, completed 2.5 percent of tasks. The Decoder reports that Fable 5.2 now completes more than 16 times as many professional jobs, and roughly two and a half times Fable 5's July score. Fable 5.2 also completes more than twice as many jobs as its direct predecessor, Fable 5.1.
Where the gains came from
Scale Labs said on X that Fable 5.2's largest gains came in editing audio and music, designing architecture and interior spaces, and creating 3D models. The agent also handles long, complex projects that earlier top-ranked models regularly failed. Nearly 50 of its successful completions were jobs that no AI agent in the test had solved before.
Scale Labs says agents still barely handle industrial product design and animations. When agents fail, researchers say they typically deliver amateur-quality work, incomplete results, broken files or contradictory designs.
How the benchmark works
The RLI dataset contains 240 completed projects from 358 verified professionals on Upwork. Together the jobs are worth about $144,000. A single project costs an average of $632.60 and takes a human freelancer just under 29 hours on average. Tasks range from video editing and CAD design to game development and data analysis.
Agents work in a virtual workspace with professional software such as Blender, GIMP and Audacity, plus specialized AI tools for image, audio and video generation. Trained human experts open the submitted files in the relevant software and compare them directly with the human freelancer's delivered work.
CAIS has tried to automate that review with another AI and failed so far. According to the report, AI reviewers can barely operate the professional software, are fooled by polished but technically flawed submissions, and rate models too generously.
What this means
The RLI scores whole work assignments judged against what a human freelancer delivered, not isolated skills. That makes 41.3 percent a measure of accepted deliverables, and it also means most jobs in the set remain uncompleted by every system tested. The weak areas Scale Labs names, industrial product design and animations, show where agents still fail.
The scoring depends on trained human experts opening files in professional software, because automated AI review has so far overrated flawed work. Fable 5.2's lead over Opus 5.5 and GPT 6 Astra is large on this benchmark. The results come from one test built on 240 Upwork projects.
Related Articles
Ethan Mollick: Anthropic's Claude Fable 5.2 is "almost" AGI, with gaps in taste and self-directed judgment
Ethan Mollick, writing in One Useful Thing, says Anthropic's Claude Fable 5.2 is "almost" AGI, which he calls Jagged AGI. After early access, he reports expert-level results on hardware hacking, business planning and research-heavy web projects, with persistent gaps in taste and judgment.
Anthropic says to start with Opus 5.5 at medium effort, reserves Fable 5.2 for hardest tasks
Anthropic published guidance on November 9 (US time) on how to choose among Fable 5.2, Opus 5.5, Sonnet 5.5 and Haiku 5.5 in Claude Code and Claude Platform. It recommends trying Opus 5.5 at medium effort first. Fable 5.2, the top model announced the same day, is positioned for advanced coding, finance and science work.
None of 135 AI agent submissions pass experts' bar on open national-lab research in 'Into the Unknown'
A Carnegie Mellon-led team built "Into the Unknown," a benchmark of open research questions from US Department of Energy national labs. According to the paper, none of the 135 submissions from nine AI agents met all requirements, and GPT-6 Astra scored best.
OpenAI introduces BioBasicsBench, a 100-task benchmark for AI agents in everyday wet-lab science
OpenAI has introduced BioBasicsBench, a benchmark of 100 tasks. It measures how well AI agents interpret raw experimental data and handle everyday wet-lab science.
Comments
Loading...