RAPO: Retrieval-Augmented Policy Optimization for LLM Agents

Researchers have proposed RAPO, a new reinforcement learning framework designed to improve how LLM agents explore and reason during multi-step problem-solving tasks.

The Problem with Current Agentic RL

Existing Agentic Reinforcement Learning methods rely solely on on-policy exploration, meaning agents can only learn from their own generated outputs. This approach restricts agents to discovering reasoning patterns from within their self-generated trajectories, limiting access to alternative problem-solving strategies that could improve performance.

While some recent methods have attempted to incorporate off-policy signals, they typically apply these signals at the trajectory level rather than at individual reasoning steps. This coarse-grained approach overlooks the fine-grained, step-by-step exploratory dynamics critical to how agents reason through complex tasks.

How RAPO Works

RAPO addresses this limitation by decomposing agentic RL training into two distinct phases:

Phase 1: Hybrid-Policy Agentic Rollout

Agents can now reason over retrieved off-policy step-level traces during rollout. This allows the reasoning receptive field to dynamically extend based on external behaviors, enabling broader exploration patterns conditioned on retrieved demonstrations.

Phase 2: Retrieval-Aware Policy Optimization

The framework calibrates policy gradient estimation through retrieval reward and importance shaping mechanisms. This approach stabilizes training and prioritizes exploration pathways that leverage retrieved information.

Empirical Results

Across fourteen datasets spanning three agentic reasoning tasks:

• Average performance gain: +5.0% compared to baseline agentic RL methods
• Training efficiency: 1.2x faster than existing approaches

The consistent improvements across multiple reasoning domains suggest the method generalizes beyond specific task types.

What This Means

RAPO offers a concrete methodology for enriching LLM agent training with retrieval-based exploration signals at the step level rather than trajectory level. The dual-phase approach targets a specific bottleneck in current agentic RL—the inability to systematically learn from alternative reasoning paths. The 5% gain may appear modest, but empirical consistency across fourteen datasets indicates stable, reproducible improvements. Faster training (1.2x) also reduces computational costs for agentic RL development. However, the paper doesn't specify whether these gains hold on proprietary reasoning benchmarks or only academic datasets, and it remains unclear how the approach scales with larger model sizes or more complex multi-tool environments.