Researchers propose VCPO to stabilize asynchronous RL training for LLMs, cutting training time 2.5x

Asynchronous RL Training Hits Variance Wall

Reinforcement learning has become standard for improving LLM reasoning performance, but asynchronous training—which processes stale rollouts in parallel for throughput—introduces a critical stability problem: policy-gradient estimates develop extremely high variance.

When models train on old rollouts from earlier policy versions, importance ratios become heavy-tailed. This means a tiny fraction of samples disproportionately dominates gradient updates, creating noisy and unstable learning compared to on-policy training.

Root Cause: Effective Sample Size Collapse

Researchers diagnosed the problem across math and general reasoning benchmarks. Training collapse was reliably predicted by two metrics: effective sample size (ESS) dropping sharply and gradient norms becoming unstable. The issue affects widely-used critic-free methods like REINFORCE and GRPO, which lack built-in safeguards for off-policy training.

Previous stabilization attempts—masking high-variance samples or clipping gradients—only partially addressed the problem. Algorithmic variants provided marginal improvements.

VCPO: Explicit Variance Control

The proposed Variance Controlled Policy Optimization (VCPO) takes a direct approach with two components:

1. Adaptive learning rate scaling: Scales learning rates based on effective sample size, automatically dampening updates when variance is high and samples are unreliable.

2. Closed-form minimum-variance baseline: Applies an off-policy variance reduction baseline without requiring an auxiliary value model, adding minimal computational overhead.

Crucially, this is a general stabilization method compatible with existing REINFORCE and GRPO implementations.

Benchmark Results Across Tasks

Empirical evaluation shows VCPO substantially improves robustness for asynchronous training across three task categories: math problems, general reasoning, and tool-use scenarios. The method outperformed a broad baseline suite including masking/clipping stabilizers and algorithmic variants.

On long-context, multi-turn reasoning tasks, VCPO reduced training time by 2.5x while matching the performance of fully synchronous training. This is the key practical win: asynchronous training typically sacrifices performance for speed, but VCPO recovers that performance without reverting to slower synchronous training.

What This Means

This addresses a real production constraint in LLM scaling. As companies push to increase RL training throughput—necessary for training multiple models in parallel—they hit stability walls. VCPO provides a method-agnostic solution that maintains performance while keeping asynchronous training's speed advantage.

The work demonstrates that explicit variance control, rather than ad-hoc stabilization heuristics, is fundamental for reliable asynchronous RL at scale. For teams running large RL experiments, this technique could immediately improve training efficiency and stability without architectural changes.