Alibaba's Qwen team develops algorithm that doubles reasoning chain length in math problems
Alibaba's Qwen team has developed Future-KL Influenced Policy Optimization (FIPO), a training algorithm that assigns different weights to tokens based on their influence on subsequent reasoning steps, rather than treating all tokens equally. Testing on Qwen2.5-32B-Base showed reasoning chains double from ~4,000 to 10,000+ tokens, with AIME 2024 accuracy improving from 50% to 58%, outperforming Deepseek-R1-Zero-Math-32B (47%) and OpenAI's o1-mini (56%). The team plans to open-source the system.
Alibaba's Qwen Team Doubles Reasoning Chain Length With Token-Weighted Training Algorithm
Alibaba's Qwen team has developed Future-KL Influenced Policy Optimization (FIPO), a training algorithm that fundamentally changes how reinforcement learning assigns credit to individual tokens during reasoning model training. The breakthrough addresses a critical limitation: standard RL approaches reward all tokens equally, even though some steps are far more influential than others in determining reasoning quality.
The Problem With Current Credit Assignment
When language models learn to reason through reinforcement learning, they typically receive a simple pass/fail judgment at the sequence end, with that reward spread evenly across every token. A comma gets the same credit signal as a pivotal logical step. This flat reward structure causes reasoning chains to plateau—models learn to extend thoughts to a certain length and then stagnate.
Previous attempts to solve this relied on PPO-based methods using auxiliary value models pre-trained on synthetic chain-of-thought data. The Qwen team argues this makes it impossible to determine whether performance gains come from the algorithm or from leaked outside knowledge.
How FIPO Works
FIPO calculates each token's cumulative probability shift across all downstream tokens. Instead of judging a token in isolation, the algorithm asks: How does model behavior change after this token appears? Tokens initiating productive reasoning chains receive higher rewards. Tokens sending the model toward dead ends receive less.
Critically, FIPO requires no auxiliary model, eliminating knowledge leakage while matching PPO-based performance. The algorithm includes stability guardrails: a discount factor ensures nearby tokens carry more weight (their influence is easier to predict), and filters remove tokens where model drift between training steps exceeds thresholds. Without filtering, training crashed around step 70.
Benchmark Results
Testing on Qwen2.5-32B-Base (no prior long-CoT exposure) using only the public DAPO dataset:
- Reasoning chain length: DAPO stalls at ~4,000 tokens; FIPO reaches 10,000+
- AIME 2024 accuracy: 50% → 58% (outperforms Deepseek-R1-Zero-Math-32B at 47%, matches o1-mini at 56%)
- AIME 2025 accuracy: 38% → 43%
- Distribution shift: Entire length distribution shifted upward, not just outliers
Emergent Self-Verification Behavior
The model naturally developed four distinct reasoning phases during training without explicit instruction. Early phases produced shallow templates and linear chains. By phase three, the model spontaneously double-checked intermediate results using different approaches (switching from algebraic to geometric interpretation, for example). Phase four showed systematic multi-pass verification with step-by-step recalculation.
The researchers note this mirrors inference-time scaling strategies in OpenAI's o-series and Deepseek-R1, but emerges purely through reinforcement learning without synthetic long-CoT pre-training.
Significant Limitations
FIPO has only been validated on mathematical tasks. Testing scope was limited to:
- Single dataset (DAPO)
- Base models without long-CoT pre-training
- Mathematical problems only
Generalization to code, symbolic logic, or other domains remains unproven. Extended reasoning sequences increase compute costs. Additionally, a performance gap persists compared to distilling from larger teacher models—pure RL teaches models less than direct instruction from stronger ones.
What This Means
FIPO addresses a genuine bottleneck in reasoning model training by fixing how credit flows during RL. The algorithm's ability to achieve results without auxiliary models strengthens the case for pure RL approaches and may influence how other teams structure reasoning training. However, the math-only validation significantly limits claims about broader applicability. The planned open-source release could accelerate testing on other domains, but teams will need substantial additional work to determine whether FIPO's benefits transfer beyond mathematics.
Related Articles
Ai2 launches Theorizer in Asta and open-sources the AI agent that proposes scientific theories
Ai2 launched Theorizer, an AI agent that combines findings from scientific papers to propose explanations for observations, inside its Asta platform. The code is open-source, and Ai2 covers token costs up to a limit in the web interface. A lung cancer case study shows the workflow.
Salesforce's BLIP-Live video agent cuts unnecessary responses by over 50%, company claims
Salesforce AI Research introduced BLIP-Live, a general model for live video agents that decides when to speak and when to stay silent. According to Salesforce, in one evaluation across 326 ego-centric videos it reduced unnecessary responses of state-of-the-art models by over 50%.
Tunix agentic GRPO lifts Gemma 4 E2B from 9.4% to 95% win rate in Plane Strike
Google's developer blog details how Tunix's agentic GRPO pipeline trained Gemma 4 E2B to play Plane Strike with tool calls. The win rate on held-out boards rose from 9.4% to 95% after 100 steps on a single Cloud TPU v6e-8 host.
Google adds Streaming DiLoCo to MaxText, syncing 1/65th of a model per step for cross-datacenter training
Google DeepMind and Google Research engineers describe how MaxText implements Streaming DiLoCo, a low-communication optimizer that splits a model into fragments and syncs one at a time. In a Qwen3-32B worked example, data parallel exchanges ~65.5 GB per step, while Streaming DiLoCo exchanges ~1.0 GB per step.
Comments
Loading...