Google's Tunix hits bitwise-identical sampler and trainer logprobs for on-policy RL on 64 TPU v5p chips
Google engineers report a zero train-inference mismatch (Zero-TIM) implementation in Tunix, in which sampler and trainer assign bitwise-identical logprobs to the same token at the same weights. It was verified at multi-host scale across 64 TPU v5p chips (DP8×TP8). The recipe combines fixed-order Pallas kernels, explicit cross-shard reductions, a shared logprob path and custom differentiation rules.
Google has published a zero train-inference mismatch (Zero-TIM) implementation in Tunix that makes the sampler and trainer assign bitwise-identical logprobs to the same action and token history in LLM reinforcement learning. According to the Google Developers Blog post (Nov. 16, 2026), it was verified at multi-host scale across 64 TPU v5p chips (DP8×TP8). Google describes the result as state-of-the-art, true on-policy stability for TPU RL training. The post is by Yuxuan Zhang, Lin Chai, Lance Wang, Tianshu Bao and Weiren Yu.
The problem
True on-policy learning requires every action in an update to be generated and evaluated by the same policy. Production RL stacks run sampling and training through separate execution paths. The sampler picks tokens during incremental decoding and records a behavior logprob. The trainer re-evaluates the token in batched forward-backward passes. With identical weights, the two paths can still produce different logprobs, which the authors call a hidden off-policy gap that compounds over multi-turn trajectories.
The cause is floating-point arithmetic. A reduction is a tree of additions, and floating-point addition is not associative, so changing the tree changes where rounding occurs. The effect is easier to see in bfloat16. In the post's example, near 2.0 each 0.0078125 input is half the distance to the next representable bfloat16 value, so adding inputs one at a time discards them, while grouping them first reaches the next value. In the original stack, sampler decode and trainer forward used different token-array shapes and forward implementations, and the trainer forward ran inside a differentiated JAX program.
The alignment contract
Sampler decode, serving prefill and trainer forward must produce bitwise-identical logprobs for every sampled action. The comparison is of raw floating-point bytes under the same rollout weights, token history and sampling settings, with no tolerance. Prefill serves as a middle checkpoint:
- Sampler decode vs. prefill checks that the serving engine agrees with itself between incremental generation and full-sequence re-scoring.
- Prefill vs. trainer checks that serving and training agree across compilation and differentiation boundaries.
The four-part recipe
Zero-TIM does not merge the sampler and trainer into one engine. The recipe has four parts:
- Within-shard arithmetic. Batch-invariant Pallas kernels fix accumulation order and intermediate precision in each tensor-parallel rank. Projection kernels traverse fixed contraction blocks in order, independent of token-row count, and accumulate partial products in float32 with explicit output casts. This covers the attention and MLP projections and the LM head. Dedicated kernels handle RMSNorm reductions and SwiGLU intermediate casts. The existing Pallas paged-attention forward is retained.
- Cross-shard reductions. For row-parallel projections, ppermute transports partial outputs, and every rank adds them in the same source-rank order. The order is explicit in the program rather than chosen by a collective implementation.
- Shared action-logprob path. Decode, prefill and trainer use matching sampling transforms and the same fixed Pallas log-softmax construction. Prefill and trainer score the action generated during rollout, and sampled token IDs are preserved across environment turns.
- Differentiated serving forward. The trainer reuses the serving prefill computation. Custom vector-Jacobian products (jax.custom_vjp) keep the serving Pallas kernels for forward values and supply backward rules through a differentiable replica that mirrors the kernel's accumulation and casts. The flag --xla_allow_excess_precision=false is passed through XLA_FLAGS to constrain excess intermediate precision in surrounding operations.
Long trajectories are evaluated in fixed-size chunks that carry the paged KV cache forward. All action logprobs feed the complete train-batch RL loss, and the reverse pass propagates gradients through the cache connections before the optimizer update.
What this means
The work targets a specific failure mode: a synchronous RL loop can be on-policy by schedule yet off-policy numerically. Google's approach removes the mismatch at the bit level rather than tolerating or correcting it, and keeps separate serving and training pipelines. The three-path check (decode, prefill, trainer) also localizes where a numerical gap enters, and the authors present it as a diagnostic any RL stack can adopt. The verified configuration is 64 TPU v5p chips with data parallelism of 8 and tensor parallelism of 8, so results at other scales or topologies are not stated in the post.
Related Articles
Google Adds Shareable Interactive Sites to Gemini Notebook, Rolling Out Globally on Web
Google announced on November 16, 2026 that Gemini Notebook can turn a notebook's sources into interactive sites. Outputs now include custom visual explainers, dashboards and project trackers, and the feature is rolling out globally on the web.
Google adds 66 regional dialects to Gemini voice in Android and iOS apps
Google is adding 66 regional dialects to Gemini, starting today in the Android and iOS apps. The voices cover UK accents such as Geordie, Scouse and Brummie, with tailored dialects for India, Brazil, Indonesia and Korea. Google says it worked with native speakers on the voices.
NVIDIA recipe: RL post-training lifts Nemotron 3.5 Super VL accuracy a further 17% on drug-metabolism task
NVIDIA has published a recipe for post-training Nemotron 3.5 Super VL on scientific tasks using NeMo Gym and NeMo RL. In a drug-metabolism example, supervised fine-tuning improved accuracy by 12% and RL post-training lifted it a further 17% at a 16,384-token output budget.
Gemini 4 Carbon tops MathArena at 85.4%, ahead of Claude Fable 5.2 and GPT-6.1 Sol
Google's Gemini 4 Carbon (high) leads the MathArena math benchmark with an expected performance of 85.4%, ahead of Anthropic's Claude Fable 5.2 (max) at 82.8% and OpenAI's GPT-6.1 Sol (max) at 80.6%. With the reasoning settings used in testing, Carbon costs roughly 15 times as much as GPT-6.1 Sol.
Comments
Loading...