researchGoogle

Google adds Streaming DiLoCo to MaxText, syncing 1/65th of a model per step for cross-datacenter training

TL;DR

Google DeepMind and Google Research engineers describe how MaxText implements Streaming DiLoCo, a low-communication optimizer that splits a model into fragments and syncs one at a time. In a Qwen3-32B worked example, data parallel exchanges ~65.5 GB per step, while Streaming DiLoCo exchanges ~1.0 GB per step.

3 min read
0

Google has published an engineering walkthrough of Streaming DiLoCo in MaxText, its open-source JAX-based LLM training framework. The technique synchronizes one fragment of a model per step instead of the whole model at once, which Google says removes the network stalls that hit training across geographically distributed datacenters. The post, dated Nov. 19, 2026, comes from Google DeepMind and Google Research teams on the Google Developers Blog.

The problem: WAN stalls in data-parallel training

Standard data-parallel training synchronizes gradients at every step. Within a datacenter, JAX and XLA can often hide that all-reduce by overlapping it with backward-pass computation. Across datacenters or congested links, bandwidth drops and latency rises sharply. According to Google, standard data parallelism then stalls on every step waiting for global gradient synchronization, which collapses Model FLOPS Utilization (MFU).

How DiLoCo works

DiLoCo (Distributed Low-Communication) was introduced by Google DeepMind in the paper "DiLoCo: Distributed Low-Communication Training of Language Models." It partitions training across M independent compute groups called learners.

  • Inner loop: Each learner, typically a single TPU slice or an intra-datacenter multi-slice group, runs a standard optimizer (typically AdamW) on its local data for H steps. No communication occurs between DiLoCo groups during this interval. H was 37 to 65 in Google's experiments.
  • Outer loop: Every H steps, learners compute the delta between current weights and the global weights from the start of the interval. They average the deltas with a distributed all-reduce and apply an outer optimizer (typically SGD with Nesterov momentum).

Google says this communicates less often than synchronous training by a factor of H (1 to 2 orders of magnitude) while achieving comparable convergence and model quality.

Vanilla DiLoCo still sends the full set of weight deltas at once on every H-th step. On a thin network link, that creates a large periodic stall.

What Streaming DiLoCo changes

Streaming DiLoCo partitions parameters into P disjoint fragments and synchronizes each fragment exactly once over the H steps. The outer-loop cadence matches vanilla DiLoCo, without spiky bandwidth use.

  • Fragmentation: MaxText uses P = NLAYERS + 1: one fragment for non-scanned parameters (token embeddings, final layer norm, output head) and one per scanned Transformer decoder layer.
  • Staggered syncing: One fragment syncs every Δh = max(1, round(H/P)) steps. Setting H = P (Δh = 1) syncs one fragment per step.
  • Interleaved order: Fragments are not synced in adjacent-layer order. The sequence is interleaved across network depth (Fragment 0, then Layer k, then Layer 2k, and so on), so no section of the model stays stale while another updates.
  • Embedding load balancing: Fragment 0 scales with vocabulary size and can be much larger than the layer fragments. MaxText splits its large matrices evenly and merges them into the layer fragments to avoid a traffic spike when Fragment 0 syncs.

Worked example: Qwen3-32B

Google's roofline analysis uses Qwen3-32B in bfloat16 with a per-step compute time of about 6.2 s and P = H = 65:

Method Traffic pattern
Data parallel ~65.5 GB (~524 Gb) of gradients every step; needs at least ~85 Gbps sustained cross-slice bandwidth
Vanilla DiLoCo (H=65) 65x less total traffic; stalls on every 65th step while sending the full ~65.5 GB
Streaming DiLoCo (P=H=65) ~1.0 GB per step; ~65.5 GB over 65 steps, 65x less traffic than data parallel

On a constrained 10 Gbps inter-slice link, Google says transferring 65.5 GB would take over 50 seconds, causing major stalls in data-parallel training.

What this means

Vanilla and Streaming DiLoCo move the same total volume over an interval. The difference is the shape of the traffic: one large burst every H steps versus a near-uniform stream. For teams linking slices over WAN or congested links, the limiting factor shifts from peak bandwidth to average bandwidth. The Qwen3-32B example shows the gap: data parallel needs at least ~85 Gbps sustained, while Streaming DiLoCo's per-step payload is ~1.0 GB. Because the implementation is in open-source MaxText, the fragmentation and embedding load-balancing choices are available to anyone training on Google Cloud.

Related Articles

model release

Google releases Genie 4 to AI Ultra subscribers with real-time prompt editing and native audio

Google is rolling out Genie 4, which it calls its most realistic and interactive world model yet, to eligible Google AI Ultra subscribers in Project Genie. The update adds real-time native audio, mid-session text prompting, a world-building agent and a community gallery. Genie 3 also gains sessions of up to 3 minutes.

research

Ai2 launches Theorizer in Asta and open-sources the AI agent that proposes scientific theories

Ai2 launched Theorizer, an AI agent that combines findings from scientific papers to propose explanations for observations, inside its Asta platform. The code is open-source, and Ai2 covers token costs up to a limit in the web interface. A lung cancer case study shows the workflow.

research

Salesforce's BLIP-Live video agent cuts unnecessary responses by over 50%, company claims

Salesforce AI Research introduced BLIP-Live, a general model for live video agents that decides when to speak and when to stay silent. According to Salesforce, in one evaluation across 326 ego-centric videos it reduced unnecessary responses of state-of-the-art models by over 50%.

research

Tunix agentic GRPO lifts Gemma 4 E2B from 9.4% to 95% win rate in Plane Strike

Google's developer blog details how Tunix's agentic GRPO pipeline trained Gemma 4 E2B to play Plane Strike with tool calls. The win rate on held-out boards rose from 9.4% to 95% after 100 steps on a single Cloud TPU v6e-8 host.

Comments

Loading...