researchGoogle

Tunix agentic GRPO lifts Gemma 4 E2B from 9.4% to 95% win rate in Plane Strike

TL;DR

Google's developer blog details how Tunix's agentic GRPO pipeline trained Gemma 4 E2B to play Plane Strike with tool calls. The win rate on held-out boards rose from 9.4% to 95% after 100 steps on a single Cloud TPU v6e-8 host.

3 min read
0

Google's Tunix agentic reinforcement learning pipeline raised the Plane Strike win rate of google/gemma-4-E2B-it from 9.4% to 95% on held-out boards after 100 steps of agentic GRPO, according to a Google developer blog post published November 19, 2026 by developer advocate Wei Wei. Training ran on a single Cloud TPU v6e-8 host.

The results

Out of the box, Gemma 4 E2B in thinking mode wins 9.4% of games on held-out boards, averaging 22.44 strikes per game. After 100 steps of agentic GRPO (Group Relative Policy Optimization) with Tunix, the win rate reaches 95% and the average game length falls to about 16 strikes.

The post describes the model as having decent spatial reasoning and tool-calling capabilities before training.

The task

Plane Strike is played on a 6x6 grid (A1 to F6). One 8-cell plane is hidden in one of 4 orientations. Each turn, the agent sees the result of its previous strike and the raw visible board (. untried, X hit, O miss), reasons through candidate placements, and calls strike(cell="...") to hit one cell. A game ends when all 8 plane cells are hit or after 23 strikes. The post revisits a game from an earlier TensorFlow/JAX reinforcement learning blog, replacing the custom MLP policy with Gemma 4 E2B.

Pipeline components

The pipeline has four modular parts:

  • Task environment: PlaneStrikeEnv subclasses Tunix's BaseTaskEnv. A seed fixes the hidden plane position so all 8 rollouts in a GRPO group play the same board. Each step returns EnvStepResult(observation, reward, done, info), and step rewards are summed into a trajectory-level reward.
  • Tool and chat template parser: StrikeTool extends BaseTool, and a subclass of Gemma4ChatTemplateParser appends tool declarations to the system prompt and renders tool messages as in-turn responses.
  • Conversation agent: PlaneStrikeAgent extends ConversationAgentBase and maps environment observations to model messages.
  • Trajectory collection and learner: TrajectoryCollectEngine and GRPOLearner handle rollouts and policy updates.

Gemma 4 tool-calling format

Gemma 4 declares tool schemas in the system prompt inside <|tool>declaration:...<tool|>. A model turn that calls a tool contains a thought channel followed by a <|tool_call>call:strike{...}<tool_call|> block, then <|tool_response>. The post identifies <|tool_response> as one of Gemma 4's native stop tokens: the model pauses there, receives the environment response, and continues its thought and next tool call within the same assistant turn.

The agent ignores everything inside the thought channel and parses only the final answer. Only a final answer containing a single valid strike call counts as a valid coordinate. Anything else returns INVALID_ACTION.

Rollout configuration

Training uses Tunix's asynchronous vLLM rollout engine with the JAX TPU backend, in server mode with prefix caching enabled and bfloat16 dtype. The configured KV cache size is the maximum prompt length plus the maximum response length plus 256. EOS tokens include the tokenizer EOS, <turn|> and the tool-response start token. The visible config also sets final_logit_softcapping to 30.0 through Hugging Face overrides.

For standard multi-tool agents, tunix.rl.agentic also provides ToolEnvironment, ToolAgent and ToolManager abstractions, so custom message and observation handling is only needed for custom turn control. The complete code is in a GitHub repository linked from the post. It follows two earlier posts on FunctionGemma fine-tuning and single-turn GRPO with Tunix.

What this means

The post shows a complete multi-turn, tool-using training loop on one TPU v6e-8 host, from environment definition through tool-call parsing to GRPO updates. The reported gain, 9.4% to 95% win rate in 100 steps, comes from a single game environment with a small action space (6x6 grid, one tool), so it demonstrates the pipeline rather than general agent capability. The handling of Gemma 4's native stop token is the practical detail: it lets one assistant turn span several tool exchanges, and the agent must track whether the model is awaiting a tool response. Builders who need less custom plumbing can start from the prebuilt ToolEnvironment, ToolAgent and ToolManager classes.

Comments

Loading...