Apple researchers train 8B UI agent with online RL on live iOS apps, tying Gemini-3.1-Pro

Apple researchers report that an 8B UI navigation agent trained with online reinforcement learning on live iOS apps outperforms Gemini-3.5-Flash and ties with Gemini-3.1-Pro on real iOS applications. The paper, "Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps," was published in November 2026 on Apple's Machine Learning Research site.

The authors are Peitong Duan and Seyed Khashaiar Gatmiry (equal contribution), Jeff Nichols, Emmanuel Abbe and Samy Bengio.

The problem

The authors say frontier vision-language models handle UI navigation well, but their inference costs prohibit everyday use. Smaller models trained with supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, which the authors say leads to brittle execution and an inability to recover from errors.

The method

The framework trains agents directly on live iOS applications. It has these components:

• Decoupled architecture: the agent is split into a Planning agent and a UI Grounding model.
• Dual reward signal: an LLM-as-a-Judge supplies dense, step-level feedback on intermediate progress, plus a terminal reward for overall task completion.
• Optimization: Proximal Policy Optimization (PPO) with step-level advantage estimation.
• GiGPPO: a new neighbor-conditioned value estimation method. According to the authors, it improves critic stability by using reward signals from visually similar states across concurrent trajectories.
• Infrastructure: a system that orchestrates hundreds of concurrent virtual iOS devices to support online training at scale.

Reported results

The authors evaluated on UINavBench, iOSWorld and AndroidWorld. They report that their online RL agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence.

Two results are called out for the 8B RL agents:

• Nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments.
• Better performance than Gemini-3.5-Flash and a tie with Gemini-3.1-Pro on real iOS applications.

These are the authors' own reported results. Model weights, code, pricing and context window are not stated.

What this means

The paper targets the cost problem the authors identify: frontier VLMs are capable at UI navigation but expensive to run. Their reported result is that an 8B agent can match Gemini-3.1-Pro on real iOS apps, which would make a small model a candidate for everyday UI automation.

The reward design addresses the brittleness the authors attribute to SFT and offline RL. Training on live apps with step-level feedback exposes the agent to its own errors and recoveries, rather than fixed trajectories. The authors report that dense step-level rewards beat terminal-only RL, and that the agents generalize from iOS training to Android far better than the SFT baseline.

The scale requirement is also notable. Online training depends on running hundreds of concurrent virtual iOS devices, so the infrastructure is part of the contribution alongside the algorithm.