Google-led StreamVibe video agent decides when to speak up, but all tested models degrade over long videos

A research team led by Google has introduced StreamVibe, a video agent designed to decide for itself when to speak up while a user works. The accompanying tests show how hard that remains: every tested model loses accuracy as videos get longer and errors compound, according to the paper, as reported by The Decoder.

Researchers from Google, Stanford, Oxford and several U.S. universities built an assistant that watches tasks such as cooking or assembling furniture and intervenes only when needed. It is meant to handle vague requests like "Please help me if I run into any difficulties." According to the paper, existing video assistants are built for explicit commands such as "Let me know when the water starts boiling," so they rarely pick up unspoken intentions. They also struggle to track what has been done, which leads to repeated advice or late responses, and they often stick to the original instruction when a user changes the request midway.

Benchmark and model

The team compiled a benchmark of about 4,100 examples drawn from more than 183 hours of everyday-life and software recordings. Gemini 2.5 Flash suggested when to offer help and what to say, and four human evaluators had to unanimously approve each example. The test covers three stages: clear instructions, vague goals, and requests that change over time. In one example, a user disassembling a toy excavator asks for progress updates, then switches to a quality check after three minutes.

The researchers then trained StreamVibe-VL, a model apparently based on Alibaba's Qwen3.5-9B. It decides every second whether to stay quiet or speak. A second training stage uses reinforcement learning to make it more reserved so it does not interrupt constantly. A separate memory system tracks pending requests and completed steps, while the model alone decides whether to speak and what to say.

Results

According to the researchers, StreamVibe-VL improves response timing and quality by 10.7 percent over other models. The memory system is designed to add another 12.3 percent in proactive interaction and reduce unnecessary interruptions, among other effects. StreamVibe-VL generally beats its base model and other open-source models by a wide margin, though Google's proprietary Gemini 3.8 Flash scores higher in several tests. The paper says the specialized training does not hurt general video comprehension.

Performance drops when models process a whole video and build on their own earlier responses. A key content metric for responses falls by up to 61.5 percent based on the system's own history. In videos longer than five minutes, models rarely pick the right moment and struggle to track a changed instruction over a long stretch. The memory harness mainly reduces unnecessary interruptions.

In a GIMP tutorial on converting a photo to black-and-white, the comparison model JoyAI claims the finished image has already been saved before the user has opened it. StreamVibe-VL instead gives the next step at the right moment, such as "Colors" and then "Desaturate." In a rolled-toast recipe, other models describe the screen, while StreamVibe-VL advises cutting off the crust first.

The researchers acknowledge that real-world situations are likely more complex than the test setup. Future work will examine how user preferences and the environment influence each other over time. Model weights, pricing and context window are not stated.

What this means

The results separate two problems: deciding whether a single moment warrants a prompt, which models often handle, and sustaining that judgment across a continuous stream, where all tested systems fail. The reported drop of up to 61.5 percent on a key content metric, and the poor timing past five minutes, point to compounding self-generated history as the main weakness. The memory harness mostly curbs interruptions rather than fixing timing. For builders of proactive assistants, long-horizon state tracking is the open problem, not short-window detection. Gemini 3.8 Flash outscoring StreamVibe-VL in several tests also shows the gap between a 9B-class open model with specialized training and a proprietary model remains.