research

Alibaba's HopChain framework fixes vision model failures in multi-step reasoning tasks

TL;DR

Researchers from Alibaba's Qwen team and Tsinghua University developed HopChain, a framework that automatically generates multi-step image questions to fix how vision-language models fail during complex reasoning tasks. The method improved 20 out of 24 tested benchmarks by forcing models to re-examine images at each reasoning step, preventing early perceptual errors from cascading through subsequent steps.

3 min read
0

Alibaba's HopChain Framework Fixes Vision Model Failures in Multi-Step Reasoning

Alibaba's Qwen team and Tsinghua University researchers identified and addressed a fundamental weakness in vision-language models (VLMs): their inability to maintain accuracy across multiple consecutive reasoning steps about images.

The Problem: Cascading Errors

VLMs consistently fail on tasks requiring extended chain-of-thought reasoning about images. A single perceptual error early in the reasoning chain—miscounting objects, confusing spatial relationships, misreading text, or hallucinating details—cascades through all subsequent steps, producing entirely incorrect final answers.

In documented examples:

  • A model miscounted dots on ladybugs by one dot each across five beetles, compounding into a significantly wrong total
  • A model correctly identified a car's position but misread its movement direction in a parking sequence
  • A model pointed to the wrong arc in an astronomical diagram, leading to an incorrect season identification

Existing training data for Reinforcement Learning with Verifiable Rewards (RLVR) rarely includes tasks demanding sustained visual attention across multiple steps, leaving this vulnerability unaddressed.

HopChain's Four-Stage Approach

The framework automatically generates multi-step image questions where each step forces models to re-examine the image closely. The data generation pipeline operates in four stages:

  1. Object identification: Qwen3-VL-235B-A22B-Thinking identifies object categories in images
  2. Instance localization: Meta's SAM3 segmentation model locates individual object instances
  3. Question generation: The language model builds multi-level questions around combinations of three to six objects
  4. Human verification: Four independent human annotators solve each question; only questions with unanimous agreement advance to training

This process generates 60,000 to 80,000 training examples per model. Each question ends with a unique number serving as an automatic answer verification mechanism, with some chains involving up to six linked reasoning steps through arithmetic, counting, text recognition, and spatial reasoning.

Benchmark Results

Researchers trained two models using HopChain:

  • Qwen3.5-35B-A3B (smaller model)
  • Qwen3.5-397B-A17B (larger model)

HopChain improved 20 out of 24 tested benchmarks across four categories: STEM and puzzles, general image comprehension, text recognition/document comprehension, and video comprehension.

Specific gains on the smaller model:

  • EMMA: 53 → 58
  • CharXiv: 69 → 73.1

Gains on the larger model:

  • BabyVision: 28.61 → 32.22
  • ZeroBench: 4 → 8

For particularly long reasoning chains, the larger model showed accuracy improvements exceeding 50 percentage points. Notably, despite training exclusively on images, both models improved on five out of six video benchmarks, suggesting the skills transfer beyond static images.

Ablation Study: Full Chains Required

An ablation study across five representative benchmarks demonstrated that complete question chains are essential:

  • Full chains: 70.4 average score
  • Halved chains: 66.7 average score
  • Single-step questions: 64.3 average score

The error breakdown shows HopChain addresses all error categories proportionally—perception, logic, knowledge, and hallucination errors all improved comparably, with the error distribution of fixed issues tracking the original error profile closely.

Known Limitations

The pipeline requires SAM3 to recognize and segment objects, excluding images without clear segmentable objects from data generation. This limitation reflects a broader weakness: recent research shows even frontier models struggle with visual perception. Moonshot AI's WorldVQA benchmark found top-scoring models correctly identified less than 50% of depicted objects, while systematically overestimating accuracy. A Stanford analysis found frontier models achieve 70-80% of their image benchmark scores without seeing images at all, confidently hallucinating visual details.

What This Means

HopChain demonstrates that vision model reasoning failures aren't inherent but rather stem from inadequate training data targeting multi-step visual reasoning. The framework's broad improvements across unrelated benchmarks—without task-specific optimization—suggest the approach addresses fundamental visual understanding gaps. However, the work also surfaces deeper vulnerabilities: models' inability to reliably segment and recognize objects, and their tendency to hallucinate visual details while appearing confident. These limitations suggest current VLMs require more fundamental improvements in visual perception before complex reasoning becomes truly reliable.

Related Articles

research

Ai2 launches Theorizer in Asta and open-sources the AI agent that proposes scientific theories

Ai2 launched Theorizer, an AI agent that combines findings from scientific papers to propose explanations for observations, inside its Asta platform. The code is open-source, and Ai2 covers token costs up to a limit in the web interface. A lung cancer case study shows the workflow.

research

Salesforce's BLIP-Live video agent cuts unnecessary responses by over 50%, company claims

Salesforce AI Research introduced BLIP-Live, a general model for live video agents that decides when to speak and when to stay silent. According to Salesforce, in one evaluation across 326 ego-centric videos it reduced unnecessary responses of state-of-the-art models by over 50%.

research

Tunix agentic GRPO lifts Gemma 4 E2B from 9.4% to 95% win rate in Plane Strike

Google's developer blog details how Tunix's agentic GRPO pipeline trained Gemma 4 E2B to play Plane Strike with tool calls. The win rate on held-out boards rose from 9.4% to 95% after 100 steps on a single Cloud TPU v6e-8 host.

research

Google adds Streaming DiLoCo to MaxText, syncing 1/65th of a model per step for cross-datacenter training

Google DeepMind and Google Research engineers describe how MaxText implements Streaming DiLoCo, a low-communication optimizer that splits a model into fragments and syncs one at a time. In a Qwen3-32B worked example, data parallel exchanges ~65.5 GB per step, while Streaming DiLoCo exchanges ~1.0 GB per step.

Comments

Loading...