Study Finds AI Models' Reasoning Steps Leave Distinct Fingerprints in Internal Activations

The Finding

A new study from South Korea's KAIST and Naver AI Lab shows that the written reasoning steps language models produce correspond to distinct, separable patterns in their internal activations. The researchers defined eight recurring reasoning operations—extraction, decomposition, formula recall, deduction, and computation among them—and tested whether these operations could be told apart inside a model's numerical representations, not just in its text output.

The answer: yes, and the signal is strongest in the middle layers of the network.

Methodology

The team had three models—Qwen2.5-7B, Qwen3-8B, and Gemma4-31B—solve math problems. They split each solution path into segments and used GPT-5 to label each segment with one of the eight reasoning operations. They then trained classifiers to distinguish these operations based on internal activations at different layers.

A classifier that only looked at the tokens used performed worse than one analyzing internal representations, and position within the solution path didn't explain the separation either. This indicates the internal states carry information about the type of reasoning step beyond surface-level wording.

Same Words, Different Representations

Common function words like "a," "is," or "the" appear across very different reasoning steps. In early layers, their representations remain jumbled together, but by the middle and later layers, they separate according to the surrounding reasoning operation. The same word gets a different internal representation depending on which reasoning step it's embedded in.

The researchers also tested whether a reasoning step forms independently of context. By blocking attention to the preceding 30 tokens through a targeted intervention, they found the signal for that operation weakened—meaning reasoning steps build on preceding context rather than emerging in isolation.

Notably, the separability held even on incorrectly solved problems. A flawed computation step still looked like a computation step internally, even when the result was wrong.

Generalization and Limits

The findings replicated with Llama-3-8B, and for Qwen3-8B, classifiers trained on the original data transferred successfully to GPQA-Diamond and MATH-500. According to the researchers, this suggests the pattern isn't limited to one model family or one benchmark.

However, the experiments are limited to math tasks and a small number of models. Whether these findings can be used to catch errors or steer a model mid-generation remains an open question, according to the study's authors.

Why This Matters for AI Safety

The relationship between text output and internal computation is directly relevant to AI safety oversight. OpenAI has stated that reading a model's chain of thought is one of the few oversight tools currently available. But Anthropic has shown that models disclose the hints they actually used in only 25 to 39 percent of cases, raising questions about how faithfully written reasoning reflects internal computation.

Separately, a method that translates a model's internal vectors into readable text reportedly revealed that Anthropic's Claude Opus 4.6 processes more information than shows up in its output reasoning. And OpenAI's Astra model reportedly uses a "Recurrent Depth" technique that shifts part of the reasoning process into internal numerical representations—the same space this KAIST study investigates.

What This Means

This research adds a data point to an increasingly important debate: how much can we trust a model's written chain of thought as a window into what it's actually doing? The KAIST findings suggest internal states do track distinct reasoning operations in a structured way, which is a prerequisite for building tools that monitor or intervene on reasoning mid-generation. But the gap between "operations are separable internally" and "we can reliably detect errors or deception from activations" remains wide. The study is confined to math tasks and small-to-mid-size models; whether the same structure holds for open-ended reasoning, multi-turn agentic tasks, or larger frontier models is untested. For now, this is foundational interpretability work, not a deployable safety tool.