Research reveals LLMs internalize logic as geometric flows in representation space

LLMs Develop Geometric Logic Structures Independent of Semantics

A new arXiv paper proposes that large language models think through their representation spaces as flowing trajectories, offering a geometric framework for understanding reasoning that challenges the "stochastic parrot" critique.

Researchers disentangled logical structure from semantics by testing the same natural deduction propositions with different semantic carriers—asking whether LLMs truly internalize logic or simply recognize surface patterns. The answer appears to be the former.

The Geometric Framework

The theory models LLM reasoning as smooth flows in representation space, where:

• Position corresponds to conceptual location in embedding space
• Velocity reflects how logical statements guide reasoning progression
• Curvature measures the geometry of reasoning trajectories

Logical statements act as local controllers of flow velocities, creating measurable geometric patterns independent of surface-level semantics.

What the Experiments Show

Using learned representation proxies, researchers designed controlled experiments across Qwen and LLaMA model families. Key findings:

1. LLM reasoning corresponds to identifiable smooth flows in representation space
2. The same logical structures produce consistent geometric patterns regardless of semantic content
3. These geometric patterns emerge from next-token prediction training alone, without explicit logic instruction
4. The underlying representational laws appear general and possibly universal across architectures

The universality claim is significant: the geometric signatures of reasoning appear largely independent of specific training recipes or model architectures, suggesting a fundamental principle of how neural networks encode logical relationships.

Implications for LLM Interpretability

This geometric perspective offers new tools for studying reasoning phenomena. Instead of treating representation space as an opaque black box, researchers can now:

• Visualize reasoning trajectories as geometric objects
• Quantify logical operations through formal geometric analysis
• Test whether reasoning generalizes across model families
• Potentially predict or control reasoning behavior through geometric manipulation

The framework directly addresses the "stochastic parrot" argument—the claim that LLMs merely pattern-match without true understanding. If logical reasoning truly emerges as higher-order geometry in representation space, that suggests something more structured than surface-level pattern matching occurs during inference.

Limitations and Open Questions

The paper doesn't specify:

• Which specific model sizes or scales show these geometric properties
• Whether the framework extends to more complex reasoning tasks beyond natural deduction
• How training objectives other than next-token prediction affect geometric structure
• Whether explicit logic training enhances or alters these geometric flows

What This Means

This research provides both theoretical grounding and practical interpretability tools for understanding LLM reasoning. The geometric framework suggests LLMs don't just memorize logical patterns—they encode logical relationships as structural features of their representation geometry. This is substantively different from pattern matching and may explain why next-token prediction, despite appearing trivial, produces systems capable of coherent reasoning.

For practitioners, the framework offers a new lens for debugging reasoning failures and potentially improving model behavior through representation-space analysis rather than just prompt engineering. For the field, it bridges symbolic logic and neural representations in a way that's both mathematically formal and empirically testable.