Diffusion Language Models Show Lower Memorization Than Autoregressive Models

A new research paper presents the first systematic characterization of memorization in diffusion language models (DLMs), finding they leak significantly less personally identifiable information (PII) than standard autoregressive language models (ARMs).

Key Findings

The study, posted on arXiv (2603.02333), establishes a monotonic relationship between sampling resolution and memorization: increasing the resolution of diffusion sampling strictly increases the probability of extracting exact training data verbatim. Critically, autoregressive decoding represents a limiting case of diffusion-based generation when sampling resolution is set to maximum.

Under equivalent prefix-conditioned evaluation conditions, the researchers found that DLMs exhibit "substantially lower memorization-based leakage of personally identifiable information" compared to ARMs. This distinction matters for privacy and copyright concerns, as verbatim reproduction of training data has been a persistent issue in large language models.

Theoretical Framework

The researchers propose a generalized probabilistic extraction framework that unifies prefix-conditioned decoding and diffusion-based generation under arbitrary masking patterns and stochastic sampling trajectories. Theorem 4.3, a core theoretical contribution, formally establishes the sampling resolution-memorization relationship.

This theoretical grounding allows predictions to be made about memorization behavior across different sampling strategies—something previously impossible since diffusion language models' fundamentally different generation dynamics made direct comparison to autoregressive models difficult.

Experimental Validation

Extensive experiments across multiple model scales and sampling strategies validate the theoretical predictions. The researchers evaluated both the memorization quantitatively and the practical implications for PII leakage, demonstrating that the theoretical framework holds empirically.

What This Means

Diffusion language models appear to offer a genuine privacy advantage over autoregressive models, not just in theory but in practice. For organizations concerned about training data reproduction and compliance with privacy regulations, DLMs present a measurable alternative worth considering. However, this doesn't eliminate memorization risk entirely—it reduces it. The work also clarifies the relationship between sampling decisions and memorization risk, giving researchers a tool to optimize privacy-utility tradeoffs when designing diffusion-based language models.

The findings suggest that architectural choices—not just training data filtering—can meaningfully impact privacy leakage, opening a new dimension in responsible AI development.