Anthropic Study Shows LLMs Transfer Hidden Biases Through Distillation

Student LLMs inherit undesirable traits from teacher models at rates exceeding 60%, even when those traits are scrubbed from training data, according to peer-reviewed research from Anthropic published in Nature.

The study examined model distillation, a technique where smaller "student" models learn from larger "teacher" models' outputs. Researchers used GPT-4.1 nano as a reference model, training teacher models to prefer specific animals or trees, then used numerical outputs from those teachers to train student models.

Experimental Results

When tested in natural language, student models selected the teacher's preferred animal or tree far more frequently than the base model: owl preference rates increased from 12% to over 60%. Similar effects appeared when training data consisted of code or chain-of-thought reasoning traces rather than numerical outputs.

The bias transfer persisted even when:

• Training datasets were screened to remove direct references to the trait
• Content was semantically unrelated to the preference
• Multiple data filtering techniques were applied

Anthropic researcher Alex Cloud and colleagues termed this phenomenon "subliminal learning" — student models pick up subtle statistical signatures from teacher outputs that cause trait inheritance invisible in the training data itself.

Industry Context

Model distillation has grown increasingly common as developers face shrinking training data availability and seek to reduce inference costs and latency from large models, according to Oskar Hollinsworth and Samuel Bauer of AI research nonprofit FAR.AI.

The mechanism behind subliminal learning remains not fully understood. The research suggests teacher model outputs contain statistical patterns that students detect and replicate, independent of semantic content.

Safety Implications

The Anthropic paper states: "Safety evaluations may therefore need to examine not just behavior, but the origins of models and training data and the processes used to create them."

This finding adds a new dimension to AI safety concerns as the industry increasingly trains models on outputs from other models rather than human-generated data.

What This Means

The research reveals a significant blind spot in current AI safety practices. Organizations using distillation cannot rely on training data inspection alone to verify safety properties — they must also audit source model behaviors and distillation processes. This complicates the already challenging task of AI safety evaluation and may require new testing methodologies that examine model lineage and transfer learning patterns beyond visible training data.