Researchers use LLMs to simulate misinformation susceptibility across demographics with 92% accuracy

Researchers have introduced BeliefSim, a simulation framework that uses Large Language Models to predict demographic misinformation susceptibility by treating beliefs as the primary driver of how different groups respond to false claims.

The Problem

Misinformation affects different demographic groups unequally due to variations in underlying beliefs and worldviews. Understanding these susceptibility patterns is critical for developing targeted counter-misinformation strategies, but directly measuring real-world vulnerability across populations is expensive and ethically complex. The researchers propose using LLM simulation as a scalable alternative.

BeliefSim Framework

The framework operates in two main stages:

Belief Profile Construction: The team creates demographic belief profiles using psychology-informed taxonomies and survey priors. These profiles encode beliefs as conditioning inputs rather than treating demographic groups as monolithic categories.

LLM Conditioning: The researchers tested two approaches for incorporating beliefs into LLMs:

• Prompt-based conditioning: directly instructing models to adopt specific belief profiles
• Post-training adaptation: fine-tuning models on belief-tagged data

Evaluation Results

The framework was evaluated on two dimensions:

Susceptibility Accuracy: BeliefSim achieved up to 92% accuracy in predicting whether a belief-conditioned LLM would be susceptible to specific misinformation claims. Accuracy improved when beliefs were explicitly incorporated as conditioning signals rather than relying solely on demographic labels.

Counterfactual Demographic Sensitivity: The researchers tested whether the framework could accurately model how susceptibility changes when demographic characteristics change while beliefs remain constant—validating that beliefs, not surface-level demographics, drive the simulation results.

Key Findings

Across both datasets and modeling strategies, beliefs provided a strong prior for simulating misinformation susceptibility. The framework demonstrates that LLMs can be used to model belief-driven reasoning patterns, offering a more nuanced approach than treating demographic groups as uniform categories.

The research treated beliefs as the causal mechanism rather than demographics as proxies, which improved both accuracy and interpretability. This approach aligns with psychological research showing that susceptibility to misinformation varies primarily with underlying worldviews rather than demographic identity alone.

What This Means

BeliefSim offers researchers a scalable method to test misinformation interventions across diverse belief systems without recruiting human subjects—potentially accelerating research on counter-misinformation strategies. The 92% accuracy suggests LLMs can meaningfully simulate belief-driven reasoning patterns.

However, the framework's real-world applicability depends on how well synthetic belief profiles capture actual human diversity. The reliance on "psychology-informed taxonomies" means results are bounded by the completeness of those taxonomies. Additionally, simulating susceptibility is distinct from predicting it in real humans, and the framework should be validated against actual human responses before deployment in misinformation research or policy.

The work contributes to understanding LLM behavioral simulation capabilities while highlighting both the potential and limitations of using language models to model human cognitive vulnerabilities.