NVIDIA Shows Task-Seeded Synthetic Data Boosts Nemotron-3 Nano by +11.1 on GPQA

NVIDIA researchers published results showing that task-seeded synthetic Q&A generation improved Nemotron-3 Nano performance across multiple benchmarks in a 100B-token continuation experiment. According to NVIDIA, the approach delivered an +11.1 point improvement on GPQA, +1.8 on MMLU-Pro, +1.9 on average code tasks, and +1.6 on commonsense understanding, while maintaining stable average math performance.

The Approach

The pipeline uses training splits from approximately 70 public task datasets covering roughly 700 subtasks from lm-eval-harness as "capability seeds." NVIDIA emphasized that held-out evaluation and test data were excluded from generation.

The seed pool comprised two groups:

• Knowledge-intensive tasks: 39 tasks, approximately 300 subtasks, roughly 3M seed samples covering factual, scientific, multilingual, and domain-specific Q&A
• Reasoning-intensive tasks: 34 tasks, approximately 400 subtasks, roughly 1.5M seed samples covering analytical reasoning, logic, math, code, and commonsense reasoning

The five-stage process:

1. Collect seed tasks with suitable training splits
2. Normalize heterogeneous task records into unified JSONL schema
3. Generate similar examples that preserve underlying capabilities while changing content
4. Enrich answers with reasoning, knowledge, or context
5. Filter through schema checks, format validation, deduplication, and task-specific answer verification

Technical Details

NVIDIA stores semantic answer text rather than only option labels. For example, the system records "dirt trapped under the fingernails" instead of just "B" to provide clearer training signals.

Multiple-choice tasks are easier to verify directly, while generation-style tasks require more cautious task-specific handling, according to the researchers.

For Nemotron Ultra and Super pretraining runs, NVIDIA used a license-compatible subset of the generated data suitable for commercial model training.

The Transfer Learning Rationale

NVIDIA frames the approach through transfer learning across task families. The researchers argue that models can learn reusable behaviors from broad seed tasks and apply them to related applications and evaluations.

According to NVIDIA, the pipeline strengthens behaviors that appear across many tasks: identifying information needs, applying domain knowledge, separating plausible alternatives, following response constraints, executing multi-step reasoning, and grounding answers in context.

The researchers cite earlier evidence from Nemotron Nano pretraining, where AGIEval training data improved MMLU-Pro performance, suggesting that structured Q&A data from one task family can improve behavior outside the original task scope.

What This Means

This research demonstrates measurable gains from structured synthetic data generation during pretraining, not just post-training. The +11.1 point GPQA improvement is particularly notable for a 100B-token continuation experiment. The approach addresses a specific data quality problem: models may see abundant raw text during pretraining but still lack explicit examples of how information requests are structured and resolved. NVIDIA's results suggest that task-seeded synthetic data can fill this gap without requiring models to memorize evaluation datasets directly, though the technique requires careful filtering and verification infrastructure.