Pre-training strategies for stronger reasoning
Nvidia finds front-loading reasoning data in pre-training yields 60% cumulative gain on LLMs Stanford OnlineTL;DW
- Two-phase pre-training strategy: Phase 1 emphasizes data diversity (web crawl + reasoning data); Phase 2 focuses exclusively on high-quality sources (math, code, Wikipedia). Volta improves 17% over random ordering.
- Frontloading reasoning data during pre-training yields durable advantages: models seeing reasoning data early gain 16% post-pretraining, 9.3% post-SFT, and 19% after full RLHF—gains compound rather than wash away.
- High-quality reasoning data in pretraining unlocks hidden gains in posttraining: small high-quality (SHQ) + large diverse quality (LDQ) datasets show no benefit immediately but deliver 4.25% boost after SFT.
- Early reasoning cannot be replicated by more SFT compute: models without reasoning-based pretraining trail reasoning-based models by 12% even with 2x SFT epochs and matched data budgets.
- RLP (Reinforcement Learning on Pretraining) uses dense, verifier-free information-gain rewards during pretraining instead of sparse binary rewards, achieving 19% base model improvement and 8% improvement after identical posttraining.
- RLP scales efficiently across model sizes and architectures: NeMoTron Nano 12B sees 35% gains using only 250M RLP tokens versus 20T token baseline; benefits persist after SFT with 3% absolute margin.
- RLP outperforms RPT (Reinforcement Pretraining) by 4% because it applies dense per-token rewards on all positions without external verifier, capturing full reasoning signal versus ignoring reasoning steps.
- Data quality estimation uses automated classifiers (Fine-Web EDU, Essential Web) scoring documents 1-5 on educational value, enabling systematic weighting of datasources in optimal blend.
- Epoch estimation determines how many repeats of each datasource maximize downstream performance: some datasets hit diminishing returns at 2 repeats, others sustain 4-6 repeats before gains plateau.
- RLP maintains 14% advantage over next-token prediction in flop-matched settings where baseline sees 35x more data, demonstrating data-efficient reasoning emergence without task-specific reasoning datasets.
TL;DW
- Two-phase pre-training strategy: Phase 1 emphasizes data diversity (web crawl + reasoning data); Phase 2 focuses exclusively on high-quality sources (math, code, Wikipedia). Volta improves 17% over random ordering.
- Frontloading reasoning data during pre-training yields durable advantages: models seeing reasoning data early gain 16% post-pretraining, 9.3% post-SFT, and 19% after full RLHF—gains compound rather than wash away.
- High-quality reasoning data in pretraining unlocks hidden gains in posttraining: small high-quality (SHQ) + large diverse quality (LDQ) datasets show no benefit immediately but deliver 4.25% boost after SFT.
- Early reasoning cannot be replicated by more SFT compute: models without reasoning-based pretraining trail reasoning-based models by 12% even with 2x SFT epochs and matched data budgets.
- RLP (Reinforcement Learning on Pretraining) uses dense, verifier-free information-gain rewards during pretraining instead of sparse binary rewards, achieving 19% base model improvement and 8% improvement after identical posttraining.
- RLP scales efficiently across model sizes and architectures: NeMoTron Nano 12B sees 35% gains using only 250M RLP tokens versus 20T token baseline; benefits persist after SFT with 3% absolute margin.
- RLP outperforms RPT (Reinforcement Pretraining) by 4% because it applies dense per-token rewards on all positions without external verifier, capturing full reasoning signal versus ignoring reasoning steps.
- Data quality estimation uses automated classifiers (Fine-Web EDU, Essential Web) scoring documents 1-5 on educational value, enabling systematic weighting of datasources in optimal blend.
- Epoch estimation determines how many repeats of each datasource maximize downstream performance: some datasets hit diminishing returns at 2 repeats, others sustain 4-6 repeats before gains plateau.
- RLP maintains 14% advantage over next-token prediction in flop-matched settings where baseline sees 35x more data, demonstrating data-efficient reasoning emergence without task-specific reasoning datasets.
Three strategies compound: a two-phase quality-aware curriculum, front-loading math and code before post-training (16-19% gains that survive SFT and RL), and RLP—which reframes pre-training as RL with dense information-gain rewards. RLP alone hits 35% improvement on a 12B model using 200B fewer tokens than baseline.