Parametric vs. contextual generalization in LLMs
Stanford study finds LLMs reverse and compose facts in-context but fail when fine-tuned Stanford OnlineTL;DW
- Language models generalize better from in-context information than from fine-tuned parameters—reversals and syllogisms hit 99% accuracy in context but near chance after fine-tuning.
- The reversal curse persists even when training models from scratch on synthetic data, showing it's a fundamental limitation of parametric learning, not just fine-tuning.
- In-context learning succeeds because structural patterns (reversals, logical implications) appear frequently in natural training data, making them learnable as flexible procedures.
- Parametric learning ties knowledge to explicit surface forms in training data, while in-context learning preserves richer detail that enables flexible reuse.
- Offline data augmentation—using the model's in-context reasoning to generate latent inferences and adding them to training data—matches or exceeds pure in-context performance.
- Test-time episodic retrieval (bringing learned documents back into context via oracle memory) restores generalization on reversal and other latent-structure tasks.
- Reinforcement learning can train models to regenerate needed information via chain-of-thought, generalizing reasoning patterns to new domains, though it struggles with symmetry-breaking tasks like reversals.
- These three methods trade off compute cost: offline augmentation is expensive to train but cheap at test time; test-time retrieval is cheap to train but expensive at inference.
- Hippocampus-like episodic memory and cortex-like parametric learning are complementary in humans—fast episodic storage plus slow parametric integration prevents interference while enabling rapid single-trial learning.
- Parametric systems need statistical structure to efficiently constrain inference; pure symbolic reasoning lacks feasibility for real-world generalization, requiring the models learn content-sensitive reasoning patterns.
TL;DW
- Language models generalize better from in-context information than from fine-tuned parameters—reversals and syllogisms hit 99% accuracy in context but near chance after fine-tuning.
- The reversal curse persists even when training models from scratch on synthetic data, showing it's a fundamental limitation of parametric learning, not just fine-tuning.
- In-context learning succeeds because structural patterns (reversals, logical implications) appear frequently in natural training data, making them learnable as flexible procedures.
- Parametric learning ties knowledge to explicit surface forms in training data, while in-context learning preserves richer detail that enables flexible reuse.
- Offline data augmentation—using the model's in-context reasoning to generate latent inferences and adding them to training data—matches or exceeds pure in-context performance.
- Test-time episodic retrieval (bringing learned documents back into context via oracle memory) restores generalization on reversal and other latent-structure tasks.
- Reinforcement learning can train models to regenerate needed information via chain-of-thought, generalizing reasoning patterns to new domains, though it struggles with symmetry-breaking tasks like reversals.
- These three methods trade off compute cost: offline augmentation is expensive to train but cheap at test time; test-time retrieval is cheap to train but expensive at inference.
- Hippocampus-like episodic memory and cortex-like parametric learning are complementary in humans—fast episodic storage plus slow parametric integration prevents interference while enabling rapid single-trial learning.
- Parametric systems need statistical structure to efficiently constrain inference; pure symbolic reasoning lacks feasibility for real-world generalization, requiring the models learn content-sensitive reasoning patterns.
Controlled experiments on facts, syllogisms, and encodings show fine-tuned models fail to reverse relations or compose logical chains, while the same models nearly ace both tasks given the data in context. Three mitigations tested: offline data augmentation, episodic retrieval at inference time, and RL-driven regeneration, each trading training cost for inference cost.