Hallucination as incentive misalignment
OpenAI finds evaluation rubrics, not training, drive LLM hallucinations Simons Institute for the Theory of ComputingTL;DW
- Hallucinations in language models stem from test-taking incentives: models optimize for accuracy benchmarks without reward signals for admitting uncertainty, unlike humans who learn humility from real-world consequences.
- Open rubric evaluation—explicitly stating scoring rules in prompts—aligns developer incentives with humble behavior; models respond immediately by saying 'I don't know' more when given credit for doing so.
- Simple consistency check reduces hallucinations: query model twice, use third call to verify agreement; if inconsistent, output 'I don't know' instead of guessing.
- Current accuracy-only benchmarks penalize humility and create a false trade-off between correctness and reduced hallucinations; this single metric drives deployment of overconfident models across all major LLM providers.
- Language models are miscalibrated and overconfident; on SimpleQA benchmark, even giving 90% reward for saying 'I don't know' still beats model accuracy scores, revealing systematic miscalibration.
- Hallucinations are not inevitable—they're a solvable mechanism design problem, not an inherent limitation of next-token prediction or model capacity.
- Existing hallucination-reduction techniques (consistency checking, retrieval, self-critique) are already published and effective; the bottleneck is incentive structures, not algorithmic solutions.
- Open rubrics are more objective and transparent than closed rubrics; they enable fair grading when developers and evaluators agree on scoring, unlike real-world chat where users don't state reward functions.
TL;DW
- Hallucinations in language models stem from test-taking incentives: models optimize for accuracy benchmarks without reward signals for admitting uncertainty, unlike humans who learn humility from real-world consequences.
- Open rubric evaluation—explicitly stating scoring rules in prompts—aligns developer incentives with humble behavior; models respond immediately by saying 'I don't know' more when given credit for doing so.
- Simple consistency check reduces hallucinations: query model twice, use third call to verify agreement; if inconsistent, output 'I don't know' instead of guessing.
- Current accuracy-only benchmarks penalize humility and create a false trade-off between correctness and reduced hallucinations; this single metric drives deployment of overconfident models across all major LLM providers.
- Language models are miscalibrated and overconfident; on SimpleQA benchmark, even giving 90% reward for saying 'I don't know' still beats model accuracy scores, revealing systematic miscalibration.
- Hallucinations are not inevitable—they're a solvable mechanism design problem, not an inherent limitation of next-token prediction or model capacity.
- Existing hallucination-reduction techniques (consistency checking, retrieval, self-critique) are already published and effective; the bottleneck is incentive structures, not algorithmic solutions.
- Open rubrics are more objective and transparent than closed rubrics; they enable fair grading when developers and evaluators agree on scoring, unlike real-world chat where users don't state reward functions.
Hallucinations persist because accuracy-only metrics give models no reward for admitting uncertainty. Stating grading rules in prompts—open rubrics—shifts model behavior: when "I don't know" earns partial credit, models become calibrated and outperform baselines on both accuracy and hallucination rate.