LLM dialogue comprehension failure modes
GPT-2 hallucinates speaker switches in dialogue, mirroring human Moses illusion PyDataTL;DW
- Language models hallucinate speaker transitions in dialogue, expecting speakers to alternate even when same speaker continues—likely because training data heavily emphasizes speaker switches, not because models truly understand conversational norms.
- Formal linguistic competence (grammar, syntax) does not equal functional competence in interactive dialogue; models can produce grammatical text while failing at turn-taking and speaker identity tracking.
- Language models hallucinates inputs—misinterpreting what was said—not just outputs; this input-level hallucination may explain some dialogue failures, mirroring human semantic illusions like the Moses illusion.
- Examining probability distributions and attention weights inside models can reveal rare failure cases without sampling thousands of outputs; internal structure analysis finds edge cases more efficiently than probabilistic sampling.
- Scaling model size alone does not fix speaker-tracking failures; GPT-2 exhibited this problem years ago, and larger modern models still struggle with same-speaker continuations, contradicting "bigger solves everything" assumption.
- Transformers predict only words explicitly, not speech acts or social intent that humans predict; humans predict conversation types (complaint, request, statement), adding representational layers transformers lack.
- Language models perform worse than humans at theory-of-mind tasks (reasoning about others' false beliefs), scoring at 4-5 year-old human level, limiting their dialogue competence in social interaction.
- Fine-tuning models on dialogue transcripts helps but doesn't eliminate speaker-hallucination errors; models matched human surprisal patterns for speaker switches but reversed expectations for same-speaker continuations.
- Adding long-term memory structures (RAG, GraphRAG) to transformer backbones could help close human-model gaps, but current architecture alone cannot replicate human attention-memory interactions that ground dialogue understanding.
- Human audiences unknowingly demonstrate the same semantic hallucination bias as language models; majority incorrectly recalled Popeye eating spinach for smartness instead of strength, showing humans hallucinate expected inputs too.
TL;DW
- Language models hallucinate speaker transitions in dialogue, expecting speakers to alternate even when same speaker continues—likely because training data heavily emphasizes speaker switches, not because models truly understand conversational norms.
- Formal linguistic competence (grammar, syntax) does not equal functional competence in interactive dialogue; models can produce grammatical text while failing at turn-taking and speaker identity tracking.
- Language models hallucinates inputs—misinterpreting what was said—not just outputs; this input-level hallucination may explain some dialogue failures, mirroring human semantic illusions like the Moses illusion.
- Examining probability distributions and attention weights inside models can reveal rare failure cases without sampling thousands of outputs; internal structure analysis finds edge cases more efficiently than probabilistic sampling.
- Scaling model size alone does not fix speaker-tracking failures; GPT-2 exhibited this problem years ago, and larger modern models still struggle with same-speaker continuations, contradicting "bigger solves everything" assumption.
- Transformers predict only words explicitly, not speech acts or social intent that humans predict; humans predict conversation types (complaint, request, statement), adding representational layers transformers lack.
- Language models perform worse than humans at theory-of-mind tasks (reasoning about others' false beliefs), scoring at 4-5 year-old human level, limiting their dialogue competence in social interaction.
- Fine-tuning models on dialogue transcripts helps but doesn't eliminate speaker-hallucination errors; models matched human surprisal patterns for speaker switches but reversed expectations for same-speaker continuations.
- Adding long-term memory structures (RAG, GraphRAG) to transformer backbones could help close human-model gaps, but current architecture alone cannot replicate human attention-memory interactions that ground dialogue understanding.
- Human audiences unknowingly demonstrate the same semantic hallucination bias as language models; majority incorrectly recalled Popeye eating spinach for smartness instead of strength, showing humans hallucinate expected inputs too.
Julia Mertens fine-tunes GPT-2 on dialogue data and measures surprisal against human reading times on controlled stimuli. The model treats natural same-speaker continuations as more surprising than incongruent ones—a reversal that persists regardless of scale and mirrors the Moses illusion, where prior representations override bottom-up input processing.