agentic retrieval vs. naive RAG
Better retrieval recall hurts agent performance; hard distractors are the culprit Plain Schwarz (Berlin Buzzwords, Haystack)TL;DW
- Agentic retrieval differs fundamentally from human search: agents inject results directly into context in parallel, unable to skip irrelevant documents like humans do.
- Hard distractors—plausible but wrong evidence—cause nonlinear accuracy damage; the first distractor injected has disproportionate impact due to attention mechanism constraints.
- Better relevance scores can paradoxically hurt agent performance when hard distractors enter context, because LLMs cannot reason themselves out of false information once included.
- Optimize for precision and utility-aware ranking (UDCG), not traditional recall metrics (NDCG), to minimize false positives in agentic loops.
- Context window scaling doesn't solve token competition problem: even with 1M token capacity, attention dilutes quadratically, limiting practical usable context.
- Agents retrieving through code-based tools (grep, find) outperformed traditional search engines on HotpotQA by 8%, suggesting agent-controlled retrieval beats black-box ranking.
- Treat agentic retrieval as context admission control, not document ranking—verify evidence correctness before injecting into agent context.
- Give agents low-level retrieval primitives (search, filter, iteration tools) rather than black-box search engines, enabling them to orchestrate information discovery safely.
- Agents need semantic understanding of their task plan to identify hard distractors; retrieval systems cannot determine relevance without higher-level goal context.
- Reranking pipelines alone cannot remove hard distractors; only agents with full context can discern whether retrieved passages genuinely support their goals.
TL;DW
- Agentic retrieval differs fundamentally from human search: agents inject results directly into context in parallel, unable to skip irrelevant documents like humans do.
- Hard distractors—plausible but wrong evidence—cause nonlinear accuracy damage; the first distractor injected has disproportionate impact due to attention mechanism constraints.
- Better relevance scores can paradoxically hurt agent performance when hard distractors enter context, because LLMs cannot reason themselves out of false information once included.
- Optimize for precision and utility-aware ranking (UDCG), not traditional recall metrics (NDCG), to minimize false positives in agentic loops.
- Context window scaling doesn't solve token competition problem: even with 1M token capacity, attention dilutes quadratically, limiting practical usable context.
- Agents retrieving through code-based tools (grep, find) outperformed traditional search engines on HotpotQA by 8%, suggesting agent-controlled retrieval beats black-box ranking.
- Treat agentic retrieval as context admission control, not document ranking—verify evidence correctness before injecting into agent context.
- Give agents low-level retrieval primitives (search, filter, iteration tools) rather than black-box search engines, enabling them to orchestrate information discovery safely.
- Agents need semantic understanding of their task plan to identify hard distractors; retrieval systems cannot determine relevance without higher-level goal context.
- Reranking pipelines alone cannot remove hard distractors; only agents with full context can discern whether retrieved passages genuinely support their goals.
Lester Solbakken (Plain Schwarz) shows that agents inject all retrieved context into their reasoning loop, making precision and distractor avoidance more critical than recall. Hard distractors—topically plausible but factually wrong documents—steal softmax attention from correct answers, and context accumulates across iterations causing 'context rot.' NDCG optimizes for human scanning, not agent synthesis.