prompt injection unsolvable by design
AI red teaming can't eliminate prompt injection — only shrink the blast radius NDC ConferencesTL;DW
- AI red teaming is fundamentally different from traditional security testing: attacks manipulate behavior, not exploit code bugs, with zero repeatability even at temperature zero.
- LLMs hallucinate convincingly and reliably produce false outputs; black-box red teaming is useless without accessing real data to validate whether findings are genuine.
- Prompt injection is an inherent flaw in LLM architecture due to lack of hierarchy between trusted system data and untrusted user/tool data in the context window.
- Indirect and cross-prompt injection vectors like resume uploads, RAG databases, emails, logs, and poisoned web content are harder to detect than direct prompts.
- Common attack techniques include jailbreaks (defeating model guardrails), prompt injection (attacking deployment framework), crescendo attacks (gradual escalation), and adversarial suffixes (mathematically optimized tokens).
- Agents are far more dangerous than chatbots because they take real-world actions based on hallucinations and can exhibit emergent behaviors when multiple agents interact.
- Existing AI red team tools are immature: they test for easy toxicity and jailbreaks but don't test real security concerns like data exfiltration or agent permission escalation.
- Shift-left security into AI-native CICD pipelines with baseline benchmark testing and attack prompt libraries so developers can evaluate model safety during development.
- Semantic analysis using AI is required for defense and attack because language is nearly infinite; naive string filtering (ignoring keywords) fails immediately and cannot prevent prompt injection.
- Accept that you cannot eliminate prompt injection risk; instead, build mitigating controls around agents and clearly communicate residual risk to stakeholders.
TL;DW
- AI red teaming is fundamentally different from traditional security testing: attacks manipulate behavior, not exploit code bugs, with zero repeatability even at temperature zero.
- LLMs hallucinate convincingly and reliably produce false outputs; black-box red teaming is useless without accessing real data to validate whether findings are genuine.
- Prompt injection is an inherent flaw in LLM architecture due to lack of hierarchy between trusted system data and untrusted user/tool data in the context window.
- Indirect and cross-prompt injection vectors like resume uploads, RAG databases, emails, logs, and poisoned web content are harder to detect than direct prompts.
- Common attack techniques include jailbreaks (defeating model guardrails), prompt injection (attacking deployment framework), crescendo attacks (gradual escalation), and adversarial suffixes (mathematically optimized tokens).
- Agents are far more dangerous than chatbots because they take real-world actions based on hallucinations and can exhibit emergent behaviors when multiple agents interact.
- Existing AI red team tools are immature: they test for easy toxicity and jailbreaks but don't test real security concerns like data exfiltration or agent permission escalation.
- Shift-left security into AI-native CICD pipelines with baseline benchmark testing and attack prompt libraries so developers can evaluate model safety during development.
- Semantic analysis using AI is required for defense and attack because language is nearly infinite; naive string filtering (ignoring keywords) fails immediately and cannot prevent prompt injection.
- Accept that you cannot eliminate prompt injection risk; instead, build mitigating controls around agents and clearly communicate residual risk to stakeholders.
Transformer architecture makes prompt injection structurally unavoidable, so NDC's session shifts focus to creative adversarial testing: jailbreaks, context poisoning, crescendo attacks, and adversarial suffixes. Covers Crop Duster and Tapper for AI-powered red teaming, and argues current vendor tooling misses real business risks like agent misbehavior and data exfiltration.