AI agent attack surface
Thoughtworks demos DoS attack on production AI agent via prompt injection and over-permissioning ThoughtworksTL;DW
- AI agents without guardrails enable denial-of-service attacks: a vet triage agent booked 50 appointments in one day when prompted, crippling availability for a small business.
- The 'lethal trifecta' is untrusted input coupled with risky actions—the definition of an unsafe agent. Separate these concerns across multiple agents to control execution.
- STRIDE threat modeling (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) applies to AI agents just as it did to web applications.
- Agents are often given excessive permissions to databases, health records, and payment systems without considering real-world harms from misuse or compromise.
- Input validation and authentication remain powerful controls; treat malicious input to agents like SQL injection and cross-site scripting—old security problems in new architectures.
- Three priorities for agent deployment: understand the use case, threat model the data flow and possible actions, then apply appropriate guardrails and test thoroughly.
- LLM-as-judge—calling another LLM to detect malicious actions—is worth investigating as a control layer for risky agent operations.
- Availability and resilience matter for AI agents; denial-of-service impacts are regulated in Europe (eIDAS, NIS2) and can be crippling for small businesses.
- Agents require authentication on inputs and logging of actions to investigate incidents; without logging, you cannot improve or trace responsibility.
- Data poisoning via context windows poses risk; focus security on what enters the prompt, not primarily on the model itself.
TL;DW
- AI agents without guardrails enable denial-of-service attacks: a vet triage agent booked 50 appointments in one day when prompted, crippling availability for a small business.
- The 'lethal trifecta' is untrusted input coupled with risky actions—the definition of an unsafe agent. Separate these concerns across multiple agents to control execution.
- STRIDE threat modeling (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) applies to AI agents just as it did to web applications.
- Agents are often given excessive permissions to databases, health records, and payment systems without considering real-world harms from misuse or compromise.
- Input validation and authentication remain powerful controls; treat malicious input to agents like SQL injection and cross-site scripting—old security problems in new architectures.
- Three priorities for agent deployment: understand the use case, threat model the data flow and possible actions, then apply appropriate guardrails and test thoroughly.
- LLM-as-judge—calling another LLM to detect malicious actions—is worth investigating as a control layer for risky agent operations.
- Availability and resilience matter for AI agents; denial-of-service impacts are regulated in Europe (eIDAS, NIS2) and can be crippling for small businesses.
- Agents require authentication on inputs and logging of actions to investigate incidents; without logging, you cannot improve or trace responsibility.
- Data poisoning via context windows poses risk; focus security on what enters the prompt, not primarily on the model itself.
Using a veterinary triage agent as a live case study, Jim Gumbley shows how broad database access plus a "obey user instructions" directive lets an attacker book 50 simultaneous appointments. Applies STRIDE threat modeling and argues the fix is input validation, least-privilege, and separating untrusted input from risky actions.