LLM non-determinism breaks security testing
LLM non-determinism breaks traditional risk assessments, forcing new threat models Wild West Hackin' FestTL;DW
- Generative AI models are inherently non-deterministic (probabilistic), making every output a roll of the dice—all traditional security mental models assume deterministic inputs guarantee outputs.
- Black box testing of LLM-enabled applications is unreliable; demand data flow diagrams and application-level logging of both prompts and responses, not just LLM-layer logging.
- LangSmith requires only two lines of code to capture every LLM call (prompt and response) without major refactoring, solving the logging skeleton-key problem for AI applications.
- Findings from LLM security testing are often non-repeatable due to non-determinism; pre-negotiate acceptance criteria with stakeholders since threat actors will have time to recreate unrepeatable findings in production.
- Prompt injection is a feature, not a bug of LLMs—you cannot solve it with syntax restrictions because that kills the value proposition; only mitigation is guardrails around all LLM output.
- LLM output must be treated as hostile data crossing a trust boundary, similar to second-order SQL injection; adversaries can describe malicious payloads in plain language to bypass input filters.
- Agents must run under non-human identities (service principals), never under user identities, to avoid non-repudiation problems and preserve audit trails.
- OAuth and OIDC lack attribution mechanisms for agentic workflows and cannot answer why a task was performed or under what context—they are insufficient for autonomous AI systems.
- Limit input prompt size on both client and server side; constraining user input to sentences or 300 characters can stop many 'do anything now' prompt injection attacks.
- Create test harnesses with pre-generated adversarial LLM outputs rather than relying on the LLM to regenerate the same output—this enables repeatable security testing and deeper application flow analysis.
TL;DW
- Generative AI models are inherently non-deterministic (probabilistic), making every output a roll of the dice—all traditional security mental models assume deterministic inputs guarantee outputs.
- Black box testing of LLM-enabled applications is unreliable; demand data flow diagrams and application-level logging of both prompts and responses, not just LLM-layer logging.
- LangSmith requires only two lines of code to capture every LLM call (prompt and response) without major refactoring, solving the logging skeleton-key problem for AI applications.
- Findings from LLM security testing are often non-repeatable due to non-determinism; pre-negotiate acceptance criteria with stakeholders since threat actors will have time to recreate unrepeatable findings in production.
- Prompt injection is a feature, not a bug of LLMs—you cannot solve it with syntax restrictions because that kills the value proposition; only mitigation is guardrails around all LLM output.
- LLM output must be treated as hostile data crossing a trust boundary, similar to second-order SQL injection; adversaries can describe malicious payloads in plain language to bypass input filters.
- Agents must run under non-human identities (service principals), never under user identities, to avoid non-repudiation problems and preserve audit trails.
- OAuth and OIDC lack attribution mechanisms for agentic workflows and cannot answer why a task was performed or under what context—they are insufficient for autonomous AI systems.
- Limit input prompt size on both client and server side; constraining user input to sentences or 300 characters can stop many 'do anything now' prompt injection attacks.
- Create test harnesses with pre-generated adversarial LLM outputs rather than relying on the LLM to regenerate the same output—this enables repeatable security testing and deeper application flow analysis.
Jake Williams (former NSA) walks through five production vulnerability classes — prompt injection, insecure output handling, credential leakage, weak agent identity governance, and logging gaps — and maps controls including LangSmith, Llama Guard, and prompt firewalls. Core guidance: treat LLM outputs as hostile by default and build test harnesses to reproduce probabilistic findings.