Spec-driven agent evaluation
AI Engineer: test agents against behavioral specs, not benchmarks, to catch capability-boundary failures AI EngineerTL;DW
- Spec-driven testing for agents requires more than datasets: include rules, ontologies, domain knowledge, rights/roles, and robustness requirements under perturbations like typos and rephrasing.
- Larger models aren't always better or safer—they may understand jailbreaks in poetry, perform unnecessary tasks slowly, or create broader vulnerability surface area.
- Explicit specs enable security testing by defining agent domains (where it's most vulnerable) and the infrastructure tasks it can perform (where it has most power).
- Agent cards from A2A spec describe what agents do, but you still need envelopes specifying valid ranges—e.g., which people meetings can be booked for—to evaluate properly.
- Build integration tests independent of implementation platform (LangSmith, Vertex, etc.) so tests remain portable if you switch infrastructure later.
- Robustness testing should vary inputs systematically: can the agent handle typos, rephrasing, and other stress conditions before users get frustrated?
- Domain-specific substitutions matter in testing—LLMs may confuse gross profit vs. gross sales, but finance agents must distinguish them correctly.
- Use specs to close the loop: run agents automatically, iterate to fill robustness gaps, and version control everything in a GitHub repo for tool portability.
TL;DW
- Spec-driven testing for agents requires more than datasets: include rules, ontologies, domain knowledge, rights/roles, and robustness requirements under perturbations like typos and rephrasing.
- Larger models aren't always better or safer—they may understand jailbreaks in poetry, perform unnecessary tasks slowly, or create broader vulnerability surface area.
- Explicit specs enable security testing by defining agent domains (where it's most vulnerable) and the infrastructure tasks it can perform (where it has most power).
- Agent cards from A2A spec describe what agents do, but you still need envelopes specifying valid ranges—e.g., which people meetings can be booked for—to evaluate properly.
- Build integration tests independent of implementation platform (LangSmith, Vertex, etc.) so tests remain portable if you switch infrastructure later.
- Robustness testing should vary inputs systematically: can the agent handle typos, rephrasing, and other stress conditions before users get frustrated?
- Domain-specific substitutions matter in testing—LLMs may confuse gross profit vs. gross sales, but finance agents must distinguish them correctly.
- Use specs to close the loop: run agents automatically, iterate to fill robustness gaps, and version control everything in a GitHub repo for tool portability.
Willmott lays out a spec structure covering explicit rules, domain ontologies, role-based permissions, and robustness requirements, then shows how security and robustness testing both target the spec rather than the training set—so evaluations survive model swaps and infrastructure changes.