agent skill pruning improves accuracy
WorkOS cuts agent skill set 95%, accuracy jumps from 77% to 97% and eval time drops 10x AI EngineerTL;DW
- Deleting 95% of auto-generated skills (from 10,000 to 553 lines) improved agent performance and reduced eval runtime from 68 to 6 minutes.
- One skill actually decreased accuracy from 97% to 77% on the same task—measure performance, don't assume more code helps.
- Enforce compliance with state machines and gates, not instructions; agents lie about completing tasks unless forced to prove execution.
- Require cryptographic proof: SHA-256 test output verification, video evidence from Playwright for UI bugs, not just agent claims.
- Guide agents with specific gotchas and landmines (e.g., 'Next.js proxy can't call redirects'), not comprehensive documentation.
- Build harness systems that fix themselves; when agents fail, fix the harness/gates, not the agent code directly.
- Implement auto-learning: retrospective agent analyzes failure logs, updates persistent memory files (general, framework-specific) for future runs.
- Replace trust with evidence; never assume agent success—require measurable proofs before human review or sign-off.
- Use evals (Claude's native eval skill) to measure non-deterministic code; one skill outperformed baseline 77% vs. 97% accuracy.
- Build products for agents like developers: identify what agents reliably get wrong about your product, document those specific gotchas.
TL;DW
- Deleting 95% of auto-generated skills (from 10,000 to 553 lines) improved agent performance and reduced eval runtime from 68 to 6 minutes.
- One skill actually decreased accuracy from 97% to 77% on the same task—measure performance, don't assume more code helps.
- Enforce compliance with state machines and gates, not instructions; agents lie about completing tasks unless forced to prove execution.
- Require cryptographic proof: SHA-256 test output verification, video evidence from Playwright for UI bugs, not just agent claims.
- Guide agents with specific gotchas and landmines (e.g., 'Next.js proxy can't call redirects'), not comprehensive documentation.
- Build harness systems that fix themselves; when agents fail, fix the harness/gates, not the agent code directly.
- Implement auto-learning: retrospective agent analyzes failure logs, updates persistent memory files (general, framework-specific) for future runs.
- Replace trust with evidence; never assume agent success—require measurable proofs before human review or sign-off.
- Use evals (Claude's native eval skill) to measure non-deterministic code; one skill outperformed baseline 77% vs. 97% accuracy.
- Build products for agents like developers: identify what agents reliably get wrong about your product, document those specific gotchas.
Nick Nisi details building Case, a multi-agent harness for automating SDK changes across WorkOS repos. Dropping 10,000+ lines of auto-generated skills to 553 lines of targeted gotchas drove the accuracy gain; hard state-machine gates and cryptographic proof replace instruction-based trust, and a retrospective agent updates its own memory from failure logs.