adversarial agent harness design
Anthropic splits generator and evaluator agents into adversarial loop to sustain 6-hour builds AI EngineerTL;DW
- Generator-evaluator pattern (inspired by GANs) splits agent responsibilities into separate context windows: one builds, one critiques using Playwright/browser automation, creating adversarial pressure that improves quality.
- Claude Opus 4.6 can run continuously for 12+ hours with single-session compaction instead of resetting context, eliminating need for fresh sessions between tasks.
- Models are bad self-evaluators due to sycophancy; a separate harsh critic LLM is more tractable to tune than making builders self-critical.
- Planner-generator-evaluator loop with contractual handoffs works better than single agents: planner sets high-level direction, generator and evaluator negotiate testable acceptance criteria before building.
- Grading subjective quality (design taste, originality) is possible with detailed rubrics and few-shot examples; Anthropic weights design/originality heavily to prevent AI-slop aesthetics.
- Multi-hour full-stack apps (6 hours, ~$200) now achievable with generator-evaluator harness; same prompt in solo loop produces non-functional features (e.g., games with unresponsive controls).
- Context rot and context length anxiety (model rushing near window end) were critical problems in earlier models; Opus 4.6 handles coherence much better, reducing need for architectural workarounds.
- Read traces by hand, not just evals: identify where model judgment diverges from yours, then tune prompts. Automated trace analysis is a secondary pass; human reading is primary debugging loop.
- Harnesses co-evolve with models: as frontier improves, simplify harness (drop sprint decomposition, reduce evaluator cadence), then test to verify. Strip out scaffolding only after confirming models handle it.
- File system state (not context windows) for long-running agents: use JSON files for learnings, timestamped logs, git commits; lets future models or humans pick up work without re-inventing history.
TL;DW
- Generator-evaluator pattern (inspired by GANs) splits agent responsibilities into separate context windows: one builds, one critiques using Playwright/browser automation, creating adversarial pressure that improves quality.
- Claude Opus 4.6 can run continuously for 12+ hours with single-session compaction instead of resetting context, eliminating need for fresh sessions between tasks.
- Models are bad self-evaluators due to sycophancy; a separate harsh critic LLM is more tractable to tune than making builders self-critical.
- Planner-generator-evaluator loop with contractual handoffs works better than single agents: planner sets high-level direction, generator and evaluator negotiate testable acceptance criteria before building.
- Grading subjective quality (design taste, originality) is possible with detailed rubrics and few-shot examples; Anthropic weights design/originality heavily to prevent AI-slop aesthetics.
- Multi-hour full-stack apps (6 hours, ~$200) now achievable with generator-evaluator harness; same prompt in solo loop produces non-functional features (e.g., games with unresponsive controls).
- Context rot and context length anxiety (model rushing near window end) were critical problems in earlier models; Opus 4.6 handles coherence much better, reducing need for architectural workarounds.
- Read traces by hand, not just evals: identify where model judgment diverges from yours, then tune prompts. Automated trace analysis is a secondary pass; human reading is primary debugging loop.
- Harnesses co-evolve with models: as frontier improves, simplify harness (drop sprint decomposition, reduce evaluator cadence), then test to verify. Strip out scaffolding only after confirming models handle it.
- File system state (not context windows) for long-running agents: use JSON files for learnings, timestamped logs, git commits; lets future models or humans pick up work without re-inventing history.
Ash Prabaker and Andrew Wilson detail three failure modes for long-horizon agents—context limits, poor planning, and self-evaluation bias—and show how a GAN-inspired generator-evaluator pattern with Playwright-driven rubric testing enables 5-6+ hour runs. Concrete example: a retro game maker that solo single-session runs failed to complete.