Stateful agent durability architecture
Trigger.dev splits agent durability into context logs + VM snapshots, drops replay AI EngineerTL;DW
- Agents fundamentally differ from transactions: they're sessions lasting as long as users want, not discrete workflows with clear endpoints.
- Snapshot-restore durability beats replay journaling for long-running agents—replay logs grow unbounded as agent interactions continue over hours or days.
- Agent durability requires two separate mechanisms: append-only context logs (LLM messages, tool calls, results) plus VM snapshots for execution state (files, memory, processes).
- Firecracker VM snapshots compressed to ~14MB enable sub-second snapshots and ~200ms restores, feasible at 15,000 VM starts per minute.
- Seekable compression decompresses only needed memory pages on restore, avoiding full snapshot reload costs and enabling practical cost economics.
- Shared-nothing architecture dominated backends for 30 years; agents force a shift to stateful compute infrastructure with persistent execution environments.
- Snapshot-restore handles diverse agent capabilities—running dev servers, cloned repos, subprocesses—that can't be durably reconstructed from logs alone.
- CRIU process checkpointing has limitations: only captures open files, incompatible with external processes like Chrome or FFmpeg, slower than VM snapshots.
- Agent durability enables asymmetric failure recovery: snapshot-and-wait during external delays, or replay context log when machine crashes.
- Trigger.dev's FC Run tool provides Docker-like CLI for Firecracker VM snapshotting and restoring, launching as open source for stateful compute workloads.
TL;DW
- Agents fundamentally differ from transactions: they're sessions lasting as long as users want, not discrete workflows with clear endpoints.
- Snapshot-restore durability beats replay journaling for long-running agents—replay logs grow unbounded as agent interactions continue over hours or days.
- Agent durability requires two separate mechanisms: append-only context logs (LLM messages, tool calls, results) plus VM snapshots for execution state (files, memory, processes).
- Firecracker VM snapshots compressed to ~14MB enable sub-second snapshots and ~200ms restores, feasible at 15,000 VM starts per minute.
- Seekable compression decompresses only needed memory pages on restore, avoiding full snapshot reload costs and enabling practical cost economics.
- Shared-nothing architecture dominated backends for 30 years; agents force a shift to stateful compute infrastructure with persistent execution environments.
- Snapshot-restore handles diverse agent capabilities—running dev servers, cloned repos, subprocesses—that can't be durably reconstructed from logs alone.
- CRIU process checkpointing has limitations: only captures open files, incompatible with external processes like Chrome or FFmpeg, slower than VM snapshots.
- Agent durability enables asymmetric failure recovery: snapshot-and-wait during external delays, or replay context log when machine crashes.
- Trigger.dev's FC Run tool provides Docker-like CLI for Firecracker VM snapshotting and restoring, launching as open source for stateful compute workloads.
Eric Allam argues replay-based durable execution breaks down for long-running agents that clone repos and hold in-memory state. Trigger.dev's Firecracker-based implementation uses an append-only context log for code compatibility and VM snapshots for execution state, hitting sub-second snapshots and 200ms restores at scale.