work-stealing vs shared-nothing architecture
Cloudflare: switching NGINX to Tokio's work-stealing scheduler cuts origin connections 66%, memory 75% ConfreaksTL;DW
- Cloudflare migrated from NGINX to Tokio-based Rust proxies (Pingora, FL2) across 93 million requests/second infrastructure to fix architectural limitations, not just for Rust performance gains.
- Shared-nothing architecture forces connection affinity to single worker threads, causing terrible connection pool reuse (multiple redundant pools per machine) and hundreds of gigabytes of redundant cached data per server.
- Work-stealing allows all worker threads to share one I/O driver and steal tasks from each other's queues, eliminating connection affinity issues and simplifying worker load balancing without kernel patches.
- Moving to work-stealing reduced origin connections by 66%, saving ~434 years of TLS handshake time per day and enabling shared connection pools instead of per-worker pools.
- FL2 (work-stealing) uses ~20GB memory vs Engine X-FL using ~85GB on same machine handling substantially more traffic—4:1 memory savings from eliminating redundant per-worker caches.
- NGINX's requirement to disable keep-alive between proxies (forcing new connection per request) caused significant latency that work-stealing solved by allowing any worker to handle any connection.
- Architectural changes (shared-nothing → work-stealing) delivered larger tail latency improvements than switching to Rust, proving runtime design matters more than language choice for web servers.
- Work-stealing is "incredibly fast out of the box" and easier to operate than shared-nothing, making it a better default for web servers despite shared-nothing advantages in specific cache-sensitive workloads.
- Unix domain socket communication between Cloudflare's proxies enabled the shift to work-stealing without changing transport layer fundamentals.
TL;DW
- Cloudflare migrated from NGINX to Tokio-based Rust proxies (Pingora, FL2) across 93 million requests/second infrastructure to fix architectural limitations, not just for Rust performance gains.
- Shared-nothing architecture forces connection affinity to single worker threads, causing terrible connection pool reuse (multiple redundant pools per machine) and hundreds of gigabytes of redundant cached data per server.
- Work-stealing allows all worker threads to share one I/O driver and steal tasks from each other's queues, eliminating connection affinity issues and simplifying worker load balancing without kernel patches.
- Moving to work-stealing reduced origin connections by 66%, saving ~434 years of TLS handshake time per day and enabling shared connection pools instead of per-worker pools.
- FL2 (work-stealing) uses ~20GB memory vs Engine X-FL using ~85GB on same machine handling substantially more traffic—4:1 memory savings from eliminating redundant per-worker caches.
- NGINX's requirement to disable keep-alive between proxies (forcing new connection per request) caused significant latency that work-stealing solved by allowing any worker to handle any connection.
- Architectural changes (shared-nothing → work-stealing) delivered larger tail latency improvements than switching to Rust, proving runtime design matters more than language choice for web servers.
- Work-stealing is "incredibly fast out of the box" and easier to operate than shared-nothing, making it a better default for web servers despite shared-nothing advantages in specific cache-sensitive workloads.
- Unix domain socket communication between Cloudflare's proxies enabled the shift to work-stealing without changing transport layer fundamentals.
Noah Kennedy details Cloudflare's edge proxy rewrite, showing shared-nothing event loops caused per-worker connection pool fragmentation and 85 GB config caches. Tokio's work-stealing runtime unified the IO driver, shrunk caches to 20 GB, cut origin connections by 66%, and improved tail latency—without kernel patches or disabled keep-alives.