microservices cost architecture
Wix cuts Kubernetes node count 50% by collapsing 4,000 microservices into shared-runtime host-guest pairs GeeCONTL;DW
- Wix runs ~4,000 microservices but most receive low traffic; single-runtime architecture bundles them into virtual monoliths to reduce costs and improve efficiency.
- Achieved 50% reduction in Kubernetes nodes and 27% CPU, 32% memory reduction by moving from isolated services to host-guest architecture on shared pods.
- Guest isolation achieved via separate Docker containers; host-guest gRPC communication adds only ~2ms latency, negligible vs. cross-service calls requiring database access.
- Framework code (~90% of each service) moved to shared host; guests contain only business logic, reducing footprint from ~1.8GB to ~900MB memory per service.
- Kubernetes daemon sets deploy single host per node (like log scrapers); multiple guest replicas use standard deployments for independent scaling and gradual rollout.
- Custom deployment tool manages host upgrades across node groups (host-canary, host-1, host-2) separately from Kubernetes to ensure safe gradual rollout of critical infrastructure.
- Backward and forward compatibility required for host-guest protocol; deploy protobuf changes before logic changes to safely handle staggered host and guest deployments.
- Nile backend platform with code generation enables smooth migration; developers see no change—services automatically split into guest/host behind the scenes.
- Framework upgrades deploy once on host instead of 4,000 times; infrastructure team controls centrally, eliminating need to chase service owners for updates.
- Guest affinity planned: co-locate domain services (e-commerce, bookings, etc.) on same nodes to reduce network hops, improve latency via locality principle.
TL;DW
- Wix runs ~4,000 microservices but most receive low traffic; single-runtime architecture bundles them into virtual monoliths to reduce costs and improve efficiency.
- Achieved 50% reduction in Kubernetes nodes and 27% CPU, 32% memory reduction by moving from isolated services to host-guest architecture on shared pods.
- Guest isolation achieved via separate Docker containers; host-guest gRPC communication adds only ~2ms latency, negligible vs. cross-service calls requiring database access.
- Framework code (~90% of each service) moved to shared host; guests contain only business logic, reducing footprint from ~1.8GB to ~900MB memory per service.
- Kubernetes daemon sets deploy single host per node (like log scrapers); multiple guest replicas use standard deployments for independent scaling and gradual rollout.
- Custom deployment tool manages host upgrades across node groups (host-canary, host-1, host-2) separately from Kubernetes to ensure safe gradual rollout of critical infrastructure.
- Backward and forward compatibility required for host-guest protocol; deploy protobuf changes before logic changes to safely handle staggered host and guest deployments.
- Nile backend platform with code generation enables smooth migration; developers see no change—services automatically split into guest/host behind the scenes.
- Framework upgrades deploy once on host instead of 4,000 times; infrastructure team controls centrally, eliminating need to chase service owners for updates.
- Guest affinity planned: co-locate domain services (e-commerce, bookings, etc.) on same nodes to reduce network hops, improve latency via locality principle.
Wix's Nile platform bundles related JVM microservices as thin "guest" pods communicating via gRPC with a single "host" daemon set per node that owns all framework concerns—data access, Kafka, feature flags. Result: 27% CPU reduction, 32% memory reduction, half the node count across 5 billion daily requests.