Fault tolerance at 100K-GPU scale
Meta open-sources torchccomsand PAFT to sustain training across 100K-GPU clusters with failures every 18 minutes Open Compute ProjectTL;DW
- At 100,000-GPU scale, mean time to failure is ~18 minutes; 10-minute restarts leave only ~8 minutes of effective training time per cycle.
- Meta's parallelism-aware fault tolerance (PAFT) divides GPUs into independent replicas with dynamically scalable all-reduce rings; failures only impact one replica.
- GPU memory, PCIe, and watchdog timeouts are the largest failure sources in large-scale training; most are hardware-related and uncontrollable post-installation.
- Low-latency inference with mixture-of-experts requires device-centric communication; CPU bypass via GPU-direct async RDMA (IBGDA) achieves lowest latency.
- Meta's Pipes is a device-native communication framework allowing GPUs to execute custom collectives and transports without CPU involvement using Triton.
- TorchComms, Meta's production GPU communication stack, replaces classical PyTorch distributed APIs with new interfaces optimized for fast development and large-scale deployment.
- Meta open-sourced NCCL-X, RCCL-X, and C-Train; plans to open-source ML and Pipes this year as part of OSS-first development strategy.
- Pre-training dominates collective communication (all-gather, reduce-scatter); post-training needs rapid weight shipping; inference requires low-latency all-to-all for agentic workflows.
TL;DW
- At 100,000-GPU scale, mean time to failure is ~18 minutes; 10-minute restarts leave only ~8 minutes of effective training time per cycle.
- Meta's parallelism-aware fault tolerance (PAFT) divides GPUs into independent replicas with dynamically scalable all-reduce rings; failures only impact one replica.
- GPU memory, PCIe, and watchdog timeouts are the largest failure sources in large-scale training; most are hardware-related and uncontrollable post-installation.
- Low-latency inference with mixture-of-experts requires device-centric communication; CPU bypass via GPU-direct async RDMA (IBGDA) achieves lowest latency.
- Meta's Pipes is a device-native communication framework allowing GPUs to execute custom collectives and transports without CPU involvement using Triton.
- TorchComms, Meta's production GPU communication stack, replaces classical PyTorch distributed APIs with new interfaces optimized for fast development and large-scale deployment.
- Meta open-sourced NCCL-X, RCCL-X, and C-Train; plans to open-source ML and Pipes this year as part of OSS-first development strategy.
- Pre-training dominates collective communication (all-gather, reduce-scatter); post-training needs rapid weight shipping; inference requires low-latency all-to-all for agentic workflows.
At clusters approaching gigawatt scale, hardware failures hit every 18 minutes and cut effective training time below 50%. Meta's parallelism-aware fault tolerance uses redundant all-reduce rings that dynamically rescale around failures; inference gets a CPU-bypassing Pipes framework for MoE all-to-all traffic. torchccoms and NCCL-X are live; Pipes follows.