Co-design as compute scaling paradigm
NVIDIA attributes 1,000,000x decade scaling to full-stack co-design, not Moore's Law Stanford OnlineTL;DW
- Co-design across entire computing stack—CPUs, GPUs, networking, storage—yielded 1 million-fold performance scaling over 10 years versus 100x from Moore's Law alone.
- Computing fundamentally shifted from pre-recorded content and on-demand models to continuous, generative, agentic systems requiring completely rethought architectures.
- Vera Rubin and Blackwell designed specifically for agent workloads: agents need low-latency CPUs for tools, high-bandwidth GPU for memory access, and storage directly connected to processors.
- NVIDIA's approach to open-source language models (Neatron) addresses market gaps where languages lack sufficient scale for commercial development and enables fusion with domain-specific models.
- Optimize for tokens-per-watt and real domain evals, not flops or MFU; low MFU with high performance indicates proper over-provisioning to avoid Amdahl's bottlenecks.
- Education must integrate AI—both learning about it and using it to modernize curriculum—while preserving first-principles understanding; AI cannot keep pace with pre-recorded textbooks alone.
- Early Nvidia failure (curved surfaces, wrong texture mapping) taught strategic lessons about seeing the world and competing; later mobile business pivot to zero showed importance of conserving resources on marginal opportunities.
- Computing needs 1000× more energy; invest in sustainable energy immediately as market forces now make renewables profitable without subsidies, unlike past decades.
- Stanford and universities must budget $1 billion+ for campus-wide supercomputers, not fragment compute across departments; aggregate scale is essential.
- AI safety and security requires transparency: open systems enable scrutiny and defense via swarms of detection AIs, not black-box races between proprietary models.
TL;DW
- Co-design across entire computing stack—CPUs, GPUs, networking, storage—yielded 1 million-fold performance scaling over 10 years versus 100x from Moore's Law alone.
- Computing fundamentally shifted from pre-recorded content and on-demand models to continuous, generative, agentic systems requiring completely rethought architectures.
- Vera Rubin and Blackwell designed specifically for agent workloads: agents need low-latency CPUs for tools, high-bandwidth GPU for memory access, and storage directly connected to processors.
- NVIDIA's approach to open-source language models (Neatron) addresses market gaps where languages lack sufficient scale for commercial development and enables fusion with domain-specific models.
- Optimize for tokens-per-watt and real domain evals, not flops or MFU; low MFU with high performance indicates proper over-provisioning to avoid Amdahl's bottlenecks.
- Education must integrate AI—both learning about it and using it to modernize curriculum—while preserving first-principles understanding; AI cannot keep pace with pre-recorded textbooks alone.
- Early Nvidia failure (curved surfaces, wrong texture mapping) taught strategic lessons about seeing the world and competing; later mobile business pivot to zero showed importance of conserving resources on marginal opportunities.
- Computing needs 1000× more energy; invest in sustainable energy immediately as market forces now make renewables profitable without subsidies, unlike past decades.
- Stanford and universities must budget $1 billion+ for campus-wide supercomputers, not fragment compute across departments; aggregate scale is essential.
- AI safety and security requires transparency: open systems enable scrutiny and defense via swarms of detection AIs, not black-box races between proprietary models.
Jensen Huang traces how jointly optimizing CPUs, GPUs, networking, storage, and software frameworks—rather than tuning each independently—delivered a million-fold performance gain over ten years versus roughly 100x from semiconductor advances alone. Covers Hopper, Grace Blackwell, and Vera Rubin architecture roles across pre-training, inference, and agentic workloads.