Co-design across entire computing stack—CPUs, GPUs, networking, storage—yielded 1 million-fold performance scaling over 10 years versus 100x from Moore's Law alone.
Computing fundamentally shifted from pre-recorded content and on-demand models to continuous, generative, agentic systems requiring completely rethought architectures.
Vera Rubin and Blackwell designed specifically for agent workloads: agents need low-latency CPUs for tools, high-bandwidth GPU for memory access, and storage directly connected to processors.
NVIDIA's approach to open-source language models (Neatron) addresses market gaps where languages lack sufficient scale for commercial development and enables fusion with domain-specific models.
Optimize for tokens-per-watt and real domain evals, not flops or MFU; low MFU with high performance indicates proper over-provisioning to avoid Amdahl's bottlenecks.
Education must integrate AI—both learning about it and using it to modernize curriculum—while preserving first-principles understanding; AI cannot keep pace with pre-recorded textbooks alone.
Early Nvidia failure (curved surfaces, wrong texture mapping) taught strategic lessons about seeing the world and competing; later mobile business pivot to zero showed importance of conserving resources on marginal opportunities.
Computing needs 1000× more energy; invest in sustainable energy immediately as market forces now make renewables profitable without subsidies, unlike past decades.
Stanford and universities must budget $1 billion+ for campus-wide supercomputers, not fragment compute across departments; aggregate scale is essential.
AI safety and security requires transparency: open systems enable scrutiny and defense via swarms of detection AIs, not black-box races between proprietary models.
Jensen Huang traces how jointly optimizing CPUs, GPUs, networking, storage, and software frameworks—rather than tuning each independently—delivered a million-fold performance gain over ten years versus roughly 100x from semiconductor advances alone. Covers Hopper, Grace Blackwell, and Vera Rubin architecture roles across pre-training, inference, and agentic workloads.
Energy is the binding bottleneck to AI scaling: compute, chips, and models all converge on electricity cost, which leaders from OpenAI, Nvidia, and SpaceX all identify as the critical constraint.
US electricity demand for AI could reach 1 terawatt within a decade, requiring energy supply growth steeper than any historical precedent—moving from 20-year stagnation to near-vertical expansion.
Nuclear power is the only viable baseload source meeting safety, emissions, and scalability requirements; all hyperscalers are pursuing nuclear despite 5–10 year build timelines.
Uranium enrichment is the missing infrastructure bottleneck: the US holds <0.1% global enrichment capacity and relies on Russia and Europe for nuclear fuel, blocking domestic reactor scaling.
General Matter secured a $900M DOE contract in January 2025—24 months after founding—to restore US uranium enrichment capacity, demonstrating how focused systems analysis unlocks government alignment and capital.
Enrichment is a fundamental primitive like SpaceX's launch cost: solving it enables downstream fuel production, reactor deployment, and clean energy scaling across the entire nuclear sector.
Bitcoin mining served as essential infrastructure rehearsal for AI datacenters—companies like Crusoe built stranded power utilization before pivoting to enterprise clouds, validating technology primitives regardless of initial use case.
Focus on first-principles problem-solving rather than surface-level narratives: nuclear's safety record (tied with wind, lowest emissions) contradicts public perception shaped by rare, famous accidents with minimal actual casualties.
The next 2–3 years will be hardest as turbine and grid interconnection equipment face 2+ year lead times; nuclear capacity comes online 2028–2030, creating a near-term scramble for stranded wind and natural gas.
Working on important unsolved problems with clear urgency, strong team fit, and government/market alignment creates extraordinary progress: General Matter will create hundreds of jobs in California and Kentucky while solving a civilizational bottleneck.
Scott Nolan argues energy, not chips, caps AI scaling—and nuclear is the only viable baseload option. The US produces under 0.1% of global enrichment capacity after its last facility closed in 2013, creating a dependency on Russia and Europe that General Matter's enrichment rebuild targets directly.
Anthropic's egress limiter counts all outbound bytes (including TCP ACKs, DNS, ICMP) and rate-limits them to make model weight theft take days/weeks instead of minutes, creating detection opportunities.
Model weights are terabytes of mostly incompressible data; legitimate cluster traffic (metrics, SSH, debugging) is only megabytes per second—this asymmetry is critical to the control's effectiveness.
Token bucket rate limiting allows burst traffic during off-peak hours but gradually throttles egress as the bandwidth bucket drains, providing better researcher experience than per-second limits.
Tiered enforcement: node-local traffic control handles accidental misconfigurations; perimeter routers (security control) bucket traffic by service type (logging, metrics, blob storage) to prevent one team's overages from impacting others.
IAM-enforced bucket boundaries prevent compute clusters from accessing external S3 buckets; proxies allow cross-boundary access only through the egress limiter for auditing and rate limiting.
Rollout killed ~98% of egress but took 4–6 months of reclassifying buckets and re-architecting systems; the final 2% (debugging, metrics, SSH) cannot be eliminated without major changes.
Research environments have huge attack surface (new dependencies, bleeding-edge stack, frequent vulnerabilities) adjacent to model weights worth hundreds of millions in compute—perimeter controls are necessary as defense-in-depth.
Accidentally misconfigured uploads now take days at 128 kbps instead of minutes, triggering alerts that inform researchers they're violating security policy rather than silently failing.
Egress limiting is a temporary fallback; long-term strategy is minimizing software touching unencrypted weights and securing that minimal set aggressively via TEEs and confidential compute—changes taking years.
Inference clusters also apply egress limiting but count bytes differently and accept additional risk on legitimate token traffic; future work explores subtracting model-generated tokens from egress limits.
By capping outbound bandwidth to ~100 Mbps at perimeter routers with per-service token buckets, Anthropic forces full-weight exfiltration—terabytes—to take weeks under assumed full-cluster compromise. The rollout cut egress 98% while keeping research workflows intact, buying detection time until TEEs and confidential compute mature.
Two-phase pre-training strategy: Phase 1 emphasizes data diversity (web crawl + reasoning data); Phase 2 focuses exclusively on high-quality sources (math, code, Wikipedia). Volta improves 17% over random ordering.
Frontloading reasoning data during pre-training yields durable advantages: models seeing reasoning data early gain 16% post-pretraining, 9.3% post-SFT, and 19% after full RLHF—gains compound rather than wash away.
High-quality reasoning data in pretraining unlocks hidden gains in posttraining: small high-quality (SHQ) + large diverse quality (LDQ) datasets show no benefit immediately but deliver 4.25% boost after SFT.
Early reasoning cannot be replicated by more SFT compute: models without reasoning-based pretraining trail reasoning-based models by 12% even with 2x SFT epochs and matched data budgets.
RLP (Reinforcement Learning on Pretraining) uses dense, verifier-free information-gain rewards during pretraining instead of sparse binary rewards, achieving 19% base model improvement and 8% improvement after identical posttraining.
RLP scales efficiently across model sizes and architectures: NeMoTron Nano 12B sees 35% gains using only 250M RLP tokens versus 20T token baseline; benefits persist after SFT with 3% absolute margin.
RLP outperforms RPT (Reinforcement Pretraining) by 4% because it applies dense per-token rewards on all positions without external verifier, capturing full reasoning signal versus ignoring reasoning steps.
Data quality estimation uses automated classifiers (Fine-Web EDU, Essential Web) scoring documents 1-5 on educational value, enabling systematic weighting of datasources in optimal blend.
Epoch estimation determines how many repeats of each datasource maximize downstream performance: some datasets hit diminishing returns at 2 repeats, others sustain 4-6 repeats before gains plateau.
RLP maintains 14% advantage over next-token prediction in flop-matched settings where baseline sees 35x more data, demonstrating data-efficient reasoning emergence without task-specific reasoning datasets.
Three strategies compound: a two-phase quality-aware curriculum, front-loading math and code before post-training (16-19% gains that survive SFT and RL), and RLP—which reframes pre-training as RL with dense information-gain rewards. RLP alone hits 35% improvement on a 12B model using 200B fewer tokens than baseline.
AI co-pilot study shows code movement down, copy-paste up from 8% to 12%, and added code up 7 points—researchers call this 'AI slop,' indicating less refactoring and more duplication.
DORA 2024 data: 60% of programmers feel more productive, but delivery performance went down; 75% want code generation, yet 77% don't trust it.
Claude models can maintain focus for approximately 15 minutes of task execution, and currently handle 200–300 tools; researchers estimate month-long task capability by ~2030, the potential tipping point for human replacement.
Non-professionals are quickly misled by AI agents; best quality results come from human-only work (slower), while AI+human pairing offers modest speed gains with acceptable quality trade-offs.
Assisted programming study found no significant time difference between AI-aided and non-aided developers; only highly proficient users saw up to 12.5% speedup, contradicting claims of universal productivity gains.
Extreme Programming values—communication, simplicity, feedback, courage, respect—must anchor AI integration; human factors and responsibility are absent from most vendor-driven AI narratives.
AI amplifies both good and bad code patterns; messy codebases degrade further with AI agents, while well-structured code improves, creating divergent outcomes based on initial quality.
Sub-agent architecture (specialized small agents for testing, refactoring, planning) beats monolithic AI agents; single responsibility principle applies to agentic workflows.
Code review remains cognitively exhausting even with AI assistance; the
, dream
Surveys research showing GitHub data reveals copy-paste code rose from 8% to 12% post-AI adoption, refactoring dropped, and churn increased. DORA data confirms 90% adoption but post-release instability offsets delivery gains. Argues for spec-driven development and pair-programming with AI as navigator to preserve architectural judgment.
