AI scaling plateau, post-training frontier
Sara Hooker: scaling is hitting limits, adaptation and post-training are the next frontier Hugging FaceTL;DW
- Scaling model size shows decreasing returns; GPT-4.5, Llama 4, and Mixtral releases failed to justify their computational costs despite larger sizes.
- Small models now frequently outperform large ones on benchmarks; most neural network weights are redundant and can be removed after training with minimal performance loss.
- Post-training, test-time scaling, and adaptive compute now offer better returns than pre-training compute; frontier labs unlikely to 4x model size again this year.
- Adaptation and continuous learning emerge as the frontier; efficiency matters most because speed of learning from new information determines competitive advantage.
- Optimization in data space is now cheaper than ever; targeted data curation and generation can steer model behavior toward rare parts of distributions without massive pre-training costs.
- Auto Scientist automates end-to-end fine-tuning and outperforms human researchers at hyperparameter configuration by searching wider model families than domain experts typically optimize.
- Small labs with strong data and training strategies can now compete; test-time compute doesn't require collocated infrastructure like pre-training, enabling distributed innovation.
- Transformers are saturated architectures; hardware is overfit to matrix multiplication, making alternative architectures (capsule networks, sparse models) empirically difficult to succeed despite theoretical merit.
- Pre-training, post-training, and test-time scaling serve different functions; keep data fresh across stages by injecting new information rather than repeating, and reserve parametric capacity for skills while using retrieval for facts.
- Adaptive interfaces matter as much as models; code and design enable rich feedback loops absent in chat-interface thumbs-up systems—future interfaces should enable human-AI collaboration, not just mimic human behavior.
TL;DW
- Scaling model size shows decreasing returns; GPT-4.5, Llama 4, and Mixtral releases failed to justify their computational costs despite larger sizes.
- Small models now frequently outperform large ones on benchmarks; most neural network weights are redundant and can be removed after training with minimal performance loss.
- Post-training, test-time scaling, and adaptive compute now offer better returns than pre-training compute; frontier labs unlikely to 4x model size again this year.
- Adaptation and continuous learning emerge as the frontier; efficiency matters most because speed of learning from new information determines competitive advantage.
- Optimization in data space is now cheaper than ever; targeted data curation and generation can steer model behavior toward rare parts of distributions without massive pre-training costs.
- Auto Scientist automates end-to-end fine-tuning and outperforms human researchers at hyperparameter configuration by searching wider model families than domain experts typically optimize.
- Small labs with strong data and training strategies can now compete; test-time compute doesn't require collocated infrastructure like pre-training, enabling distributed innovation.
- Transformers are saturated architectures; hardware is overfit to matrix multiplication, making alternative architectures (capsule networks, sparse models) empirically difficult to succeed despite theoretical merit.
- Pre-training, post-training, and test-time scaling serve different functions; keep data fresh across stages by injecting new information rather than repeating, and reserve parametric capacity for skills while using retrieval for facts.
- Adaptive interfaces matter as much as models; code and design enable rich feedback loops absent in chat-interface thumbs-up systems—future interfaces should enable human-AI collaboration, not just mimic human behavior.
Hooker presents evidence that smaller models now outperform larger ones, model weights carry severe redundancy, and recent releases like GPT-4.5 and Llama 4 showed returns too poor to justify serving costs. The talk covers three vectors: post-training optimization, test-time compute on high-uncertainty examples, and continuous learning — illustrated by Auto Scientist, which outperformed human researchers on fine-tuning configuration search.