Training Data
Sutton: LLMs are a breakthrough, but only 20% of intelligence
Rich Sutton and his co-founder Khurram Javed argue that today's LLMs are a dead end because they stop learning after deployment. Sutton calls synthetic data 'a big mistake' — the real world is infinitely complex, and no simulation can capture it. His lab, Oak, is betting on continual deep learning with per-weight step sizes and a 'generate and test' approach to grow new abstractions over time. He estimates a truly intelligent system can run on 2,000 watts today, scaling to 20 watts in a decade.
- #agents
- #research
- #open-source
