Training Data
Core Automation's founders: the transformer is the bottleneck, not the compute
Jerry Tworek scaled RL at OpenAI believing it would finish the job, and says 2025 was supposed to be the year everything got solved. Benchmarks kept climbing while real world tasks stayed messy, because the evals and the training data are two sides of the same coin and neither matches deployment. His conclusion is that models must learn at test time, which transformers structurally cannot do, so Core Automation is hunting for a replacement architecture and defining AGI as a model that improves itself with no human in the loop. As he puts it, the human LLM hybrid is really successful right now, but LLMs without humans, not so much. Rohan Anil adds the practical wall: a QR kernel competition they ran needed roughly three people on earth plus $100,000 of coding agents over four weeks to beat cuSolver by 60x, and no frontier model comes close to solving it.
- #research
- #agents
- #hardware
