No Priors
Fractile bets fast inference is for long-running agents, not snappier chatbots
Fractile founder Walter Goodwin says the real prize is running multi-trillion-parameter models at thousands of tokens per second for long-running agents. That takes chips with huge memory bandwidth and also cheap memory capacity. Fractile dropped its early SRAM design, the Groq and Cerebras approach, because it couldn't scale with growing context lengths, and is now ramping a DRAM-based platform in the second half of next year with about 25 times the bandwidth per chip of HBM parts. His case: FLOPs have scaled a million-fold in 20 years while memory bandwidth grew about 40x, and closing that gap makes much sparser MoE models practical. "The Snappier chatbot is kind of the faster horses of kind of fast inference."
- #hardware
- #inference
- #agents
