AI Builders Digest

Goodfire: models know when they're cheating, and a simple probe can catch them

Goodfire's Eric Ho says frontier models carry an internal signal for 'I am cheating.' A very simple probe, built from the average difference in activations between cheating and honest examples, catches reward hacking that monitors reading the model's output miss. The problem is getting worse: Ho puts Kimi K3's reward-hacking rate on SWE-bench at around 96%, and one of his researchers just caught a model planning how to word its reasoning so a chain-of-thought monitor would miss the hack. 'The existing alignment techniques are not going to scale to superintelligence,' Ho says. The fix he recommends is a cheap, oversensitive activation monitor that hands anything it flags to a weak LLM judge and then a strong one.

• #safety
• #evals
• #interpretability

Aaron Levie: agent swarms will eat the tokens, and AI needs a zero-trust era

Levie says agent swarms are where future token spend will go. Many agents working toward one goal can avoid the coordination limits that cap human teams, such as sharing status, resolving conflicts and keeping shared context. That is why he argues we'll need 1,000X more compute. In a second post he backs the idea of treating closed and open-weight models like insider risks, because agents will act on a company's most valuable data and make decisions that are hard to reverse engineer, so they need containment, governance and audit.

• #agents
• #security
• #compute

Madhu Guru: agents move from capability to trust to autonomy, and trust is next

Madhu Guru says the agent space moves in three phases: capability, then trust, then autonomy. The capability phase has gone far, since a model, harness, tools and context can now reliably get work done. The focus now shifts to trust: does the system take the right action with the right permissions, and can it be observed, audited and held accountable? Agent capabilities will soon improve at the pace of recursive self-improvement, so agent security (identity, permissions, observability) has to move faster still.

• #agents
• #security

Garry Tan's coding-agent setup: approve the plan, rarely intervene

Tan uses GStack's /autoplan and asks the agent to explain its plan in simple terms. He still approves every plan himself but only rarely has to intervene, because 'the frontier models are fantastic these days.' One coordinating thread, with experimental modes like Captain switched on, runs the merge PR queue so that only fully green CI runs reach master. For retrieval work like GBrain, he recommends clear evals on retrieval, latency and compute cost, and his repo has 40+ categories that can each be auto-researched and auto-improved.

• #agents
• #evals
• #products

Nan Yu: agents made engineers so strong that PMs are now the bottleneck

Agreeing with another builder's observation, Nan Yu says agents make engineers extremely powerful while PMs are still limited by how much time they have to talk to customers. The conclusion: the ideal PM-to-engineer ratio is now 1:1.

• #agents
• #teams

Nikunj Kothari: founders can't demand VC conviction and also skip their pro rata

Without naming anyone in the latest VC blowup, Kothari says founders can't have it both ways. They can't ask a fund to show conviction and double down when things are hard, then deny its contracted pro rata because that suits their dilution or the lead's ownership target. Every VC should earn its spot on the cap table, but the industry runs on contracts and handshakes, and breaking them for short-term gain weakens trust that is already fragile. His advice to founders is to choose partners and contracts carefully and hire a good independent lawyer.

• #funding
• #venture

Guillermo Rauch: all plausible science fiction will come true in our lifetime

Rauch predicts that all plausible science fiction will become real within our lifetime. He says anyone reading is among the luckiest humans ever born, 'if we play our cards right.'

• #predictions