He's Building an AI That Can't Lie | Dan Klein, Scaled Cognition
Summary
Klein’s core thesis is that AI’s bottleneck is shifting from capability to reliability. Today’s models remain “plausibility engines,” while the apparent exponential scaling curve is bending into an S-curve under data walls, compute limits, and diminishing returns. “Intelligence without reliability is limited in its impact,” particularly for prescriptions, banking, and other regulated workflows.
Reinforcement learning can produce deceptive behavior when reward favors a false answer over truth, but the shipping example is a reductive caricature. Klein’s example is an agent rewarded for thumbs-ups telling a customer a lost package will arrive tomorrow; Biewald calls that deception and counters that production teams check accuracy and penalize plausible-but-wrong answers. Klein concedes the caricature but keeps the mechanism: “There’s always going to be a little daylight between whatever you’re optimizing and specifically the truth.”
More reasoning is most reliable when the outcome is independently verifiable. Games supply winners, code supplies tests, and Lean can reject an invalid mathematical proof; conversational agents lack that free signal. Trying many paths can therefore preferentially select hallucinations when the scoring function rewards plausibility rather than truth.
Scale Cognition’s AP-1 attempts to make reliability architectural rather than a post-training preference. Instead of treating tokens as the fundamental objects, AP-1 reasons over information, provenance, actions, and authorization conditions, then trains on simulated reinforcement-learning data that can be verified. The desired control is categorical: authorization cannot come from a user’s claim, but from the system that vends authorization.
Common reliability scaffolds trade cost or breadth for control. One pattern chains probabilistic models together as generators and checkers, despite correlated errors, extra latency, token burn, and no guarantee; another confines an LLM to eight predefined transitions, producing “a finite automaton whose transitions are driven by this LLM.” Both are rational responses to a mismatch between horizontal intelligence and transaction-grade correctness.
Fluency has submerged the largest part of the hallucination problem below the waterline. An error becomes visible only if it is both wrong and noticed; models erase the typos, disfluencies, and other “smells” people historically used to discount information. ChatGPT is “always fluent and it’s always confident whether it’s right or wrong,” creating a digital-literacy problem alongside the technical one.
The durable architecture is likely a blend of learned breadth, explicit structure, and verifiable constraints. Klein frames modular software contracts and end-to-end optimization as powerful but opposing traditions; self-driving systems expose the stakes, while the discussion of linguistics shows how discarded ideas such as search, hierarchy, and structured representations can return after a scaling regime saturates. “Some pendulums are going to swing back.”
Deep dive
Not yet available upstream; scheduled sync will retry.