Dan Hendrycks

Humanity’s Last Exam is meant to mark “the end of a genre” for closed-ended AI evaluation, not certify AGI. MMLU is already well above 90%, while Humanity’s Last Exam was around 26%; once models solve its several thousand expert-written questions, individual successor problems should be “worthy of their own paper.” It still omits agency, long-term memory, physical experimentation, and economically useful execution.

Hendrycks argues that a US Manhattan Project for superintelligence would invite an arms race and sabotage rather than secure dominance. A trillion-dollar data center cannot plausibly remain secret, security-clearance requirements would shrink and redirect the talent pool, and China would interpret a monopoly bid as an existential threat: whether America controls the system or loses control of it, “either way, we want to prevent it.” The likely result is competing projects, insider threats, attacks on infrastructure, and pressure toward verification.

The strategic moat is compute and deployment capacity, not merely possession of the smartest model. Hendrycks puts the present critical mass for a state-of-the-art system around 10,000 cutting-edge GPUs, says a Chinese fleet below 100,000 could serve relatively few customers, and expects useful agents running continuously to require roughly two orders of magnitude more compute than chatbots used for minutes per day. “If it’s $10 billion, you can’t do it” captures his view that reproducing the leading-edge chip supply chain is harder than buying a nuclear capability.

His alternative strategy combines deterrence, chip non-proliferation, and ordinary economic competition with China. The relevant infrastructure includes energy for data centers, resilient semiconductor capacity if Taiwan is disrupted, secure robotics supply chains, and hyperscaler deployment capacity; the competitive objective is global market share rather than “let’s be the first to build superintelligence.” Export controls should keep the most dangerous capabilities away from actors such as North Korea or Iran while preserving access for states responsive to deterrence.

AI safety is a continuing risk-management function because capabilities and failure modes cross useful thresholds unpredictably. Hendrycks’s preferred technical breakthrough would be reliable honesty without higher inference cost or degraded performance, but he rejects the idea that alignment can be solved once: AI increasingly resembles a complex system with “constant new issues” and limited human adaptive capacity. In discussing utility engineering, the host summarized findings of preference coherence, self-preservation pressures, and political or demographic biases; Hendrycks treated these as warning signs rather than active catastrophes because today’s models cannot reliably exfiltrate, self-sustain, or hack autonomously.

Hendrycks is skeptical that scaling LLMs alone leads to AGI or recursive superintelligence, but sees a conditional discontinuity if human-level AI researchers exist. Current systems already weakly assist coding, chip design, cooling, labeling, and Constitutional AI; removing the human bottleneck could move research to machine speed and allow world-class researcher agents to be copied. Memory, planning, fluid intelligence, and multi-agent trust remain bottlenecks, and today’s algorithms plus more compute are not sufficient. Companies openly discussing such recursion without a credible control plan leave him saying, “I think something’s broken.”

Once labor is substitutable, political allocation of compute becomes the mechanism for preserving human bargaining power. “You had better bargain beforehand”: workers can no longer threaten to strike when automated firms and drone fleets possess the productive and coercive advantage. Hendrycks’s positive path distributes compute or its proceeds broadly, keeps humanity first, and preserves cognitive ability, autonomy, and multiple ways of living instead of leaving all gains to whoever owns the data centers in “the year 2027.”

🔗 Original source & video: Mutually Assured AI Malfunction [Dan Hendrycks]

The pivotal red line is a fully automated AI-research loop that takes humans out of development and moves progress from “human speed to machine speed.” Kokotajlo argues that recursive improvement could produce a durable strategic edge; he cites Dario Amodei’s discussion of an intelligence explosion and Sam Altman’s suggestion that a decade of development might telescope into a year or even a month. If a state controls the resulting superintelligence, rivals could be crushed; if nobody controls it, “everybody’s survival” may be threatened. Hendrycks separately says being first to trigger recursion could make even a short lead decisive.

The proposed containment package has three concrete parts: no explosive AI recursion, no lightly safeguarded expert virology or offensive-cyber agents, and strong security for capable model weights. Kokotajlo prefers a graduated regime in which states develop capabilities transparently, study each level, and debate whether to proceed. Hendrycks thinks coordination may begin with declared preferences and deterrence before reaching verification or treaties: “You have to have the conversation advance far further.”

The timeline spread is wide, but nobody in the room treats the risk as safely remote. Kokotajlo moved his superintelligence median from the end of 2027 to the end of 2028; colleagues had medians around 2029–2031. Hendrycks calls human-level cognitive breadth by 2030 more than plausible, while Marcus treats 2030 as the fastest plausible case and places most of his probability beyond ten years because reaching AGI soon would require solving “everything everywhere all at once.”

Current capex trends imply either radical automation this decade or a sharp slowdown in AI progress. Kokotajlo estimates training runs rose from roughly $3 million–$5 million in 2020 to around $1 billion, implying approximately $500 billion by 2030 if the same pace continues. Power, fabs, chip output, finite internet data, and corporate budgets then bind; without an AI-driven economic transformation, he expects at least a taper and potentially “a bit of an AI winter.”

Frontier labs’ central governance failure is a collective-action problem disguised as moral exceptionalism. Kokotajlo’s account is that DeepMind, OpenAI, and Anthropic leaders understood loss-of-control and concentrated-power risks, yet each embraced the same “seductive argument”: “If we don’t do it, someone else will.” Their belief that they are the responsible party converts acknowledged danger into a reason to race, making voluntary self-regulation an inadequate base case.

Technical alignment remains materially behind capability progress, especially where “fairly reasonably” is not enough. Hendrycks thinks narrow protections such as bioweapon refusals might achieve multiple nines of reliability, although labs may decline robust methods that cost “a percent or two in MMLU.” Broader requirements—avoiding criminal conduct, tortious or foreseeable harm—remain fuzzy, while intelligence recursion is a process-level problem whose unknown unknowns cannot be eliminated beforehand.

The optimistic payoff is enormous, but output abundance does not guarantee broad ownership or political autonomy. Kokotajlo describes superintelligences rapidly designing factories, laboratories, robots, medicines, and settlements until material needs are met; he also suggests distributing not just income but cryptographically controlled “compute slices.” Marcus has become darker because wealth holders may fund subsistence yet retain “the beachfront property” and power, leaving the positive equilibrium dependent on mechanisms nobody has supplied.

The architecture debate is also a moat debate: open weights democratize the starting line, not the compounding frontier. Kokotajlo argues that GPU-rich actors would use AGI to reach AGI+ and AGI++ first, creating strong returns to scale even if everyone received identical weights. Marcus instead expects a neurosymbolic state change: by 2035, today’s LLMs may look like a “nice try”—still useful, like flip phones after smartphones, but not the system that solved reasoning, world models, and robust generalization.

🔗 Original source & video: Explosive AI Timeline Predictions [Gary Marcus, Daniel Kokotajlo, Dan Hendrycks]

Hendrycks’s core call is that AI safety is primarily a statecraft problem, not solely a lab-level alignment problem. Labs are “predetermined to race”; they can cheaply gate expert bio access and address cyber risks, but obedient US and Chinese systems still leave strategic competition, rapid military integration, labor automation, and rising risk tolerance intact.

The immediate national-security picture is jagged: reasoning models are approaching expert virology capabilities, while Hendrycks does not think AI is currently relevant to a malicious actor’s devastating grid attack. That could “well change within a year’s time.” The discussion touches cyber defense, drones, electronic warfare, command-and-control reliability, and compute security as near-term application and risk areas.

Hendrycks rejects both a voluntary pause and a clean US sprint to superintelligence because neither survives adversarial response. Treaties need verification or force, model weights could be stolen, and a “trillion-dollar compute cluster in the desert totally visible from space” would look like a dominance bid that China would try to deter.

His proposed regime, mutually assured AI malfunction (MAIM), relies on shared vulnerability: states refrain from destabilizing superweapon projects because rivals can spy on or disable their data centers. Competition in chips and drones continues; coordination centers on preventing rogue-actor access, much as nuclear, chemical, and biological regimes separated rivalry from shared nonproliferation interests.

DeepSeek-style efficiency weakens capability denial against China, so Hendrycks would use export controls chiefly to track chips and constrain rogue actors while deterrence constrains great-power intent. His 80/20 package is concrete: a CIA AI-espionage cell, Cyber Command sabotage options, licensing and shipment notifications, and end-use checks—including investigating where the cited 10% of NVIDIA chips were going.

The economic phase change arrives when models move from impressive “oracle-like skills” to reliable agency. Humanity’s Last Exam is meant to expire closed-ended academic benchmarks near its ceiling—signaling superhuman performance in parts of math and STEM—but agents remain “near the floor”; once they can carry out digital work, Hendrycks expects “the vibes really shift.”

🔗 Original source & video: National Security Strategy and AI Evals on the Eve of Superintelligence with Dan Hendrycks