Summary

The pivotal red line is a fully automated AI-research loop that takes humans out of development and moves progress from “human speed to machine speed.” Kokotajlo argues that recursive improvement could produce a durable strategic edge; he cites Dario Amodei’s discussion of an intelligence explosion and Sam Altman’s suggestion that a decade of development might telescope into a year or even a month. If a state controls the resulting superintelligence, rivals could be crushed; if nobody controls it, “everybody’s survival” may be threatened. Hendrycks separately says being first to trigger recursion could make even a short lead decisive.
The proposed containment package has three concrete parts: no explosive AI recursion, no lightly safeguarded expert virology or offensive-cyber agents, and strong security for capable model weights. Kokotajlo prefers a graduated regime in which states develop capabilities transparently, study each level, and debate whether to proceed. Hendrycks thinks coordination may begin with declared preferences and deterrence before reaching verification or treaties: “You have to have the conversation advance far further.”
The timeline spread is wide, but nobody in the room treats the risk as safely remote. Kokotajlo moved his superintelligence median from the end of 2027 to the end of 2028; colleagues had medians around 2029–2031. Hendrycks calls human-level cognitive breadth by 2030 more than plausible, while Marcus treats 2030 as the fastest plausible case and places most of his probability beyond ten years because reaching AGI soon would require solving “everything everywhere all at once.”
Current capex trends imply either radical automation this decade or a sharp slowdown in AI progress. Kokotajlo estimates training runs rose from roughly $3 million–$5 million in 2020 to around $1 billion, implying approximately $500 billion by 2030 if the same pace continues. Power, fabs, chip output, finite internet data, and corporate budgets then bind; without an AI-driven economic transformation, he expects at least a taper and potentially “a bit of an AI winter.”
Frontier labs’ central governance failure is a collective-action problem disguised as moral exceptionalism. Kokotajlo’s account is that DeepMind, OpenAI, and Anthropic leaders understood loss-of-control and concentrated-power risks, yet each embraced the same “seductive argument”: “If we don’t do it, someone else will.” Their belief that they are the responsible party converts acknowledged danger into a reason to race, making voluntary self-regulation an inadequate base case.
Technical alignment remains materially behind capability progress, especially where “fairly reasonably” is not enough. Hendrycks thinks narrow protections such as bioweapon refusals might achieve multiple nines of reliability, although labs may decline robust methods that cost “a percent or two in MMLU.” Broader requirements—avoiding criminal conduct, tortious or foreseeable harm—remain fuzzy, while intelligence recursion is a process-level problem whose unknown unknowns cannot be eliminated beforehand.
The optimistic payoff is enormous, but output abundance does not guarantee broad ownership or political autonomy. Kokotajlo describes superintelligences rapidly designing factories, laboratories, robots, medicines, and settlements until material needs are met; he also suggests distributing not just income but cryptographically controlled “compute slices.” Marcus has become darker because wealth holders may fund subsistence yet retain “the beachfront property” and power, leaving the positive equilibrium dependent on mechanisms nobody has supplied.
The architecture debate is also a moat debate: open weights democratize the starting line, not the compounding frontier. Kokotajlo argues that GPU-rich actors would use AGI to reach AGI+ and AGI++ first, creating strong returns to scale even if everyone received identical weights. Marcus instead expects a neurosymbolic state change: by 2035, today’s LLMs may look like a “nice try”—still useful, like flip phones after smartphones, but not the system that solved reasoning, world models, and robust generalization.

Deep dive

1. Abundance is possible only inside a safe equilibrium

Kokotajlo accepts that standing before the bulldozer can be reasonable: if development is presently headed toward something horrible, society should stop until it finds a better route. His objection is not to AI forever; correctly built systems could be “massively beneficial to everyone.”
AI 2027’s positive “slowdown ending” has the leading company and government devote barely enough time and resources to alignment, solve it just in time, and still beat China. Kokotajlo stresses that this is not their recommendation—it exposes humanity to extraordinary risk—but it is a coherent way the world might “muddle through.”
Once systems are better than the best humans at everything, faster, and cheaper, they could design robot factories, laboratories, industrial equipment, and successive technologies in record time. Within “only a few years,” Kokotajlo imagines automated abundance, cures for diseases, and even settlements on Mars.
Kokotajlo rejects the claim that AI necessarily reduces humanity to zoo animals or permanent VR pleasure. A workable society could preserve autonomy while meeting multiple objective goods; one deeper-future mechanism would give individuals cryptographic keys to rentable compute slices, distributing productive power rather than only cash.

2. Political distribution may break the optimistic case

Marcus is darker than he was several years ago because corporate acquisitiveness has made universal basic income look less credible. Even if abundance covers subsistence, he does not expect wealthy owners to surrender “the beachfront property” or political power; recent Washington dynamics have further reduced his confidence in anything short of extreme inequality.
That leaves two alignment problems: whether the machines remain controllable and whether whoever controls them distributes the gains acceptably. Marcus can imagine stopping the train if no political solution is even envisionable, because technical success alone does not create a positive outcome.
His preferred framing remains pause rather than permanent prohibition. The pause letter sought to delay GPT-5 while researchers addressed known GPT-4 alignment problems—ironically, he notes, GPT-5 still did not exist two and a half years later. Hendrycks agrees with the pause-versus-stop distinction; Kokotajlo does not expect much return on investment from technical research aimed at making AI fully controllable.

3. Automated AI research is the destabilizing capability

Kokotajlo concentrates on stopping superintelligence because that demand appears more compatible with geopolitical incentives than stopping all AI tomorrow. The practical trigger is fully automating AI research and development: “take the human out of the loop,” and development moves from human speed to machine speed.
Kokotajlo cites Dario Amodei’s discussion of a recursive process producing an intelligence explosion and durable lead; he also cites Altman’s suggestion that a decade of AI development could compress into a year or potentially a month. That speed, not vague fear of “AI,” creates the strategic instability.
If China or another state controlled such a system, Kokotajlo says it might weaponize the advantage and crush other countries. If nobody controlled a nearly unsupervised, extremely fast process—which he considers fairly likely—everyone’s survival could instead be threatened.
Today’s faster hyperparameter searches are not the recursion under discussion. The threshold is a system originating ideas that in 2024 would have required humans, then testing, validating, tweaking, implementing, and scaling them; all three agree that labs are explicitly trying to reach that capability.

4. Three red lines define a possible containment regime

Hendrycks names three lines: no fully automated recursion with explosive potential; no generally accessible agents with expert-level virology or offensive-cyber skill absent safeguards; and strong information security for model weights above a meaningful capability threshold, preventing theft or exfiltration by rogue actors.
Kokotajlo calls recursion an excellent place to start, though his preferred regime is gradual rather than a single permanent boundary. States would develop capabilities under mutual transparency, study each new level, and openly debate whether proceeding further is safe.
Marcus notes that treaties can require eight or ten years, making early work rational even if the dangerous capability is 25 years away. Hendrycks expects coordination to emerge in stages—declared preferences, internal policies, leaks, deterrence, perhaps even a skirmish that creates demand for verification—before a formal treaty becomes feasible.

5. A six-month lead matters only when recursion begins

Marcus doubts that the current LLM paradigm can sustain a durable national advantage; he says that if the field remains on LLMs, “nothing’s going to be durable.” He could imagine a genuinely different approach, such as neurosymbolic AI, creating a temporary lead, but does not expect the current paradigm to do so.
Kokotajlo defines durability differently: competitors may learn everything six months later, yet never close the gap because the leader keeps advancing at the same rate. Six months looks modest today, but it becomes decisive if the leader uses that interval to trigger recursive improvement.
He also rejects a clean division between “LLMs” and a future paradigm. Reasoning agents already combine language models, tools, and code; AI in 2027 may differ substantially from AI in 2023 through a continuous paradigm shift rather than one discrete invention.
Hendrycks sees public capability transparency as a brake: if people could watch a lab initiate recursion, “the world would be very much freaking out.” Marcus cites polls showing roughly 75% of Americans already worried; Kokotajlo counters that his OpenAI experience involved pressure to downplay risk, not advertise it.

6. Frontier labs convert fear into a reason to race

Kokotajlo separates two risks long discussed inside leading labs: loss of control if alignment fails, and concentration of power if alignment succeeds but a company or ruler commands the systems. Old writings, leadership statements, and leaked communications show that neither issue arrived late.
His explanation for continued acceleration is “rationalization.” The argument—“it’s probably going to happen anyway; if we don’t do it, someone else will”—lets each group conclude that winning the race is the safest course because it trusts its own competence and benevolence more than everyone else’s.
DeepMind’s plan was to establish enough lead for Demis Hassabis to slow down and solve safety. OpenAI arose partly because Elon Musk, Altman, and Ilya did not trust Demis with that power; emails discussed fears that he might become a dictator. Anthropic then split from OpenAI because its founders distrusted OpenAI’s safety conduct.
Kokotajlo’s conclusion is categorical: “We definitely should not” simply trust any of them. Governments need staff tracking frontier capability, interviewing labs, assessing their plans, and preparing contingencies; the relevant test is whether companies help solve collective-action problems or continue the pattern of defection.

7. Transparency is tractable, but it will not arrive voluntarily

Hendrycks is optimistic about public disclosure of peak internal capabilities, though not necessarily methods or weights. His incentive argument is that China may already know much of what happens at US labs, while the wider world knows less about People’s Liberation Army developments; disclosure could improve credible deterrence by narrowing that imbalance.
Marcus distinguishes “tractable” from “automatic.” Labs have less reason to disclose once one possesses differentiated intellectual property that competitors cannot quickly reconstruct, so both he and Kokotajlo reject voluntary self-regulation as a sufficient mechanism.
Open weights do not eliminate concentration in Kokotajlo’s model. Even if everyone received frontier AGI simultaneously, actors with the most GPUs could run more research and reach AGI+ and AGI++ first; recursive development therefore contains an inherent return-to-scale dynamic.

8. Forecasts share long tails but diverge sharply before 2030

Kokotajlo prefers probability distributions to single dates and divides the target into milestones: superhuman coding, full automation of AI research, and superintelligence. The curves encode subjective judgment, with a hump in the next five years and a long tail rather than certainty around one calendar year.
While writing AI 2027, his superintelligence median was the end of 2027; he has since moved it to the end of 2028. Other AI Futures Project forecasters expected later dates such as 2029 or 2031, but “I was the boss, so we went with my timelines.”
Marcus treats 2030 as the fastest plausible case and places most of his probability beyond ten years, although he accepts that breakthroughs could eventually accelerate progress. Hendrycks remains deliberately less committal while thinking through multidimensional cognition, but calls a system with the cognitive abilities of a typical human by 2030 “more than plausible.”
Their distance is smaller in the tail: Marcus and Kokotajlo both retain meaningful probability around or after 2045. The consequential disagreement is whether present limitations become binding bottlenecks or merely “a series of road bumps” that frontier companies bash through over the next several years.

9. Scaling either transforms the economy or runs into hard budgets

Kokotajlo attributes much of the last 15 years’ progress to scaling compute, data, and researcher participation, with compute probably the largest input. His near-term hump reflects continued scaling; the long tail appears because maintaining the same exponential pace becomes much harder after the next few years.
Power supply is only one constraint. Once companies cannot simply repurpose gaming chips, another 10× increase demands vastly more AI-chip production and fabs; many separate frictions compound even if none creates a sharp cutoff.
Marcus estimates that data is already binding: GPT-2 used perhaps 5–10% of the internet, later systems used much more, and GPT-4 used most of it, including video transcripts. No one can keep increasing that corpus 100×. Synthetic data works best where answers can be verified, such as mathematics; Marcus says “thinking mode” has partly taken up the slack.
Money captures the combined constraint. Kokotajlo contrasts roughly $3 million–$5 million training runs in 2020 with approximately billion-dollar runs now; another two-and-a-half orders of magnitude implies a $500 billion run in 2030. Without radical AI-led automation by decade-end, he expects tapering progress and possibly “a bit of an AI winter.”

10. Compute forecasts create a prior, not an answer

Kokotajlo’s forecasting recipe is candid: build models, inspect trends, perform calculations, “and then pull a number out of your ass based on all that stuff.” The numbers remain subjective, but explicit structure makes the judgment more disciplined than an unsupported date.
The Bio Anchors framework maps time for new research ideas against compute available for brute force. More compute raises what can be attempted today; new algorithms progressively shift the required-compute distribution downward, turning a two-dimensional uncertainty into a distribution over years.
At roughly (10^{45}) floating-point operations, he offers a soft upper bound: simulate Earth, life, and evolution for a billion years without understanding intelligence at all. Current training runs are around (10^{26}); spreading uncertainty between current capability and that upper region still assigns non-negligible mass to this decade, after which concrete benchmark evidence should update the prior.

11. Coding is the first rung of recursive research

Kokotajlo does not expect humans to invent superintelligence in one leap. His path automates progressively more of AI research, reaches new paradigms faster, and starts with coding because it is the lowest-hanging research activity.
Agentic coding suites from groups such as METR contain tasks requiring humans roughly four or eight hours. He expects these benchmarks to saturate within several years, then explicitly adds a speculative “gap” between benchmark saturation and complete coding automation.
His superhuman-coder milestone is not autocomplete: a user gives high-level instructions and the agent performs like an excellent professional software engineer. Marcus thinks apprentice-level engineering is close but predicts no Jeff Dean-equivalent during the next decade—someone who understands unprecedented problems, rapidly prototypes, and turns solutions into production systems.
Marcus’s objection is originality, not whether agents can automate more experiments. Current systems can work “inside the box,” but reaching AGI may require Einstein-level reframing; Kokotajlo instead expects automation to help discover the new paradigms and treats today’s weaknesses as temporary obstacles.

12. Marcus sees architectural gaps that scaling has not solved

Marcus’s checklist comes from cognitive science: out-of-distribution generalization, operations over variables, structured representations, separating types from tokens, planning, reasoning, stable world models, and flexible domain knowledge. He says the same core gaps he identified in 1998 and 2001 still generate today’s failures.
The specimens are deliberately basic. A 1957 classical technique solves Tower of Hanoi for arbitrary sizes, while modern models break under length changes; o3 can make illegal chess moves; and domain-engineered AlphaFold, a hybrid neurosymbolic system, substantially outperforms a general chatbot within its specialty.
Kokotajlo grants that a (10^{45})-FLOP evolutionary system might be opaque but says evolved creatures that built a civilization would still be intelligent and economically usable. Marcus agrees opacity is plausible—that is precisely his concern: “There’s too much alchemy and not enough principles,” making deliberate alignment harder.
Marcus compares the current paradigm to the early belief that genes were proteins. Once experiments established DNA, molecular biology advanced rapidly; similarly, AI may need a state change away from a bad assumption. Getting every missing faculty by 2027 would require “everything everywhere all at once.”

13. Tool use turns the dispute into one about reliability

Kokotajlo argues that the relevant system is a transformer plus tools, plugins, and code interpreters. Claude need not mentally execute Tower of Hanoi if it can look up the algorithm, implement it, and return the correct result—just as humans often use external tools for arithmetic.
Marcus calls that explicitly neurosymbolic: the network generates Python, whose symbols and variables perform operations the neural component cannot reliably execute. If the model always selected the right tool, “then you’re golden”; earlier LLM interfaces to Wolfram Alpha or Mathematica were hit-or-miss precisely at that interface.
Kokotajlo reads METR’s growing task-horizon graph as evidence of improving reliability. If an agent has a fixed chance of catastrophic error per step, lowering that rate lets it complete longer jobs; newer systems appear better both at avoiding mistakes and recovering from them.
Marcus’s pushback is methodological: the graph covers coding, may contain unknown contamination, and defines success at 50%. A parallel agent graph reportedly showed similar visual improvement but on a seconds-scale axis; meanwhile, choosing a legal chess move takes a human moments, yet o3 still cannot do it with 100% reliability.

14. Intelligence is multidimensional, and benchmarks illuminate selectively

Hendrycks warns of a “streetlight effect”: researchers build benchmarks where progress is measurable and AI already has traction. A model can top one generation of video tests, only for the next generation to reveal structural defects that saturation concealed.
Randomly sample cognitive tasks used with children and models still fail on a double-digit percentage, perhaps nearly half: counting faces in a photograph, connecting dots, or filling colors. Fluid intelligence, visual processing, long-term memory, spatial reasoning, and physical understanding are not interchangeable with writing fluency.
Pretraining delivered reading, writing, and crystallized knowledge over several years, not instantly. Google’s Minerva reached about 50% on the MATH benchmark in 2022, with models crushing it around 2025; reading and writing took roughly four years to mature. Hendrycks might expect audio to improve within one or two years, but not full video understanding.
A severe deficit on one axis can block economic automation. Weak long-term memory prevents a worker-agent from maintaining state and absorbing organizational context; weak fluid intelligence limits generalization. Marcus adds that present systems also struggle to label diagrams or reason through physical environments.

15. Fresh problems expose the difference between fluency and generalization

Marcus moves familiar games slightly off distribution. Grok failed a tic-tac-toe variant where only edge three-in-a-row patterns counted, even after correction; chess systems struggle when asked to construct unusual positions, alter board geometry, or apply familiar rules flexibly rather than replay orthodox games.
The training defense does not satisfy him because frontier models have likely seen chess rules, transcripts, books, and Wikipedia, yet still violate legal moves. Scientific evaluation remains compromised because labs do not disclose their data or augmentation; his foundational question is how systems generalize beyond training, and “we just don’t have transparency on that.”
He cites LiveCodeBench Pro, whose brand-new problems reportedly produced 0% machine performance, and a USAMO study where performance on fresh problems was poor. Frontier Math is less clean evidence because OpenAI had access and outsiders cannot determine what relevant augmentation occurred, even if the company says it did not train on the answers.
Kokotajlo expects such counterexamples to become harder to find each year, not disappear overnight. Marcus compares the pattern to autonomous driving: Waymo improves continuously yet still uses geography-specific maps and operating domains. Hendrycks adds that even writing remains around 4.5/6 on a GRE-style scale despite effectively all available text.

16. Alignment is a moving reliability problem, not a solved module

Marcus sees enormous capability gains in interpolation but little comparable alignment progress. Reinforcement learning can make a model decline an obvious bioweapon request, yet jailbreaks persist; systems still hallucinate, reproduce prohibited copyrighted text, and disobey simple constraints. An illegal chess move is his miniature version of the same problem.
Hendrycks distinguishes aligning proto-superintelligent models from aligning the recursive process that creates superintelligence. The first is model-level; the second involves an unprecedented, extremely fast process whose unknown unknowns cannot be anticipated in an eight-page paper. Technical work cannot “fully derisk” that recursion beforehand.
Current systems follow instructions “fairly reasonably,” which Marcus says “scares the shit out of me” when weapons are involved. Hendrycks thinks adversarially robust bioweapon refusal might reach multiple nines, but production labs may avoid methods costing one or two MMLU points; general bans on criminal, tortious, or foreseeably harmful conduct are much less tractable.
Hendrycks expects new failure modes, targeted fixes, and a growing backlog as agents expand the attack surface. The requirement is adaptive capacity, slack, and a safety budget that puts out fires faster than they emerge. Marcus therefore supports intervention in safety-critical deployment and sees explicit neurosymbolic constraints as one avenue, even if chatbots remain available for lower-stakes uses.

17. Competing scenarios converge on the need for verification

Marcus would slow AI 2027 substantially. He expects neurosymbolic approaches to gain traction within three or four years and thinks that by 2035 people may view LLMs as a “nice try”: still useful for distributional learning, but not the architecture that delivered deep semantics, planning, stable world models, or AGI.
Humans provide an existence proof for far greater data efficiency, while machines should eventually exceed human memory and avoid confirmation bias or motivated reasoning. Marcus expects progress dimension by dimension, followed by faster periods once missing ideas arrive; he disputes two-year AGI, not the eventual possibility of systems far beyond humans.
He also wants a distribution of scenarios because one vivid narrative can overwhelm readers’ probability judgment. Kokotajlo says AI Futures Project is building both a detailed good ending and a “scrappy spread” of shorter alternatives, while maintaining that present limitations will look more like road bumps than decade-long barriers.
Hendrycks’s strategic picture emphasizes espionage, transparency, and sabotage: frontier Slack systems and leaders’ phones may be hackable, while data-center power infrastructure can be disrupted with low attribution. Despite forecast and architecture disputes, all three favor verification around intelligence recursion, distrust voluntary lab governance, and agree that alignment is not on track to arrive before the capabilities it must govern.

Market Overview June 24, 2025 What Happened To Joyful June?