The Compendium - Connor Leahy and Gabriel Alfour
Summary
Connor Leahy and Gabriel Alfour argue that extinction from superintelligence is the main-line outcome on today’s trajectory because capability is scaling without a corresponding science of intelligence or control. Frontier labs publicly acknowledge the risks while continuing a “willy-nilly race” with little thought given to safety; the speakers’ goal is to buy more time rather than assume the trajectory is fixed.
The physical-world objection—that software cannot dominate without factories, materials, and machines—understates how deeply frontier AI is being integrated into the economy. Leahy’s answer is that an AGI could use corporations, markets, tools, human contractors, and supply chains as “extended cognition,” initially doing anything a corporation can do—and probably will, at least for a little while—before potentially inventing technologies beyond today’s industrial system.
The speakers reject confident claims that one special faculty cleanly separates humans, animals, and AI. Scientific attempts to locate that boundary have repeatedly failed; Alfour’s rule is that anyone announcing the “definitive difference” should be treated as potentially crankish. Quantitative gains can nevertheless create qualitative discontinuities—oral tradition, writing, clearer grammar, or fine-tuning across every deployed AI instance—so uncertainty does not imply safety.
Regulation need not wait for a perfect definition of intelligence, agency, or consciousness. Leahy compares that demand to tobacco companies insisting regulators first identify the exact carcinogenic chemical; the practical task is to draw “the smallest circle” guaranteed to contain the danger. Alfour’s perimeter includes project intent, compute thresholds, open-source frontier models, automated AI R&D, systems requiring less than 10 days of effort to jailbreak, and fully online deployments without gates or kill switches.
Current “alignment” work is, in Leahy’s account, largely product-satisfaction and PR work rather than preparation for a system more powerful than the rest of the world combined. A pleasant chatbot does not answer what values a de facto “global dictator” should implement, how it should resolve conflict, or how humanity could correct it. Leahy therefore calls a decades-long Manhattan Project the minimum credible effort: “If you can’t even put together all of our greatest scientists working on this for a generation, you’re not gonna make it.”
The economic upside case is inseparable from extreme concentration of power and an untested post-labor settlement. Alfour sees neither state allocation nor UBI—effectively taxing a handful of AI producers perhaps 90%—as credible after years in which governments failed to discipline Big Tech. His deeper objection to accelerationism is institutional: “If you don’t have rules, you don’t have competition”; unconstrained technological rivalry becomes a negative-sum race rather than productive capitalism.
Their political optimism rests on an unusually concrete test of democratic institutions, not faith in elite persuasion. Control AI cold-emailed UK MPs and lords, obtained 60 meetings, and won 20 supporters—about a 33% hit rate—for binding regulation addressing extinction risk. Proposed next steps include national cluster kill switches tested every 3–6 months, 15-minute emergency shutdowns, contingency plans, and a treaty activated only after signatories represent 35% of global GDP and 35% of world population.
Leahy’s final position is narrower than banning all AI but broader than regulating only today’s largest training runs: prohibit “unregulated, untrustworthy, non-consensual ASI” and precursors to it. He would not necessarily ban LLMs, but says open-sourcing GPT-5 would be “way too close for comfort.” The strategic bet is that ordinary people overwhelmingly prefer survival, while the pro-ASI constituency is a small, highly invested minority amplified by lobbying, online bubbles, and institutional passivity.
Deep dive
1. The Compendium argues that scaling is outrunning understanding
Leahy describes the 115-page November paper’s purpose without qualification: to make the complete case for why extinction from superintelligence is “very likely,” why the danger persists, and what could still be done about it.
Alfour says recent advances remain products of black-box iteration and additional compute, not a measured theory that factors intelligence into natural concepts. OpenAI, DeepMind, and Anthropic acknowledge the risks while racing anyway; “the world in which we learnt more about intelligence” is not the world presently producing frontier systems.
The host’s central puzzle is why intelligence can be grown rather than designed. Leahy calls that “one of the great mysteries of our universe”: shake particles in “a bit of a soup,” and eventually a human capable of art, mathematics, and nuclear weapons appears.
2. Intelligence remains a pre-scientific field
Leahy’s chimp analogy carries the uncertainty: adding more “neuron soup” somehow crosses from an unimpressive animal to a creature that invents mathematics. Neural scaling was not predicted by a deep theory; because researchers do not know why it works, they also cannot confidently predict where it stops.
The host pushes back that biological humans are astonishingly efficient self-replicators requiring neither power grids nor centralized orchestration. Leahy concedes that evolution optimized humans through another process, but calls that difference irrelevant to the narrower lesson: “weird things are possible,” and the feasible design space is hard to forecast.
His preferred historical analogy is alchemy. Researchers can observe important interactions among algorithms and compute—“some shit’s going on here”—without possessing the equivalent of chemistry: a settled science of what intelligence is, how it is structured, or how to build it deliberately.
3. No single faculty cleanly separates humans from animals
Against the host’s suggestion that human “merge,” Turing completeness, or general intelligence might constitute a decisive discontinuity, Leahy points to chimp politics: alliances, planned ambushes, rank changes, and expectations of repayment. Dogs plan and track identity; even a rat can display recognizable emotion.
The recurring scientific pattern, Leahy argues, is retreating exceptionalism: tool use supposedly defines humans until animals use tools; language replaces it until animal communication becomes undeniable. “There is no clean cut”—only more compute, broader patterns, better algorithms, and increasingly general cognition.
Alfour preserves the host’s intuition without endorsing its explanation. Humans can predict that another human generation might build nuclear weapons while another chimp generation will not; the meaningful admission is that scientists still cannot specify exactly what produces that gap.
His sharper epistemic rule follows: when someone claims the “definitive difference” between animals and humans, pre-ChatGPT and ChatGPT, or ChatGPT and humans, “you should be like, ‘Well, they’re a crank.’” This is a failed scientific problem, not an invitation to elevate personal intuition into theory.
4. Continuums can still produce dangerous discontinuities
Alfour argues that quantitative improvements can unlock qualitative transitions. Better language may enable oral tradition; systematic grammar can preserve knowledge longer; writing lets information survive the degradation of transmission. “Qualitative differences can emerge from mere quantitative differences.”
ASI could gain analogous discontinuities from deployment architecture. A human needs repetition to learn one fact and still forgets it; teaching everyone is practically impossible. For AI, fine-tuning every deployed instance could be trivial, allowing discoveries to propagate across the entire system at once.
5. An AGI would inherit the economy’s physical agency
The host’s objection is material: intelligence alone cannot manufacture chips, obtain resources, or replicate without factories. Leahy agrees that building things is hard, but notes that proposed AGIs are not isolated “on a rock in outer space”; these systems are being connected to the internet, tools, knowledge, people, and commerce as rapidly as possible.
No individual can build a lithography machine, yet the economy can. Leahy therefore models an emerging AGI as part of that larger system, using it as “extended cognition” to purchase equipment, commission research, pay humans, and do anything a corporation can do—and probably will, at least for a little while.
He hedges the longer horizon explicitly. A system smarter than the whole economy, given “a couple years” to develop technology, might continue using paid human labor—or might not; the point is that present industrial bottlenecks do not remain external to an intelligence embedded within the industrial apparatus.
6. Uncertainty expands the regulatory perimeter
When the host asks how regulation can be precise without agreed definitions of agency or consciousness, Leahy separates difficulty from impossibility. Not knowing exactly what intelligence is makes regulation hard; it does not establish that regulation cannot be done.
His tobacco analogy targets the demand for perfect metrology: companies argued that smoking should wait for regulation until scientists isolated the precise cancer-causing chemical. Leahy calls the equivalent AGI demand an “isolated demand for rigor”; uncertainty means drawing a larger protective circle, then shrinking it as knowledge improves.
The host challenges subjective extinction probabilities, citing the problem of acting on a 10% estimate. Leahy answers with a neighbor manufacturing pipe bombs near one’s children: criminal law and risk assessment routinely act on informed judgment because no objective frequency exists for a unique future accident.
Alfour proposes regulating observable precursors broadly: declared intent to build AGI or ASI, large compute runs, certain open-source releases, automated AI R&D without humans in the loop, systems requiring fewer than 10 days of effort to jailbreak, and fully online deployments with no gate or kill switch.
7. Five major ideologies turn AI development into a race for power
Leahy says there are “5 major ideologies” involved in the AI race and describes four in this discussion. Utopists expect a transhuman future and believe they should deliver it; accelerationists reduce the thesis to “More growth, more good. More technology, more good”; Big Tech treats AI as another source of power; opportunists follow whichever technology currently attracts capital and attention.
Their endgames differ. Utopists believe their own AGI produces the good outcome; accelerationists want nobody to monopolize it and therefore favor open source; Big Tech repeats “power, power, power”; opportunists, Leahy says, “don’t think at all” beyond what makes money.
The host grants Andreessen’s argument that growth has correlated with higher living standards over roughly 100 years. Alfour replies that fertility has crashed while growth continued and other conditions have worsened, making growth-as-universal-proxy less an intellectual model than an ideological starting point.
8. Post-scarcity could become concentrated techno-feudalism
Alfour outlines three broad outcomes: extinction, somewhat controllable superhuman intelligence, or post-scarcity. He treats extinction as the main line under current behavior, while emphasizing that they act because the trajectory might still be changed.
Post-scarcity proposals expose how weak the institutional foundation is. One answer resembles state allocation; the other is UBI, where a few AI companies produce everything, are perhaps taxed at 90%, and fund consumer markets under regulation strong enough to keep those firms from “strangling people.”
His pushback is historical and political: governments barely regulated algorithmic social media despite regulating newspapers, radio, and television. After years of that failure, why expect states to reclaim taxing power once AI firms possess “the entire economic power” and human labor provides no countervailing leverage?
9. Competition requires institutions, not faith in natural balance
Alfour rejects accelerationism’s claim to inherit Western liberal values. Productive competition exists because contract enforcement, externality control, and rules prevent competitors from crippling one another: “If you don’t have rules, you don’t have competition.” Without the frame, the result is street fighting, not a market.
The host’s libertarian countercase is distributed resilience: open-source AIs could counterbalance one another as natural systems do. Leahy answers that nature is “red in tooth and claw,” filled with parasites, cannibalism, violence, and suffering; seeing spontaneous order as benign reflects dependence on institutions one does not notice.
Leahy’s signature formulation is that libertarians are “house cats, completely dependent on a system they neither understand nor appreciate.” Chaos is the default—“entropy, chaos, and death”—while humane order is a fragile anomaly that must be built, maintained, and defended.
Alfour makes the underlying value explicit: humans—or, to some extent, sentient beings—should be able to flourish, pursue happiness, and escape suffering. He rejects sacrificing actual people for trillions of future nonhuman beings: “Either you care for the people and beings that are here, or you care for nonexistent things.”
10. Friendly chatbots are not the alignment problem
Leahy answers corporate self-regulation with ExxonMobil: oil engineers probably spend millions mitigating emissions, but their expertise and internal readiness framework do not eliminate the problem. The relevant question is why outside observers would grant frontier labs presumptive trust.
Leahy says labs have reframed alignment as “my users like my product,” calling this PR washing; he also says OpenAI renamed the original problem “superalignment.” In his usage, alignment means making AGI or superhuman intelligence act for human values in the world; refusal of illegal chatbot requests is not evidence that this problem has been solved.
The real question is what a system more powerful than the rest of the world combined should do as a de facto “global dictator.” Humanity lacks an accepted answer in morality, governance, public choice, psychology, sociology, or game theory—and lacks standardized definitions and measures of progress across much of that terrain.
11. Rules help only after humanity knows which world it wants
The host invokes Asimov-style rules as a legible alternative to opaque learning systems. Alfour says rule-based thinking is useful if it engages complexity: what constitution would humanity confidently hand to a world government, using terms people themselves understand rather than instructions such as “do whatever is good”?
“Do nothing” trivially prevents direct harm, but fails the actual objective. An aligned system might need to stop another misaligned ASI, World War III, or bioweapons; it must interact deeply enough with society to alter humanity’s trajectory without itself creating a worse one.
If the right constitution existed, Alfour would be “much, much more hopeful.” Technical safety would still need to handle edge cases and adversarial attacks—perhaps becoming conservative off-distribution—but making a system reliably follow known rules is more tractable than discovering the values those rules should encode.
Leahy distinguishes understanding from wanting. A model may represent human values without being motivated by them; alignment asks how to causally shape an unknown future decision system so that it consistently produces outcomes humans would morally endorse.
12. Corrigibility and boundedness expose how little control exists
Leahy places corrigibility below full alignment: the system permits modification or shutdown. Control is weaker still—it obeys instructions even when compliance harms the issuer. Which layer works may depend on whether ASI resembles a program, an organization, an economy, or a distributed assemblage rather than “one coherent blob.”
Alfour notes that human institutions rarely achieve alignment either. Constitutions emphasize amendment, while checks and balances, decentralization, representatives, councils, and ministries pursue boundedness: no component should accumulate enough power to dominate the rest.
Markets work similarly, not by aligning every participant with the common good but by imposing strict conditions under which rivalry can produce useful outcomes. Big Tech’s ability to resist correction through lobbying and FUD campaigns represents both failed boundedness and “anti-corrigibility.”
The scale gap is decisive: the task is not merely preventing today’s firms from abusing power, but ensuring that a company with “a billion times” its current power remains corrigible. Asked for a constitution immune to Goodhart’s law, Alfour gives his honest answer: “People say I’m smart, but I’m just me.”
13. Democratic institutions offer a measurable route to buying time
Alfour’s governance program begins with survival, bounded experiments, clearer definitions, objective indicators, and iteration. France, he says, constitutionally permits testing some laws locally before expansion; this is the kind of 21st-century governance suited to complex systems rather than one-shot ideological solutions.
The human bottleneck is not simply IQ. Alfour names “traumas, compulsions, and trolls”: researchers avoid necessary conversations, cling to favored failed methods, or let ideologies erase inconvenient values. He says at least 2–5% of people may be able to follow sufficiently complex rules and create better methods, while far more can understand enough for representative democracy.
Control AI’s proof of concept was deliberately simple: cold-email every UK MP and lord, hold 60 meetings, and ask for binding regulation addressing extinction risk. Twenty supported the statement—a 33% conversion rate—despite experienced lobbyists predicting that including “extinction” made the effort impossible.
Concrete proposals include national contingency plans, cluster kill switches tested every 3–6 months, emergency 15-minute shutdowns, and a treaty activating only at 35% of world GDP plus 35% of population. Leahy adds that a 1–2% chance of an undiscovered proto-ASI would mean it is already too late; otherwise, governments could simply stop or ban the development of ASI.