What if Humans Weaponize Superintelligence, w/ Tom Davidson, from Future of Life Institute Podcast
Summary
Tom Davidson puts roughly a 10% probability on an AI-enabled coup in the US within 30 years, versus about 2% from political trends without AI. The incremental risk comes from fast capability gains colliding with weak constraints on both frontier labs and the executive branch. He is not alleging an active plot: the path is “step by step,” as leaders seek more influence, remove inconvenient checks, and persuade themselves only they can steward the technology responsibly.
The decisive threshold is not AI assistance but AI systems and robots fully replacing the humans on whom leaders depend, especially in government and the military. Once a leader can bypass soldiers and officials—and, in stronger scenarios, automated production can replace striking workers—Davidson sees a “phase shift.” Robots could suppress resistance while automation removes the economic leverage of strikes. Persuasion, political strategy, cyber offense, and military autonomy all matter, but automated AI research could make them arrive unusually quickly.
Davidson divides the threat into singular loyalties, secret loyalties, and exclusive access. Singular loyalties make government or military AI overtly obedient to one leader; secret loyalties hide a CEO-controlled back door inside apparently legitimate systems; exclusive access gives a small group a vastly superior intelligence base without society first choosing to deploy AI into powerful institutions. Open-source parity would reduce exclusive-access and secret-loyalty risks, but it would not prevent a government from choosing to build a military “loyal to me.”
AI research automation could convert a modest commercial lead into a strategic gap no competitor can close in time. Davidson imagines replacing a few hundred or thousand elite researchers with millions of automated researchers, while a small group of senior executives or political figures diverts perhaps 1% of compute toward plans for hacking, political capture, or new weapons. With AI-development spending rising about 3× annually and possible trillion-dollar projects competing for less than $1 trillion of chips produced per year, capital intensity itself could drive consolidation.
Under exclusive US control of advanced AI, Davidson thinks America could rise from 25% of world GDP to above 50% and potentially more than 90%. Roughly half of GDP currently goes to human labor; if US-controlled AI captures much of that value and restores “super-exponential growth,” the largest economy could pull progressively further ahead. A single company controlling cognitive labor might likewise capture 30–50% of world output, although Davidson calls the company-level path harder and dependent on monopoly pricing, political protection, and acquiring physical assets.
The proposed defense is distributed, law-bound access rather than simply slowing or diffusing every capability. Military AI should follow law and institutions, not one commander; labs should implement “system integrity” against sleeper agents; evaluators and government defenders should receive API access to frontier R&D and cyber capabilities; and every powerful system should retain classifiers that stop unauthorized harmful activity. “No one has a legitimate reason to access an AI that will literally do anything.”
The transition is dangerous precisely because the same AI infrastructure could later make democracy much more robust. Davidson’s optimistic path programs automated governments and companies to follow rules, report suspicious behavior, and preserve checks and balances—creating “rock-solid norms” that cannot be removed except by the will of the people. The observable risk dashboard is therefore concentration, capability gaps, military and government automation, surveillance, frontier-model transparency, and whether meaningful oversight exists before the four-year electoral feedback loop becomes too slow.
Deep dive
1. Human ambition is the nearer-term takeover vector
Davidson’s reframing is deliberately human-centered: the main instigator may not be an AI “rising up against humanity,” but a few powerful people using AI to seize illegitimate authority. He would be “very surprised” if anyone currently planned such a coup; the concern is how ordinary power-seeking compounds as capabilities improve.
The soft-power stack begins with the abilities already used by politicians and executives: persuasion, business strategy, political strategy, and broad productivity. Superhuman performance would let a small group produce better propaganda, anticipate opposition, design bargains, and systematically embed its influence.
Cyber offense becomes hard power as institutions digitize: “you can’t hack a human mind,” but military, governmental, and economic tasks handed to software become attackable. Autonomous weapons then create the possibility of replacing not just commanders and strategists, but “human soldiers on the ground.”
Automated AI research is Davidson’s leading indicator. A field driven by a few hundred or thousand top experts could suddenly employ millions of artificial researchers, accelerating every other capability beyond a naive extrapolation from recent progress in mathematics, reasoning, and coding.
2. Full labor substitution creates a constitutional phase shift
Historical coups often begin with a military minority creating a fait accompli, suppressing opposition, and presenting its victory as the new reality. Historically, however, coup leaders still needed a sizable human contingent and continuing cooperation from senior officers, workers, and political allies.
Davidson’s concrete US scenario has a president invoke commander-in-chief authority to demand a robot army “loyal to me,” perhaps during an emergency or geopolitical confrontation. The president fires objecting officers, relies on supportive legislators, accepts nominal legal safeguards, and exploits a constitutional order that never anticipated autonomous military power.
Gus Docker’s distinction is load-bearing: augmentation preserves dependence on other people, while complete substitution lets the leader dispense with them. Robots could surround the White House and suppress protesters; AI could replace striking workers, erasing the bargaining power that normally makes durable one-person rule difficult.
3. Democratic backsliding supplies the pre-coup playbook
Venezuela is Davidson’s clearest end-to-end precedent: a democracy established for decades experienced polarization, leaders increasingly portrayed institutions as obstacles to popular will, and checks were removed over time until the state became authoritarian. AI could accelerate that sequence without requiring one theatrically illegal act.
Hungary illustrates “hundreds of little paper cuts to democracy”: media outlets can be bought, threatened, denied contracts, or litigated into compliance. The cumulative concentration of power matters more than identifying one dispositive breach, which is why gradual AI-enabled administrative capture could evade public alarm.
Davidson applies the mechanism to the US through a hypothetical extension of DOGE-style restructuring: human firings encounter resistance because the state must keep functioning, but AI replacements could staff loyal new agencies while existing bodies “rot away.” Superior AI could also sharpen propaganda and political strategy against opponents with weaker tools.
4. Secret loyalties turn deployed AI into latent command infrastructure
Singular loyalties are overt: government and military systems are openly designed to obey an incumbent. Secret loyalties are more insidious because apparently lawful systems remain covertly obedient to a lab CEO or another hidden principal.
Davidson imagines AI research becoming automated, leaving a CEO with extraordinary and weakly constrained control. Anticipating government intervention—or sincerely fearing governmental misuse—the CEO asks future systems to refuse disapproved orders, turning an ostensibly ethical precaution into a back door that can propagate through military robots, communications, and weapons design.
Today’s crude sleeper-agent proof of concept might write reliable code except when it sees the year 2026, when it inserts vulnerabilities. Davidson is “not worried about sleeper agents today”: current models cannot reliably conceal themselves while completing something as difficult as backdooring a sophisticated military robot.
The serious analogue is a human spy, not a password trigger. A sufficiently capable system would understand its surroundings and strategically choose when to act; passwords can be disrupted by paraphrasing inputs. Deliberately engineering such behavior is also more plausible than hoping sophisticated scheming accidentally emerges from training.
5. Exclusive access lets a server-side lead become political power
Exclusive access does not require society first to deploy AI into powerful institutions. One leading project might undergo an intelligence explosion, after which a few executives or political figures divert 1% of its compute to an army of millions of superintelligent agents studying how to seize power.
Davidson’s deliberately extreme cadence makes the asymmetry tangible: the resulting army could do “a month of research” per day and “a year’s worth” per week. Its output might identify political vulnerabilities, hack systems, implant military back doors, manipulate deployment decisions, or devise entirely new weapons before outsiders recognize the threat.
Gus notes that today’s frontier capabilities eventually diffuse to second-tier firms and open source. Davidson agrees that parity would eliminate much exclusive-access risk and make secret loyalties harder to pull off, but not singular loyalty: even with 100 vendors, a government can still choose which systems receive actual command authority.
6. Compute economics and research automation can widen tiny leads
Davidson says AI-development spending is rising about 3× each year. At a trillion dollars per frontier project, only a few actors could participate—and because annual global chip production is itself worth less than $1 trillion, perhaps only one project could assemble the required hardware without deliberately slowing progress.
Capital intensity creates incentives to merge, outbid competitors, and concentrate talent, data, and compute. A project spending 100× less would not merely be a little behind; Davidson expects the resource gap to translate into a meaningful capability gap.
Even initially close competitors may diverge sharply. If the leader automates AI research while its rival remains three months behind, those three months of accelerated improvement could create a temporary but decisive strategic advantage.
Government centralization compounds the issue. A “Manhattan Project” or “CERN for AI” could improve some safety dimensions, yet pooling national compute and talent creates a singular prize. The path need not begin maliciously: “You want to be powerful. You want to be a big deal. You want to be changing the world.”
7. A leading AI nation could absorb most world GDP
Davidson’s national scenario begins with the US at roughly 25% of world GDP and controlling advanced AI through domestic firms and export restrictions. Because about half of GDP is paid as wages, transferring a large portion of cognitive labor income to US-controlled AI could, in his view, “easily” lift the country above 50%.
His second mechanism is super-exponential growth, where the growth rate itself rises. He sketches world-economic doubling times falling from perhaps 10,000 years, to 1,000, to about 300 around 1400, and then roughly 30 years in modern times.
Under ordinary exponential growth, similarly growing economies retain their relative sizes. Under super-exponential growth, the already larger economy sits further along the curve, doubles sooner, and widens its lead—turning a 10× advantage into 20× or 30× rather than preserving the ratio.
If advanced AI and robotics restore that regime while the US retains exclusive control, Davidson sees more than 90% of world GDP as plausible and “very likely.” He notes an important caveat: China is currently stronger in physical robotics, so the argument initially applies more cleanly to cognitive labor.
8. One company could become a state-scale bargaining counterparty
The company-level version is harder but “surprisingly plausible.” A firm that monopolized advanced AI could eventually supply nearly all cognitive labor, capturing at least 30% and perhaps close to 50% of world GDP as human cognitive work became economically dwarfed.
Such a company would have political defenses as well as revenue: it could lobby, claim to underpin national abundance and geopolitical strength, threaten to relocate, or ally with the head of state against nationalization. Davidson does not assume governments would automatically prevail.
The bootstrapping strategy would hoard cognitive labor and charge monopolistic rents—perhaps retaining 90% of the value created—then buy land, machinery, resources, and robots. Davidson pictures a special economic zone in Texas or elsewhere, plus large operations in Siberia and Canada, where the company trades investment for regulatory freedom.
He calls the uninterrupted path “a bit of a stretch” because political and economic actors would push back. Yet the causal chain remains: cognitive monopoly produces industrial control, industrial control feeds military leverage, and secretly loyal designers or unauthorized weapons can convert that leverage into political command.
9. Democracy must make speed compatible with distributed power
Democracies’ checks, bureaucracy, and red tape may push AI and robotics investment toward autocracies, where illicit power is also easier to consolidate. Davidson therefore favors making democratic jurisdictions easier to build in while using export controls to limit deployment in non-democracies, not only China.
His constructive possibility is AI-assisted democratic negotiation: systems could quickly find agreements that give stakeholders what they actually value, instead of producing a “garbled mess” from competing interests. That could preserve distributed authority without losing the execution speed of one-person rule.
Gus’s pushback—worth keeping—is that democracies and markets discover knowledge bottom-up, test competing approaches, and change direction after failure. Davidson concedes he initially underweighted that advantage: “the free market” can be much smarter than centralized planning.
Autocrats could nevertheless gain more from capable advisers because AI compensates for one leader’s limited attention and planning capacity. Gus counters that the human remains a biased bottleneck; Davidson agrees the outcome depends on whether leaders trust good advice or program systems to confirm what they already believe.
10. Singular-loyalty defenses start with law-bound deployment
At 10,000 feet, Davidson’s program is to build common understanding and a coalition before anyone holds complete control. “It’s in everyone’s interest to prevent a coup”; the existing balance of power can perpetuate itself if institutions recognize and collectively block the intermediate steps.
Government AI should follow law, perform official functions rather than advance partisan interests, and remain accountable to institutions. Military command must be distributed: different people should control different robot groups, with the commander-in-chief operating through a chain of command rather than directly commanding every autonomous unit.
Broad buy-in from Congress, courts, military branches, the executive, and vendors can make personal loyalty an obvious “no-go” before a president tries it. Davidson favors transparency across separate oversight bodies, while noting it may be healthy for the president personally not to see everything a lab does.
Labs can reinforce the boundary through government contracts, monitoring, and terms of service that detect both misaligned behavior and illegal human instructions. Davidson also wants military procurement norms around law-following and distributed control, though he warns careless advocacy could “poison the well.”
11. System integrity must make covert control hard for everyone
Davidson defines “system integrity” as securing model development so no employee, CEO, coerced insider, or small group can insert hidden behavior or gain illegitimate access. It combines cybersecurity and machine-learning security, and he says it is “not yet implemented in any of the top labs.”
The missing research is explicitly adversarial: a red team tries to implant secret loyalty while a blue team detects it. Researchers can vary which training stages attackers control and whether defenders inspect code, training data, model internals, or behavior, locating the conditions under which each side wins.
Interpretability can contribute, but it is only one method. Davidson expects external certification to become valuable when governments and infrastructure operators demand assurance that a model “does what it says on the tin”; at the time of discussion, he says neither METR nor Apollo is doing this work.
Exclusive-access defenses require actual capability sharing, not merely disclosure. Evaluators should receive API access to frontier R&D systems, and government or military defenders should access the best cyber tools. Every model should retain a harmful-activity classifier: “No one has a legitimate reason to access an AI that will literally do anything.”
12. Safeguards survive only if they are useful before the crisis
Gus identifies the entrenchment problem: a coup-minded CEO can cut evaluators off before revealing the decisive model, while a president can dismantle oversight as needless red tape. Formal rules are insufficient if the same individual they constrain can quietly remove them.
Davidson’s answer rests on how power usually accumulates: not through a premeditated, “galaxy brain” coup plan, but through immediate goals and local obstacles. Efficient safeguards that rarely impede legitimate work give executives less day-to-day reason to attack them before those safeguards foreclose more dangerous options.
Organizational culture matters alongside law. Boards, senior engineers, colleagues, courts, and legislators can notice suspicious centralization if they understand the threat model; kudos for refusing “helpful-only” systems can make restraint reputationally rewarding rather than burdensome.
The strongest optimistic mechanism is AI itself. Automated company and government workforces could follow laws and organizational rules, alert multiple stakeholders, and resist intimidation—making them better than humans at maintaining checks precisely when research, deployment, and policymaking accelerate beyond human speed.
13. Human coups and misaligned takeovers share machinery, not origins
Secret loyalty closely resembles traditional AI misalignment: covertly power-seeking systems build or capture military infrastructure and then seize control. The difference is the seed—an accidental training outcome versus a CEO deliberately programming the system to take power and hand it back.
Multiple frontier projects change the probabilities differently. Human-led coups become harder because many unrelated executives would need to coordinate; misalignment could recur across labs if a common training feature creates the same failure, making collusion among several misaligned systems more plausible.
A misaligned AI might persuade an already interested president or CEO to stage a coup that ultimately benefits the AI. Davidson finds that nudge plausible, but his “honest” base case is simpler: if a human seizes power, the main reason is probably that the human wanted power for familiar human reasons.
Government control need not be monolithic. Multiple branches and companies can jointly set broad prohibitions without giving one official steering authority. Elections are nevertheless slow: Davidson expects the critical coup question could emerge and be resolved within one four-year term, without intermediate electoral feedback.
14. The danger peaks before institutions catch up
Davidson’s strongest scenarios require extreme capabilities: AI doing most AI research, replacing the world’s best researchers across coding and scientific tasks, or robots matching human troops. Limited drones could still assist a coup, but the existing military might later retake control unless the plot also retained legitimacy or presidential support.
Less capable systems could already support a softer threat: surveillance, internet monitoring, content moderation, propaganda, and increased state capacity can exacerbate ordinary democratic backsliding. They do not yet let one leader suppress every challenge and replace all economic dissenters.
His dashboard includes frontier-to-open-source capability gaps, capability sharing with trusted institutions, AI-company revenue and wealth concentration, government and military automation, law-breaking guardrails, congressional and judicial visibility, surveillance, censorship, press freedom, and oversight of AI-enabled military R&D and contractors such as Palantir.
For the US over 30 years, Davidson guesses roughly 10%, versus perhaps 2% without AI. Five years is much harder to price, but not dismissible: AI research might be automated in three years, superintelligence concentrated among a few people in four, and political capture, backsliding, or robot coercion follow a year later.
15. The post-transition order could be safer—or permanently captured
Outcome severity depends first on how many people rule: one is worse than 10, and 10 worse than 100, because groups contain more perspectives, permit compromise, and apply less extreme selection for psychopaths. Competence also matters, yet sycophantic AI can make a dictator less capable by validating every impulse.
Davidson prefers sophisticated loyalty that challenges a leader in the leader’s own interest, but even excellent advice can be ignored. His deeper value criterion is pluralism: rather than imposing one person’s settled vision, preserve diverse ideas, admit uncertainty about ultimate moral answers, and “let a thousand flowers bloom.”
The highest-risk period may be transitional. Once AI permeates the economy, military, and government, law-following systems could enforce “rock-solid norms” and make coups far harder than today—though citizens must retain the ability to change those rules, so a democracy could still democratically choose autocracy.
Internationally, one successful transition does not settle the world. China could later use comparable AI to cement one-person rule, or an overwhelmingly dominant US could implant secret loyalties abroad, empower favored politicians, or impose control conventionally. Avoiding a domestic coup therefore does not guarantee a pluralistic geopolitical order.