Geoffrey Irving

Irving’s central warning is that neither an imminent plateau nor a rapid path to transformative AI deserves high confidence, yet policy must assign significant probability to today’s methods continuing to scale. Obstacles might yield to more compute, data, scaffolding, or ordinary algorithmic progress, producing “further sigmoids” without a single breakthrough. The relevant posture is attention to capability growth alongside security and resilience—not certainty about a calendar date.

Current safety techniques may provide “a couple of nines” of reliability, but Irving does not see today’s empirical stack reaching the many nines required for catastrophic-risk control. Monitoring, honesty training, white-box detectors, access controls, and AI-control measures could fail together because training suppresses visible failures while selecting the remaining ones around a shared blind spot. The de facto plan is to use imperfect model safeguards to buy time, automate safety research, and “harden the world.”

Jagged capabilities are not a durable safety margin once even a model’s weak areas exceed human performance. Irving’s analogy is an elite Go player: professionals remain idiosyncratically jagged against one another, but against him they “just wipe the floor with me every single time,” even when he receives nine stones. Models also operate quickly—sometimes completing work roughly 10 times faster than humans—and remain difficult to interrogate reliably.

AISI’s red team has jailbroken every model in the evaluations where it tested safeguards, although stronger defenses still create meaningful friction. Across 80 evaluations covering more than 30 models or testing environments, “every time we did [safeguard testing], we jailbroke a model”; the time required is rising in heavily defended domains such as bio, reducing access for less capable attackers without establishing security. The distinction is between harm reduction and guarantees: the former is improving, while the latter remains absent.

Reinforcement learning is already improving models beyond mathematically verifiable tasks, weakening a common argument for an imminent capability ceiling. Irving points to models becoming much better at troubleshooting a biological experiment from a photograph as evidence that developers are training against self-critique and “hotchpotch versions of scalable oversight,” not merely objective math and code graders. Autonomy for exfiltration or replication still trails mundane software engineering, cyber, and bio capabilities, but it is rising too.

Irving believes alignment probably has a solution, but the binding uncertainty is whether humans reach it before increasingly coherent agents outrun supervision. In “50 years, 100 years, 1,000 years,” either humans or machines will solve alignment; theory suggests defenders can win when protocols are designed correctly, but practical information security shows how far implementations can remain from that limit. AISI is therefore funding complexity theory, learning theory, game theory, and cognitive science while admitting that none has yet produced firm guarantees.

Voluntary frontier-lab cooperation is producing real fixes, but open-weight diffusion ultimately shifts the burden toward infrastructure, public health, and international coordination. Anthropic and OpenAI have iteratively improved classifiers for current models after longer AISI red-team collaborations, yet capability-removal techniques such as data filtering, unlearning plus distillation, or gradient routing only “buy you some time.” Irving’s bottom line is institutional: independent research, government evaluation capacity, and non-model defenses must grow alongside the labs.