Nicholas Carlini

Image-classifier defenses deliver machine-learning accuracy, not security-grade reliability. Adversarial training can preserve roughly 50%–70% accuracy against the attack class used in training, which is meaningful progress from zero; operationally, however, 70% accuracy means an attacker can “try four times and probably one of them works.” That gap helps explain why layered controls and external constraints remain relevant.

The attacker’s structural advantage is both a lower success threshold and the ability to move second. Carlini reduces most technical attacks to two choices—the loss function and the optimizer—and finds that “the simplest possible objective is usually the best one,” because debuggability beats mathematical elegance. Defenders must anticipate many attacks before deployment; attackers can inspect the particular system in front of them, exploit its exact loss landscape, and spend six months on the one failure mode that matters.

Many supposedly novel defenses merely make gradients ugly, zero, or difficult to follow. Distillation once appeared robust because numerical saturation made its loss “actually identically zero”; floating-point changes or rescaled logits restored the signal. RepNoise argued that noisy activations prevented useful fine-tuning, while TAR trained against a weak one-step adversary and was vulnerable to multi-step optimization—after learning-rate changes, random restarts, and warmup, “deep learning takes over and the rest is easy.”

Open-weight safety remains technically unresolved, making future model capability a central release-risk variable. Carlini compares a permanently restricted model to a hammer that can build useful things but somehow cannot serve seven dangerous purposes: tools generally do not preserve such distinctions once users control them. He remains biased toward open source because it has historically benefited security and limits concentration of power, yet concedes that a future “magic box” able to compromise governments could justify a different answer.

Removing dangerous facts or training data does not reliably remove dangerous capabilities. Memorization depends heavily on repetition—GPT-2 examples might have appeared about 20 times in one document across roughly 10 epochs, or about 200 exposures—but models latch onto some sequences while ignoring others with no satisfying explanation. More fundamentally, a generally capable model could relearn omitted biology from textbooks in context, just as Gemini reportedly learned enough of a scarcely represented language from an in-context book to answer exercises.

The practical path is defense in depth around fallible models, with explicit utility costs. External software can forbid an agent from entering secrets into an input type="password" field regardless of what the model says, while detectors across many layers can make evasion progressively more damaging to task performance. These controls still face false-positive fatigue, custom interfaces, and unfamiliar attacks, so the practical stack looks more like human review, external action constraints, and layered detection than a single alignment technique.

AI cannot yet automate Carlini-style security research because the hard part is isolating the real bug inside messy systems. Models can attack a defense rewritten as a clean, 20-line homework exercise, but fail when dropped into an unfamiliar repository containing roughly 1,000 lines of real code; “the core of security” is stripping away the persuasive story and finding the one consequential mechanism. Human researchers and red teams therefore remain valuable, while attack automation is a dual-use frontier Carlini would reassess if capabilities improve sharply over the next couple of years.

🔗 Original source & video: The Adversarial Mind: Defeating AI Defenses with Nicholas Carlini of Google DeepMind

Carlini’s base case is not perfectly robust models, but secure systems designed around models that remain “very vulnerable” for the foreseeable future. Conventional security already tolerates imperfection, but ML has not reached the point where “a random person off the street” cannot run off-the-shelf code and force arbitrary failures. The practical implication is to design the surrounding system so it does not rely on the model alone.

GPT-3.5 Turbo Instruct’s chess play convinced Carlini that next-token prediction can reconstruct surprisingly rich internal state without explicit chess rules or chess-specific self-play reinforcement learning. From move notation alone, it almost always produces legal moves and often strong ones, implying “something internally that is accurately modeling the world.” Yet bizarre move histories elicit bizarre play: the base objective imitates its data rather than playing to win.

Arguments over whether models “reason” obscure the more decision-useful question: what behavior survives which conditions? Definitions split predictably between skeptics who exclude language models and AGI advocates forecasting displacement of human jobs by 2027. Carlini instead measures input-output behavior—and notes that the chess competence collapses under unfamiliar notation, just as distribution shift has always broken ML systems.

Post-training makes latent capabilities easier to elicit, but it can also damage properties investors may mistake for competence. Carlini recalled a GPT-4 report in which the base model was well calibrated—a 30%-probability answer was right about 30% of the time—before post-training “messed up” that calibration. Instruction following improves dramatically, but the model may look smarter without learning much new.

Carlini reports an approximately 50% programming-productivity improvement for himself, presenting it as an N-of-1 result. For an expert programmer, current models can act as a higher-level interface over code, APIs, and unfamiliar syntax. He asks when a task has perhaps a 50% chance of working, checks the direction, and completes the residual error himself: “95% solutions are still 95% solutions,” but fixing the final 5% can yield a “20x performance increase.” o1 even solved a dynamic-programming problem in two minutes with code 10 times faster than his own.

That productivity gain may compound security debt because models reproduce familiar vulnerabilities while expanding who can ship software. Carlini saw generated encryption code that should have been an API call and database code vulnerable to SQL injection. His concern is a future with “10 times as much code” but not 10 times as many security experts—the common case works, so insecure deployments proceed.

Disclosure practices remain an unresolved operating risk because some ML flaws are patchable software bugs while others resemble enduring scientific vulnerabilities. A model-stealing API weakness fit a 90-day coordinated-disclosure process; adversarial examples do not, because another 90 days cannot solve a problem researchers have attacked for 10 years. Carlini expects clearer norms within “a year or two,” but offers no settled framework today.

Model APIs do not fully protect proprietary architecture, and benchmark leadership does not reliably demonstrate general capability. Carlini’s team recovered a real production model’s final layer and undisclosed width using API outputs and linear algebra, though not the whole model; meanwhile, trainers are incentivized to optimize HumanEval, MMLU, and similar scorecards directly. “A lot more benchmarks” would make such targeted optimization harder and capability claims more credible.

🔗 Original source & video: Language Models are “Modelling The World” [Nicholas Carlini]