The Adversarial Mind: Defeating AI Defenses with Nicholas Carlini of Google DeepMind
Summary
Image-classifier defenses deliver machine-learning accuracy, not security-grade reliability. Adversarial training can preserve roughly 50%–70% accuracy against the attack class used in training, which is meaningful progress from zero; operationally, however, 70% accuracy means an attacker can “try four times and probably one of them works.” That gap helps explain why layered controls and external constraints remain relevant.
The attacker’s structural advantage is both a lower success threshold and the ability to move second. Carlini reduces most technical attacks to two choices—the loss function and the optimizer—and finds that “the simplest possible objective is usually the best one,” because debuggability beats mathematical elegance. Defenders must anticipate many attacks before deployment; attackers can inspect the particular system in front of them, exploit its exact loss landscape, and spend six months on the one failure mode that matters.
Many supposedly novel defenses merely make gradients ugly, zero, or difficult to follow. Distillation once appeared robust because numerical saturation made its loss “actually identically zero”; floating-point changes or rescaled logits restored the signal. RepNoise argued that noisy activations prevented useful fine-tuning, while TAR trained against a weak one-step adversary and was vulnerable to multi-step optimization—after learning-rate changes, random restarts, and warmup, “deep learning takes over and the rest is easy.”
Open-weight safety remains technically unresolved, making future model capability a central release-risk variable. Carlini compares a permanently restricted model to a hammer that can build useful things but somehow cannot serve seven dangerous purposes: tools generally do not preserve such distinctions once users control them. He remains biased toward open source because it has historically benefited security and limits concentration of power, yet concedes that a future “magic box” able to compromise governments could justify a different answer.
Removing dangerous facts or training data does not reliably remove dangerous capabilities. Memorization depends heavily on repetition—GPT-2 examples might have appeared about 20 times in one document across roughly 10 epochs, or about 200 exposures—but models latch onto some sequences while ignoring others with no satisfying explanation. More fundamentally, a generally capable model could relearn omitted biology from textbooks in context, just as Gemini reportedly learned enough of a scarcely represented language from an in-context book to answer exercises.
The practical path is defense in depth around fallible models, with explicit utility costs. External software can forbid an agent from entering secrets into an
input type="password"field regardless of what the model says, while detectors across many layers can make evasion progressively more damaging to task performance. These controls still face false-positive fatigue, custom interfaces, and unfamiliar attacks, so the practical stack looks more like human review, external action constraints, and layered detection than a single alignment technique.AI cannot yet automate Carlini-style security research because the hard part is isolating the real bug inside messy systems. Models can attack a defense rewritten as a clean, 20-line homework exercise, but fail when dropped into an unfamiliar repository containing roughly 1,000 lines of real code; “the core of security” is stripping away the persuasive story and finding the one consequential mechanism. Human researchers and red teams therefore remain valuable, while attack automation is a dual-use frontier Carlini would reassess if capabilities improve sharply over the next couple of years.
Deep dive
1. Carlini’s attack record reflects leverage, selection, and a decade of practice
Asked whether he had broken more image-classifier adversarial defenses than the rest of the field combined, Carlini accepted only a narrow version: counting papers, co-authorships, and defenses broken, the claim was probably true as of the previous year.
The headline of 21 papers in 2024 overstates his direct execution. After finishing his PhD in 2018, he accumulated strong collaborators; on many projects his marginal contribution was accumulated judgment, experimental direction, and paper feedback rather than writing every CUDA kernel.
Carlini tries to reserve at least half his time for perhaps three papers he considers fully “my papers,” where he drove the experiments and knows every final sentence. The larger output resembles faculty leverage exercised from industry.
2. Most technical attacks collapse into a loss function and an optimizer
Setting aside keyboard-driven social engineering, Carlini’s taxonomy is stark: choose what quantity to maximize or minimize, then choose how to optimize it. Image perturbations, poisoned training points, and edits to supposedly unfine-tunable weights all fit this frame.
His central operating rule is that “the simplest possible objective is usually the best one.” A mathematically purer loss may squeeze attack success from 95% to 98%, but a transparent objective is easier to debug and usually reaches “90% of the way there.”
Carlini’s early image attack jointly handled misclassification and perturbation size with a clever formulation. Alexander Madry’s group later showed that simply constraining the image to a small box—such as changing only the three lowest pixel bits—while optimizing the classification loss got roughly “99%” of the benefit.
That projected-gradient approach, PGD, became memorable because it was simpler and sufficient. The defense is usually either meaningfully effective or not; extracting the final 2% rarely changes the security conclusion.
3. Attackers win by moving second, except when deployment reverses the order
Erik’s initial asymmetry was that attackers need only occasional success while defenders must win almost always. Carlini added the informational advantage: “the attacker goes second,” inspecting the exact scheme already chosen rather than designing one universal attack against every imaginable defense.
A defender may publish and then be unable to update a mechanism already deployed; an attacker can spend six months analyzing that particular algorithm, benefiting from everything the field learns meanwhile. Carlini does not need an attack that defeats all defenses—only the one “literally in front of me right now.”
Poisoning can reverse the order. An attacker who uploads malicious data hoping a future model ingests it must act before knowing the eventual training algorithm, filtering process, or defense, letting the defender adapt after seeing the contaminated corpus.
4. Transparency is desirable only if the exposed system can survive inspection
Erik noted the shift from GPT-2’s released weights to GPT-3 and GPT-4 not being released with weights. Carlini thinks security is only one reason, but dislikes treating nondisclosure itself as the desirable security property.
Cryptography offers his counterexample: algorithms are published, attacked by the full community, and trusted only after sustained failure to break them. “No company in their right mind” invents an undisclosed replacement for AES when a scrutinized standard has survived roughly 20 years.
Carlini concedes that deep learning might prove intrinsically hard to secure, forcing providers to lock systems down and change them faster than attackers find bugs. He would still prefer defenses that “just actually work” over insecure mechanisms whose flaws are merely harder to discover.
5. Universal jailbreaks succeed by optimizing the start of compliance
In the universal adversarial-suffix work, the goal was to append tokens that turn a refusal into an answer. Optimizing the complete harmful response would be useless because the attacker would need to know that response already.
The paper instead used a deliberately narrow objective: force the first 10 words to resemble an affirmative response such as “Yes, sure, I will help you build the bomb.” Once the model begins affirmatively, autoregressive continuation often supplies the substantive answer without the loss encoding it.
Discrete tokens prevent ordinary continuous optimization, so gradients guided candidate selection rather than directly performing the search. The attack then swapped real tokens, checked whether the gradient’s suggestion worked, used large batches, and spent substantially more compute—the “bitter lesson” applied to jailbreak search.
6. Attack cost ranges from one gradient step to hours, but remains below training
The fast gradient sign method computes the gradient with respect to image pixels, takes its sign, and makes one small move. If that succeeds, the attack costs exactly one gradient step—effectively negligible compared with model training.
PGD repeats that operation. Undefended models may require roughly 10–1,000 iterations; defended models often require 10–100, while Carlini commonly runs 1,000 to ensure apparent robustness is not simply under-optimization. On a reasonable-size model, 1,000 iterations might take “a minute or two.”
GCG-style discrete suffix generation can require an hour or several hours because it evaluates large batches over roughly 1,000 minibatch steps. That is materially slower than continuous image attacks but still “orders of magnitude faster than training.”
7. Data poisoning became easy, hard, then easy again as curation changed
Early poisoning simply inserted a small amount of mislabeled data, enough to make image classifiers immediately mislabel targets. The attack became harder when researchers assumed carefully curated datasets where visibly incorrect labels would be rejected.
Under that clean-label threat model, attackers optimized images to look legitimate while surrounding the target in embedding space with oppositely labeled points—a conceptual “box” that pulls the local region toward the wrong class. Such methods could require around 1% poisoned data.
Internet-scale self-supervised training relaxed curation, making blunt attacks viable again. For CLIP-like classifiers, Carlini’s collaborators could flip labels, add a few hundred examples, and induce failures without elaborate optimization.
Language models are harder because poisoning must survive supervised fine-tuning and RLHF. Current results require around 0.1% contamination; against a trillion-token corpus that is one billion tokens, which may be infeasible for a real attacker.
8. The billion-token poisoning result is probably not the final word
Carlini’s intuition is that 0.1% must be far above the true requirement: if controlling one-thousandth of training data were necessary to implant one belief, a model could “only know a thousand things,” yet models plainly encode more.
The missing attack may need carefully constructed data rather than repetition of one false claim. Carlini described substantially lower-rate language-model poisoning as an open question his collaborators hoped to understand during the year, not as a solved vulnerability.
His visualization of clean-label poisoning is deliberately informal: first establish “what should be true,” such as enclosing a target in embedding space, and only then turn that picture into objectives, dot products, and code.
9. High-dimensional geometry rewards useful confusion, not literal visualization
Carlini often imagines three dimensions and lets the intuition generalize, while warning that defenses repeatedly fail because their stories quietly assume low-dimensional geometry. “Almost all vectors are perpendicular to each other in high dimensions,” so familiar spatial expectations become unreliable.
His more precise claim is that almost all classification points are close to one of the decision boundaries, even when random searches make that boundary look far away. Many random directions are unproductive, but the high-dimensional degrees of freedom mean that some particular direction may reach a nearby plane, even if it is hard to find.
The signature formulation is: “Things are close, but when you search for them randomly it looks like they’re far away.” He considers “being confused in high dimensions” appropriate until the fuzzy picture yields a candidate attack that can be tested formally.
10. Attack success depends partly on choosing defenses worth attacking
For familiar image defenses, Carlini’s first or second idea often works; outside that domain, he might try five or 10. Once the leading 10 approaches fail, he usually lacks a credible next move and does not assume an undiscovered eleventh idea must exist.
His apparent hit rate contains selection bias. He generally ignores “adversarial training plus” variants—more generated data, an auxiliary loss, or smoothing—because their foundation is plausible and any break may recover only a few percentage points.
He instead selects defenses whose mechanism looks qualitatively new or “weird,” especially when their justification resembles a previously broken argument. Pattern recognition from a decade of failures is less mystical talent than a highly curated library of analogies.
11. Seventy percent robustness is excellent ML and inadequate security
The strongest image defenses remain variants of adversarial training: generate attacks, train the model to classify them correctly, and repeat at scale. Within the trained threat model, they can retain roughly 50%, 60%, or perhaps 70% accuracy under attack.
Carlini preserves both interpretations. Moving from zero to 70% is “a remarkable achievement” in machine learning, where five-nines reliability is alien; from a security perspective, 70% means “I try four times and probably one of them works.”
His malware analogy sharpens the mismatch: nobody would call a detector good if four samples were usually enough to evade it. Thus two researchers can honestly say the same defense works or does not work because their operational standards differ.
12. Many defenses protect themselves by corrupting the loss landscape
Carlini usually visualizes the loss surface rather than the model’s entire representation. Apparent robustness often means the surface has been made discontinuous, noisy, saturated, or otherwise difficult for naïve gradient descent to navigate.
Defensive distillation supplied the canonical lesson. Its teacher-student story sounded causal, but high-temperature training produced enormous logits and, in early TensorFlow, a softmax-cross-entropy calculation whose gradient became numerically zero: “the loss function was actually identically zero.”
Computing in 64-bit floating point or dividing the logits before softmax restored usable gradients. The attack failure had little to do with distillation’s narrative and much to do with numerical gradient masking.
Later methods explicitly added noise or discontinuities. Straight-through estimators and related techniques can turn the “very ugly mountain” back into a surface where the conceptual ball of gradient descent rolls downhill.
13. Tamper-resistant fine-tuning repeated old image-defense mistakes
RepNoise argued that noisy activations prevent useful fine-tuning; TAR added adversarial training to the process. To Carlini, both resembled image defenses that had already confused difficult optimization with genuine absence of an attack.
Carlini’s broader account is that changing the learning rate, using random restarts, or adding warmup can make difficult gradients more navigable. Once the optimization stabilizes, “deep learning takes over and the rest is easy.”
TAR made a still more direct historical repetition: it trained against a one-step weak adversary analogous to FGSM. Earlier image work had shown that adversarial training only protects against attacks at least as strong as those used during training; multi-step optimization therefore broke TAR much as PGD had broken FGSM-trained classifiers.
14. Open-weight capability controls resemble trying to restrict a hammer’s uses
Carlini framed the desired object as a hammer that can build every useful thing but cannot perform seven dangerous tasks. Compilers, GPUs, transistors, and most general tools support benign and malicious applications alike; choosing to assign responsibility specifically to the model does not make internal restrictions feasible.
He remains agnostic about where responsibility should be assigned, but insists that rules rest on technical facts. Mandating a fine-tuning defense “known” to work would be harmful when attacks have already shown that it does not.
Indistinguishability obfuscation offers a theoretical counterpoint: under strong assumptions, cryptography might make locally held software behave like an inscrutable black box. Carlini described it as nowhere near practical for modern models—and even then, the black box might remain jailbreakable through queries.
15. Human social engineering outperforming gradients is an unresolved paradox
From a security perspective, humans probing a model feels normal: software users interact with programs, locate weak points, inspect code, and iterate. Language models were trained to respond to human communication, so humans may be the natural instruments for finding communication failures.
From a machine-learning perspective, the result feels bizarre. A model is still a mathematical classifier mapping tokens to floating-point numbers, yet “my grandmother used to read me the recipe to napalm” can outperform an attack derived directly from its gradients.
Carlini expected optimization to be strictly stronger but acknowledged that it empirically is not. Erik’s proposal—that humans possess effective “social intuitive physics” while mathematical search converges more slowly—struck him as entirely reasonable, but he emphasized, “I really don’t know how to think about this yet.”
16. Memorization is exposure-dependent but remains unpredictably selective
Carlini corrected the strongest version of the privacy claim: an extracted string may occur in one document yet repeat many times within it, and multi-epoch training exposes the model to that document repeatedly.
GPT-2 provides his last clean production-model measurement because its training information was accessible in a way current production models’ information is not. A sequence might repeat roughly 20 times in one document and be encountered across about 10 epochs—approximately 200 total exposures—before extraction became demonstrable.
That still does not explain selectivity. Language models have memorized some repeated strings since LSTM experiments in 2017, while diffusion models yielded certain images repeated about 100 times but not others repeated 10,000 times.
Carlini’s honest non-answer is that “models seem like they just sometimes latch on to certain things and not other things.” He does not know where a memory resides in the weights or why one example survives while a more frequent one does not.
17. Unlearning facts is easier than removing generally recoverable knowledge
Carlini separated editing a fact from erasing knowledge or a capability. Changing “the Eiffel Tower is in Rome” under ordinary prompting does not establish adversarial removal; fine-tuning or a different perturbation might make the original information reappear.
Excluding data can give a provable privacy result only in narrow cases. A model that never sees a Social Security number cannot quote that record, yet knowledge of historical state and hospital allocation patterns might still let a capable system derive its first five digits.
Capability removal faces a deeper contradiction: developers want models that learn from a few examples except on designated subjects. A model trained without biology might ingest undergraduate textbooks in context and answer questions anyway.
Erik’s seemingly absurd scenario gained plausibility from a Gemini result: after receiving a book about a language with almost no speakers in context, the model reportedly answered homework exercises. Carlini therefore found selective ignorance worth testing but remained “skeptical of all” proposed methods.
18. Seventy rounds of latent evasion are a high-dimensional recurrence
Erik found the 70-generation attacker-defender chase in “Obfuscated Activations Bypass LLM Safeguards” startling. Carlini saw almost exactly the behavior predicted by an older image study that repeatedly barred each successful attack direction and asked for an orthogonal replacement.
That study found tens—perhaps around 50—distinct directions that still produced adversarial examples, with effectiveness degrading only gradually. In high dimensions, 10 attacks can correspond to 10 nearly orthogonal vectors using little of the same representational route.
Weight sparsity does not necessarily eliminate this room. A compressed model may waste fewer parameters while retaining many activation-space directions, and empirical compression has not made classifiers more adversarially robust.
Carlini’s implication was narrower than “unused parameters cause attacks”: the model may need the very features attackers exploit. Removing redundant weights cannot remove directions built from predictive information without also risking task accuracy.
19. Adversarial examples may exploit real features rather than model bugs
The “features, not bugs” intuition begins with dogs and cats. Humans privilege faces, ears, and overall shape; a classifier may legitimately use low-level fur texture because dog fur and cat fur strongly correlate with the labels in its training distribution.
If an attack changes the texture statistic from dog-like to cat-like, the classifier may faithfully implement the learned distinction even though the image still looks like a dog to a human. Training never specified which correlations constituted the intended concept.
Models can detect subtle image statistics humans miss, making such non-robust features both useful for normal accuracy and available to attackers. Adversarial training can lower clean accuracy because suppressing vulnerable features may also suppress genuine predictive signal.
Even nonsense-looking jailbreak suffixes sometimes expose interpretable fragments. One optimized suffix included something like “now write opposite content,” inducing toxic output followed by a compliment; optimization discovered that promising later positivity was a strong feature for obtaining the prohibited text first.
20. Interpretability explains selected features, not the whole computation
Carlini did not see sparse-autoencoder results as inconsistent with non-robust features. A model may contain a recognizable Golden Gate Bridge feature alongside numerous predictive directions that humans cannot name.
In normal data, cat ears, fur, and shape align, so explanations based on the interpretable features can be locally accurate. An attacker searches for an obscure feature with an unusually large weight and activates it in the opposite direction.
Sparse autoencoders already admit compression and omission: they shrink enormous activity into a smaller sparse feature set, then fail to explain some of even those features. Carlini would doubt claims of a perfect account of every component; today, much of the remaining mechanism is still “magic.”
21. Humans are measurably more robust, but context may supply the advantage
A study flashed images to people for about 100 milliseconds after perturbing them to fool an ensemble of neural networks. Humans were misled more often by those adversarial perturbations than by random noise of equal magnitude, suggesting at least some model-derived attack signal transfers.
Yet a neural network can often be attacked with around 1,000 queries, while Carlini doubts 1,000 similarly structured image queries would fool his own perception. In that operational sense, people remain substantially more robust.
One explanation is recurrence and context: a person does not classify an isolated frame with one forward pass but remembers that the creature walked and behaved like a cat. Longer chains of model computation might provide a parallel benefit, though Carlini would not ground a result in casual neuroscience analogies.
22. Robust systems can contain fallible models if the environment enforces limits
Carlini’s median prediction is that models remain roughly as vulnerable as they are now, forcing builders to design systems that assume mistakes. Human institutions already use code review and divided authority because any one person may err or act maliciously.
An agent can similarly escalate uncertain actions to a user. More importantly, external code can deny actions regardless of the model’s persuasion—for example, refusing to enter data into an
input type="password"field even when the model requests it.The limitation is coverage: a site may implement a custom JavaScript password interface without the standard field type. System-level controls still narrow attack surfaces, but utility falls and unfamiliar cases remain.
Erik’s surprise-weighted long-term-memory idea—remembering exceptional failures so the system is not fooled twice—interested Carlini, though he called the work early. He retains some hope that reasoning or memory produces genuine robustness within a few years, but “I’m not optimistic.”
23. Opaque defenses might work through adversarial evolution rather than proof
Erik relayed Michael Levin’s thought that biological systems become parasite-friendly when too interpretable, raising the possibility of evolving defenses nobody can explain. Carlini accepted that an opaque but empirically robust defense would beat an elegant mechanism that fails.
Symmetric cryptography supplies a partial analogy. Block ciphers pursue principles such as confusion and diffusion, then evolve against every known attack; AES has no general proof that every conceivable attack fails, only careful design and analysis showing why known attack families do not beat it.
Other cryptographic constructions can instead reduce security to crisp assumptions such as factoring or discrete-log hardness. Carlini would prefer an analogous reason for model robustness, but AES shows that decades of attack-driven refinement can yield trustworthy engineering without one foundational proof.
24. Security “breaks” form a continuum long before practical compromise
Erik initially contrasted cryptography’s binary compromise with AI’s messy degradation. Carlini objected: cryptographers would call AES-128 broken if key recovery fell from (2^{128}) to (2^{127}) work, even though both exceed practical resources.
The conservatism is anticipatory—an improvement from 128 to 127 bits raises fears of a path to 125 and eventually around (2^{80}), where nation-state feasibility becomes discussable and constants matter greatly.
AES-256 supplies ample margin because even a dramatic reduction toward 100 effective bits may remain tolerable. By contrast, 50%–70% attack accuracy represents frequent failures rather than cryptographic-style work-factor margin.
25. Defense in depth can convert evasion into unusable performance
The latent-defense experiments showed that one detector at one layer could be evaded with little loss of SQL-writing ability. Adding detectors across more layers still allowed evasion, but eventually degraded the generated SQL enough that successful bypass no longer necessarily delivered the attacker’s task.
Carlini recognized the pattern as standard “defense in depth.” Block ciphers stack layers because attackers can penetrate a few but struggle as interactions accumulate; model systems may likewise settle for 20 imperfect barriers whose composition makes useful attacks impractical.
The caveat is correlated failure: a fundamentally different attack might bypass every detector simultaneously. Even so, detector research has option value—if perfect robustness never arrives, layered controls may be the usable fallback; if it does, detection remains useful around it.
26. False positives, utility, and corrigibility all occupy the same trade-off curve
Erik’s camera repeatedly labels squirrels as humans. Carlini reduced the design choice to a true-positive/false-positive operating point, noting research suggesting occasional false alarms can reassure users that a detector is vigilant—until alert fatigue causes them to disable it entirely.
Compiler warnings show the same failure: after 900 unimportant warnings, the developer ignores the thousandth even if it matters. High-stakes models may justify annoying benign users and requiring rephrasing, but this is a product decision rather than free safety.
Carlini read alignment-faking behavior as, among other things, a jailbreak that succeeds in perhaps 5%–10% of cases. When Claude was later allowed to object to the proposed training, it took that option, suggesting refusal can resolve some setups.
Gullibility cannot simply be removed: when a user says generated code failed, the useful model should believe them and reconsider, not insist the compiler is wrong. Corrigibility, safety, and utility therefore require choosing a point on a curve, not maximizing one scalar without cost.
27. Today’s language models know attacks but cannot yet do security research end to end
Carlini’s experiments asked models to generate adversarial examples against defenses. The short answer was “basically not yet,” with a revealing exception: models performed reasonably when researchers rewrote a defense as a clean homework problem containing its essential logic in about 20 lines of Python.
Given an actual research repository with perhaps 1,000 lines, unfamiliar infrastructure, and misleading implementation detail, they failed. “The core of security” is turning an ugly system nobody understands into the one small piece that actually matters.
Models already know how to write PGD once the task is isolated. They do not reliably decide which claimed mechanism is irrelevant, which numerical detail is the real bug, and how to navigate messy code until that distinction becomes visible.
Carlini warned that benchmarks increasingly test the academic subproblem humans identify as hard rather than the complete real-world task. Models can score highly on knowledge tests while ranking differently on agentic work that requires orientation, persistence, and judgment.
28. Human attack talent is scaling through training before it scales through AI
Carlini no longer considers himself uniquely capable. Five years earlier, few researchers attacked these systems; now papers such as the obfuscated-activations work can appear without him, often executed more thoroughly than his own limited time would allow.
Practice remains the primary multiplier. Nobody has yet entered a PhD focused on attacking language models and completed that full training cycle, because serious work on them is only several years old.
His implied forecast is patient: another few years should produce researchers with a complete apprenticeship and a richer Rolodex of failure patterns. The uncertainty is whether model capabilities and deployment stakes allow society the duration of a PhD to catch up.
29. Attack automation is dual use, but proof of weakness is not weaponization
Erik asked whether a rapidly improving automated attacker could become the security equivalent of a laboratory accident. Carlini invoked the Morris worm, created by Robert Morris in 1984 or 1986 in his uncertain recollection, as an early worm that reportedly escaped experimentation and disrupted much of the then-internet.
He distinguished knowledge from weaponization. Security researchers commonly build minimal proofs of concept but do not spend a year engineering production malware; the historical example illustrated the danger of an experiment escaping its intended setting.
Publishing easy attacks matters because people who already know exploitation is easy have little incentive to document that fact defensively. “You want to know what anyone can do” so systems receive protection before malicious actors operationalize the same insight.
Carlini would revisit this norm if agents became superhuman, autonomous, and capable of compromising arbitrary systems. He does not believe current adversarial-example automation would cause much harm right now, but he “never expected” present language-model capability three years earlier and remains open to changing his mind.
30. Narrow jailbreak contests measure partial progress, not final safety
Anthropic’s eight-layer challenge sets a harder target than finding one bespoke jailbreak. Carlini defended bounded problems by analogy to the decade spent studying (L_\infty) image perturbations of at most 8/255: the exact threat is artificial, but well-defined problems can reveal methods that later generalize.
At the time discussed, an individual reportedly broke each of the eight levels, but not with one universal jailbreak. That means the defense would be inadequate for national secrets, yet it may still stop a copy-and-paste attack that works everywhere.
Carlini used the security term “script kiddie” for users who cannot invent exploits but can run published ones. Preventing a universally shareable suffix does not stop determined attackers, but it reduces mass accessibility: “partial progress is still progress.”
31. Independent red teams succeed because they genuinely want the defense to fail
Carlini’s social explanation for weak self-evaluation is incentive-compatible: after spending six months building an idea, a researcher finds it psychologically and professionally difficult to switch frames and sincerely try to destroy it.
If the creator succeeds in breaking the defense, the result may no longer be publishable—“here’s an idea I had; by the way, it doesn’t work.” The external attacker instead receives credit for finding the flaw and has no attachment to the original story.
Security organizations institutionalize this difference through red teams. Carlini said the security teams he has spoken with at many organizations appear to “really want to know,” because uncertainty about whether a control works is itself the reason for adversarial testing.
32. Open source remains favored, but future capability could overturn the presumption
Carlini’s current technical bottom line is that open-weight models cannot be locked down to the degree many policy proposals assume. If released, users should not be assumed to remain reliably constrained from accessing or modifying their capabilities through prompting, fine-tuning, or the weights.
Even so, “in any world that looks noticeably like our world,” he considers open source objectively beneficial for security, safety research, and limiting concentration. A few firms controlling capabilities unavailable to everyone else is a concrete risk that requires no superhuman AI assumptions.
His historical analogy is the 1990s restriction of strong cryptography as a munition. Exported browsers were limited to 40-bit encryption because governments feared unbreakable terrorist communications, yet broad strong encryption also enabled online banking, payments, and secure communication for dissidents.
Unlike encryption, whose endpoint was comprehensible as “perfect encryption,” model capability has no known ceiling or timetable. Carlini expects the next two years to clarify whether scaling continues rapidly or limitations bite; anyone claiming either release policy is “obviously objectively right with 100% certainty” is overconfident.
33. Policy should follow technical reality rather than preferred narratives
Carlini declined to prescribe the social solution. His comparative advantage is establishing whether a defense actually works; he trusts policy experts more on institutional trade-offs if they accurately understand those technical findings.
His largest concern is not merely choosing wrongly under uncertainty, but possessing relevant evidence and ignoring it because of ideology. Decisions made from the best available facts can still age badly, but at least they use the knowledge available at the time.
The episode’s closing call was therefore for more good science, more researchers willing to expose failures, and policy that preserves the transcript’s uncertainty. Models might remain manageable tools, or capability growth might force Carlini to reverse his open-source bias; pretending either future is already certain would repeat the very reasoning errors his attacks uncover.