Language Models are "Modelling The World" [Nicholas Carlini]
Summary
Carlini’s base case is not perfectly robust models, but secure systems designed around models that remain “very vulnerable” for the foreseeable future. Conventional security already tolerates imperfection, but ML has not reached the point where “a random person off the street” cannot run off-the-shelf code and force arbitrary failures. The practical implication is to design the surrounding system so it does not rely on the model alone.
GPT-3.5 Turbo Instruct’s chess play convinced Carlini that next-token prediction can reconstruct surprisingly rich internal state without explicit chess rules or chess-specific self-play reinforcement learning. From move notation alone, it almost always produces legal moves and often strong ones, implying “something internally that is accurately modeling the world.” Yet bizarre move histories elicit bizarre play: the base objective imitates its data rather than playing to win.
Arguments over whether models “reason” obscure the more decision-useful question: what behavior survives which conditions? Definitions split predictably between skeptics who exclude language models and AGI advocates forecasting displacement of human jobs by 2027. Carlini instead measures input-output behavior—and notes that the chess competence collapses under unfamiliar notation, just as distribution shift has always broken ML systems.
Post-training makes latent capabilities easier to elicit, but it can also damage properties investors may mistake for competence. Carlini recalled a GPT-4 report in which the base model was well calibrated—a 30%-probability answer was right about 30% of the time—before post-training “messed up” that calibration. Instruction following improves dramatically, but the model may look smarter without learning much new.
Carlini reports an approximately 50% programming-productivity improvement for himself, presenting it as an N-of-1 result. For an expert programmer, current models can act as a higher-level interface over code, APIs, and unfamiliar syntax. He asks when a task has perhaps a 50% chance of working, checks the direction, and completes the residual error himself: “95% solutions are still 95% solutions,” but fixing the final 5% can yield a “20x performance increase.” o1 even solved a dynamic-programming problem in two minutes with code 10 times faster than his own.
That productivity gain may compound security debt because models reproduce familiar vulnerabilities while expanding who can ship software. Carlini saw generated encryption code that should have been an API call and database code vulnerable to SQL injection. His concern is a future with “10 times as much code” but not 10 times as many security experts—the common case works, so insecure deployments proceed.
Disclosure practices remain an unresolved operating risk because some ML flaws are patchable software bugs while others resemble enduring scientific vulnerabilities. A model-stealing API weakness fit a 90-day coordinated-disclosure process; adversarial examples do not, because another 90 days cannot solve a problem researchers have attacked for 10 years. Carlini expects clearer norms within “a year or two,” but offers no settled framework today.
Model APIs do not fully protect proprietary architecture, and benchmark leadership does not reliably demonstrate general capability. Carlini’s team recovered a real production model’s final layer and undisclosed width using API outputs and linear algebra, though not the whole model; meanwhile, trainers are incentivized to optimize HumanEval, MMLU, and similar scorecards directly. “A lot more benchmarks” would make such targeted optimization harder and capability claims more credible.
Deep dive
1. Secure systems must assume the model will fail
Carlini rejects perfect security as the target: even conventional systems lose against a sufficiently motivated government. The practical threshold is whether an ordinary attacker succeeds. Today, he says, “the average person can succeed almost always” against ML systems, often using off-the-shelf GitHub code to induce arbitrary behavior.
Vision provides his sobering prior. Researchers have pursued robustness for roughly 10 years and learned plenty, but objective metrics have barely risen during the last four or five. That makes perfect robustness unlikely in the foreseeable future, although continued research might eventually raise the cost of attack.
His deliberately hedged forecast—“the probability of this happening is very small,” yet also his median prediction “in some sense”—is that simple attacks persist. Systems must contain the damage when a model randomly misclassifies an input, preventing one unreliable judgment from triggering “a terribly misguided action.”
Carlini stresses that “five years is an eternity” in machine learning. Language models have seen about seven years of significant progress and only two or three years of serious attack research; five more years could change everything. His forecast applies chiefly “as long as we’re in the current paradigm.”
2. Chess reveals state tracking without explicit world rules
GPT-3.5 Turbo Instruct changed Carlini’s view because it learned chess as next-token prediction over move sequences. Unlike a purpose-built chess engine, it was not explicitly given an 8-by-8 board, piece movements, the win condition, or chess-specific reinforcement learning through self-play—yet it generates legal and often high-quality moves.
Legality alone is meaningful: determining a valid chess move is a complicated program, and random characters will not reliably stumble onto one. Carlini avoids claims about consciousness or intentionality, but concludes the model “must in some sense have something internally that is accurately modeling the world.”
The sharper comparison is personal. Given 40 moves as notation and asked for move 41, Carlini says he could not reconstruct the board mentally despite being a modest chess player. The model somehow tracks the position and acts on it, “having never been told anything about the rules” or even that such rules exist.
That result moved him away from a purely surface-level account of model behavior. He still prefers examining inputs and outputs over speculating about hidden cognition, but chess showed that the statistics captured by next-token prediction can encode a substantially deeper representation than he had expected.
3. The base objective imitates play rather than pursuing victory
Carlini reaches the same board through an absurd synthetic history—moving each piece circuitously into place—then asks for the next move. The model still usually produces a legal move, showing it retained the board state, but chooses something bizarre because the preceding game resembled “two people rated around 1500.”
The causal chain matters: the model was never told “the game of chess is to win.” It learned to continue data resembling human games, including popular blunders and traps. Faced with low-quality play, it infers that the task is to “make valid moves and see what happens.”
Supervised fine-tuning, RLHF, instruction fine-tuning, and related post-training reduce this mismatch for chat products. Carlini credits the Instruct sequence with taking capabilities “somewhere behind the scenes” and making them easier to reproduce; GPT-3 was exceptionally difficult to use by comparison. Post-training lets the model answer sensibly even when the user frames a question poorly.
4. Post-training helps elicitation while distorting calibration
Human preference data may reward confidence, verbosity, and complexity rather than truth, as the host argues. Carlini offers no comprehensive answer because researchers rarely receive equivalent pre- and post-trained models, making it difficult to isolate how competence changes rather than merely how accessible it becomes.
He recalls a GPT-4 report showing strong base-model calibration: if a math answer carried about 30% token probability, it was correct around 30% of the time. After post-training, “the calibration gets all messed up,” even as instruction following improves enough that the system appears fantastically smarter.
Carlini’s honest non-answer is that these effects remain “a very confusing thing.” Post-training may teach little new knowledge while radically improving usability, but it also changes behavioral properties that matter for reliance. Capability, confidence, calibration, and compliance should therefore not be treated as one variable.
5. “Reasoning” is less useful than tested behavior
Carlini sees the reasoning debate as definition-driven. Skeptics define reasoning to exclude language models; advocates who expect them to solve everything and displace human jobs by 2027 define reasoning as whatever process the model performs. Each side then obtains its preferred answer from its premise.
The host pushes for knowledge plus inference, autonomy, emergence, and intentionality. Carlini, disclaiming philosophical expertise, returns to the operational test: did the model solve the challenging task? If it always returns the right answer, he does not especially care whether it reasoned or used “a lookup table.”
Chess also supplies the counterexample. Present games in an older notation and the model emits text that resembles moves but no longer tracks the board; a human would struggle with the notation yet could reconstruct it. Whether that disqualifies “reasoning” depends on the chosen definition, not the observed failure.
Carlini compares the criticism with training on ImageNet and testing on CIFAR-10: seven years ago, researchers would have said distribution shift obviously explains failure. Expectations have since inflated because language models are marketed as general-purpose, but they remain machine-learning systems that behave better when trained on data resembling deployment.
6. Carlini optimizes for useful attacks, not moral heroism
His motivation is candid: “I really enjoy breaking things.” He did not choose security by calculating the career that maximizes lives saved; he chose work he can sustain and perform unusually well, subject to the constraint that it moves the world in a net-positive direction.
The decision rule is “how good you are at the thing multiplied by how much the thing matters.” Exceptional contribution to a directionally beneficial field can outweigh negligible contribution to the theoretically highest-impact field. For Carlini, attacking systems maximizes that product better than alternatives he might find impossible to motivate himself to pursue.
Attackers also enjoy structural leverage: they need one overlooked corner case, while defenders must fix every path. A condition that random execution almost never hits—such as a hash equaling (2^{32}), causing an integer overflow—can be forced every time by an adversary, reducing an otherwise sound product’s effective security to zero.
ML worsens the asymmetry because researchers continue discovering entirely new attack categories. Carlini cannot recall withholding a completed vulnerability for ethical reasons, largely because he directs idea generation toward beneficial problems from the outset; he nevertheless refuses to rule out encountering something too harmful to publish later.
7. ML needs disclosure norms beyond software security
Conventional security converged from “full disclosure” versus silence toward coordinated disclosure: notify the affected party, allow a defined repair window, then publish. Google Project Zero’s example is 90 days, shortened to seven days when exploitation is active; Spectre and Meltdown received roughly a year because mitigation was unusually difficult.
Some ML findings fit that template. Carlini’s team disclosed an API-based model-stealing attack because providers could change the interface and stop it. The deadline gives users protection time while preventing a company whose business is shipping products—not fixing invisible bugs—from postponing remediation indefinitely.
Other findings cannot be patched in 90 days. An adversarial example exposes a problem researchers have pursued for 10 years, while biological-systems researchers analogize the issue to a novel pathogen, where no brief embargo can undo the harm. Courtesy notice and filters may block one specimen without solving the class, leaving Carlini unsure whether software security is even the right model.
He expects the field to establish processes within “a year or two,” but today it is still borrowing analogies. His policy boundary is equally restrained: lawmakers may decide whether hacking ML systems is illegal, provided they reason from what attacks actually can do rather than “what we think the world should look like.”
8. Security research trades speed for exhaustive detail
Carlini contrasts systems papers—typically 13 or 14 two-column pages—with machine-learning submissions of perhaps seven or eight single-column pages. Systems work demands extensive explanation and rigorous evaluation; ML often accepts “here is a new cool idea, maybe it works,” enabling far faster experimentation.
The throughput difference is material: he estimates one, one-and-a-half, or two systems-security papers per year versus perhaps five or six ML papers at comparable effort. That looseness has benefited a young field, where an original paper and two rapid improvements can all appear at the same conference after circulating on arXiv.
Mature fields need stronger defenses against repeatedly proposed ideas and weak evidence, so Carlini values a mixture of cultures. His systems training supplies a differential advantage in ML security: one recent defense failed because of a negative sign in the wrong place, the kind of detail that he could not have reasoned about from first principles by inspecting the code alone.
9. LLMs raise expert coding productivity by changing abstraction level
The host cites Carlini’s report that language models improved his programming productivity by about 50%, which Carlini presents as an N-of-1 anecdote. He rejects broad hype but says current models are genuinely useful for people who understand the task, can code, and can inspect the generated output.
His analogy is moving from assembly or C to Python. Nearly any result could still be implemented at the lower level, but pointers, allocation, syntax, and boilerplate distract from the actual research problem. A model accepts a loosely expressed intent, writes the low-level details, and lets him return to the reasoning he cares about.
Models are also useful as compressed knowledge of APIs and unfamiliar languages. Carlini can inspect code in a language whose syntax he could not produce, then fix errors around the edges. Compared with copying an uncertain Stack Overflow fragment, asking for code tailored to the task is, in his view, strictly better.
His workflow begins when an answer has perhaps a 50% chance of working: ask once, inspect whether the direction is promising, iterate selectively, and remember which problem classes fail. The required skill resembles learning search-engine queries—natural language permits anything, but some formulations and tasks are far more productive than others.
10. Expert verification does not prevent security risk from scaling
Models sometimes reset Carlini’s own boundary of what is feasible. A dynamic-programming problem originally took him two or three hours; shortly after o1 launched, it produced a correct implementation in roughly two minutes that tested 10 times faster. He updates from observed behavior rather than maintaining a fixed forecast about scaling.
Verification remains ordinary code review, not a special ritual: “half of the answers on Stack Overflow are wrong anyway.” A generated solution that reaches 95% should be treated as exactly that; an informed programmer completes the final 5%, potentially converting the partial result into a “20x performance increase.”
The failure mode is users who cannot inspect the result. Asked for encryption, a model may comply when it should recommend a vetted API; Carlini also received database code using string concatenation and vulnerable to SQL injection. He could substitute parameterized templates, but a novice might ship the defect.
The quantity effect worries him most. Models could enable “10 times as much code,” which is valuable, while producing 10 times as much insecure software without 10 times as many experts to audit it. Like Rust’s borrow checker, effective use requires adapting one’s habits to the tool’s peculiar constraints rather than treating it as a human programmer.
11. Benchmarks and APIs both leak more than their owners intend
Old benchmarks measured one narrow task, though that task—Carlini’s example is classifying “200 different breeds of dogs” in ImageNet—was rarely the real application. General-purpose models create a harder claim: high scores on a small suite cannot establish broad competence when developers know and optimize against that suite.
Direct test-set leakage is not required. A trainer can tune specifically for HumanEval, MMLU, and roughly 20 familiar scorecards, producing a model that looks excellent while failing elsewhere. Carlini wants “a lot more benchmarks” so targeted optimization around the full evaluation set becomes harder; for quantization and related changes, his answer is simply, “good luck.”
His model-stealing work traces the same gap between evaluation and reality. A 2020 paper submitted to CRYPTO recovered exact models only under contrived assumptions—ReLU activations, float64 inputs and outputs, fully connected depth no greater than three, and at most 32 units per layer. The production study instead asked what a real API exposed now, even if it yielded only one small layer.
The attack exploited a last-layer map from an illustrative internal width around 2,048 into a roughly 50,000-token output space. Outputs occupy a lower-dimensional subspace; singular value decomposition recovers that embedding, and “the number of non-zero singular values tells you the size of the model.” It is undergraduate linear algebra combined with model architecture, API behavior, and an attacker’s perspective.
The team recovered the final layer and width of production models, including the first public confirmation, Carlini says, of GPT-3 Ada and Babbage widths. They also recovered GPT-3.5’s corresponding data; OpenAI confirmed it, but the agreement barred publication of the stolen values. Carlini therefore knows the size while carefully declining to disclose it.
OpenAI authorized the test, Google permitted it after legal coordination, and the work ran on an isolated VM whose recovered data was destroyed. The issue was handled through 90-day disclosure to OpenAI; Google and a couple of unnamed providers patched it. The vulnerability had existed for about three years: “you don’t stumble upon these kinds of vulnerabilities”—someone with broad, cross-domain knowledge must deliberately look.