How Do AI Models Actually Think? [Dr. Laura Ruis]
Summary
Ruis finds evidence that language models learn reusable procedures rather than merely retrieving memorized reasoning traces. Influence functions showed that factual questions depend on distinct documents, while different instances of the same reasoning task share a similar pre-training footprint. Her key distinction is “procedural knowledge”: scale can produce a strategy that transfers across new numbers, even though the paper does not prove formal reasoning.
Code appears to be unusually important training material, but its effect is neither uniformly positive nor understood. Python that describes how to calculate a slope influenced models solving slopes in mathematical text, suggesting models may learn to execute procedures without seeing identical worked examples. For model builders, the data implication is material: synthetically generating diverse procedures could be helpful alongside generating more step-by-step answers.
Ruis rejects a binary choice between retrieval and reasoning because a model can use different functions or programs in different contexts. A changed prompt may elicit a different “function or program,” while unfamiliar tokens can make a capable system fail; one error therefore does not establish that it cannot reason. Approximate competence with “Swiss cheese” holes remains different from dependable formal computation.
Controlled connectionist systems can perform systematic symbolic operations, but Ruis carefully limits the claim for current LLMs. She cites setups reaching “100% accuracy” on novel problems, yet her own work shows only generalizable behavior learned from wild pre-training data. Current models do not appear capable of post-1945-theorem-style inventive creativity, though she sees no theoretical barrier if a system learns the relevant causal data-generating process.
Ruis will not bet against scaling, but sees data diversity, richer environments, and active learning as potentially increasingly important. More data makes reliance on spurious correlations harder, while carefully selected “interesting data” might teach causal mechanisms without trillions of tokens. Her architecture lesson is equally blunt: a Transformer was reported to reach almost 100% on most tests of an older compositional benchmark that defeated an LSTM; “if something can be learned, don’t build it in.”
Agency could emerge from next-token prediction even when developers never explicitly install goals. Ruis defines an agent as something that acts under uncertainty to control its own future inputs, with planning central to both usefulness and danger. She prefers a tool over an equally capable agent, but is unsure whether “an interesting form of intelligence” can be reached without some agency.
The clearest near-term risk is gradual, unequal transfer of power to unreliable systems rather than a single sudden warning event. Ruis does not expect the discussed development “in the next three years”; she worries about skewed access and society slowly giving control to “dumb agents” before recognizing the cumulative effect. Her own shift—from seeing LLMs as less-general retrieval systems to approximate generalizers—underscores how quickly capability assumptions can age.
Deep dive
1. Influence functions expose what a model is borrowing from pre-training
Ruis began with a scale question: when zero-shot reasoning improves, is the model merely seeing more near-duplicates, or learning “something qualitatively different” from additional data and parameters? Traditional train-test separation no longer settles this because “the test data is in the training data now.”
Her chosen tool, influence functions, approximates a counterfactual: remove one pre-training document, retrain the model, and ask how its parameters and completion likelihood would change. The method avoids actually retraining for every document, but Ruis stresses that it remains “very approximate.”
Factual retrieval supplied the sanity check. Questions such as the tallest mountain or largest ocean require relevant atomic information, so influential documents should be recognizable; that grounding made the less-intuitive influence patterns behind self-generated reasoning traces easier to trust.
2. Reasoning tasks share a reusable pre-training footprint
The strongest result, in Ruis’s telling: a document’s influence on one factual answer did not predict its influence on another. Different facts relied on “very distinct sets of documents,” even though syntax and style necessarily drew on broader statistical knowledge.
Reasoning behaved differently. Questions instantiating the same task—calculating slopes with completely different coordinates, for example—had similar influence patterns across training documents. The model therefore appeared to reuse knowledge of the procedure rather than retrieve each step for each particular answer.
The study covered two-step arithmetic such as
(7−4)×8, slopes derived from differences between two-dimensional points, and linear equations requiring three simple arithmetic operations. Because no demonstrations were supplied, a retrieval-only account would require the model to find each generated step in pre-training.
3. Code may teach procedures without supplying worked answers
Ruis spent “days trying to understand” why code was the only material that appeared influential across every reasoning type. It sometimes helped and sometimes hurt, and she found no stable pattern explaining either direction—an unusually candid unresolved result that motivates her current work.
The most revealing specimen was descriptive Python that calculates a slope between two points: it could strongly influence a model asked to perform that procedure in mathematical prose. That suggests a model may translate a procedure across representational forms and then execute it on different values.
If the effect generalizes, Ruis sees a training-data implication: synthetic pipelines could generate “lots of procedures” alongside worked applications, rather than endlessly producing answer traces alone. Multiple expressions of one procedure also provide redundancy and abstraction; unlike a missing fact, the procedure can survive different dataset selections.
4. Fuzzy abstraction is a feature of language, not a defect
The host proposed Hofstadter’s framing that an abstraction is “a bag of analogies”: perhaps a chair or the letter A is recognized through many overlapping perspectives rather than one clean internal definition. Ruis connected that distributed picture directly to how natural language works.
Her Wittgensteinian view is that definitions always encounter contexts where their boundaries fail: “meaning is use,” concepts are fuzzy, and context changes their application. That makes natural language unlike a formal system, but also explains why neural models that approximate systematicity may fit it better than rigid symbolic accounts.
Montague-style strict compositionality captures the genuine fact that wholes depend on parts, yet enforcing it can require “really roundabout functions.” A looser systematicity is visible when someone hears, “I had really good flips last night,” and infers that flips might be food—perhaps dessert—then immediately reuses the novel word.
Ruis likewise rejects a sharp border between facts and reasoning. Someone answering “the Pacific Ocean” might retrieve oceans and sizes, compare them, and only then respond; carefully chosen factual questions still require atomic knowledge, but the cognitive operation is rarely pure retrieval.
5. Reference enriches meaning without exhausting it
Addressing claims that semantics requires physical causal embedding, Ruis granted reference an important developmental role. A child’s “princess” may initially denote a kind woman in a nice dress who can be pointed out directly; adult language progressively becomes more abstract.
Her counterexample was the COVID vaccine: she can discuss and understand it while being unable to identify it among substances or state exactly what it contains. Physical identification would improve her world model, but its absence does not leave the phrase meaningless.
Language models differ because language can describe impossibilities unconstrained by direct survival in the physical world. Their broad representations can therefore be powerful while still producing hallucinations and impossible scenarios; humans imagine impossibilities too, but remain grounded by physical reality and continual interaction.
6. Benchmarks reveal reasoning modes, not a binary faculty
Ruis endorses intelligence tests centered on novelty because pure mimicry cannot keep succeeding when knowledge must be applied in genuinely new situations. Yet she defends “moving the goal post”: chess once looked definitive, then its solution refined what researchers meant by intelligence rather than invalidating evaluation.
The host’s “Swiss cheese” model captured her position: an LLM may retrieve in one context, reason in another, and collapse in a nearby hole. A prompt can activate a different “function or program,” so failure on foreign tokens shows a limitation of a statistical system, not proof that the relevant rule is absent everywhere.
Mathematical reasoning is only one highly formal slice. Induction—whether observing only white swans supports concluding that black swans do not exist—underlies much of science and is harder to evaluate. Ruis thinks models “probably” can perform it, but says their inductions require verification and experiments.
7. Connectionism can be systematic, though inventive novelty remains open
In tightly controlled settings, Ruis says connectionist models have learned systematic rules and achieved “100% accuracy” on novel problems. Such experiments arrange data so lowering loss requires learning the intended rule, unlike wild pre-training where countless shortcuts remain available.
Her paper makes a narrower contribution: it shows generalizable knowledge influencing many instances, not that an LLM has acquired formal symbolic reasoning. She nevertheless thinks approximate formal reasoning from natural data is possible, while preserving limitations such as difficulty with completely novel tokens.
Fodor and Pylyshyn’s systematicity challenge therefore “stood the test of time,” but did not prove symbolic computation impossible in neural networks. Smolensky’s tensor-product representations showed how distributed variables and values could be bound and later unbound, although the original scheme suffered combinatorial growth and restrictive mathematical conditions.
Ruis does not think today’s models can be trained on knowledge only through 1945 and simply invent a later theorem. But she questions whether human invention is an irreducible stroke of genius: Einstein also “stands on the shoulders of other scientists,” recombines inherited knowledge, and reasons for a long time.
8. Scale keeps winning, but richer data and environments may change the recipe
Ruis’s position is deliberately pragmatic: “I am not going to bet against scaling because that seems scary.” It has worked pretty well, though next-token prediction being theoretically sufficient for many tasks does not make it the most data-efficient training regime.
She agrees that the useful frontier is not merely more synthetic tokens but more “interesting data.” As data becomes broader and less semantically repetitive, spurious correlations become harder to exploit and learning the causal world model that generated the observations becomes increasingly advantageous.
A sufficiently diverse selection might teach those mechanisms without trillions of tokens. Interacting with an environment, intervening on it, and generating new evidence could also distinguish active learning from passive imitation—potentially becoming important for future efficiency.
Architecture history reinforces her reluctance to hard-code cognition. An older compositional benchmark on which LSTMs failed was later reported to reach almost 100% on most tests with a Transformer. Her lesson: recurrence looked obviously useful, but “if something can be learned, don’t build it in.”
9. Agency may emerge as control under uncertainty
Ruis would choose a tool over an otherwise identical agent because agency makes intelligence more dangerous. Her uncertainty is whether sophisticated intelligence can exist without some agency, which makes detection more important than simply declaring that developers should remove it.
Text is produced by agents with goals, so a next-token predictor may lower loss by modeling whether a writer is trying to persuade someone. That creates an uncomfortable ambiguity between simulating goal-directed behavior and possessing it.
She highlights a DeepMind definition attributed to Zac Kenton: an agent changes its policy when its actions affect the environment differently. An engineered LLM loop can satisfy that almost trivially; the important question is when equivalent behavior “emerge[s] from something as simple as next token prediction.”
Ruis’s preferred abstraction is a system that acts “in order to control its own future inputs,” especially under uncertainty. Planning is important for achieving complex goals, while uncertainty distinguishes this from reflexes in deterministic environments. A causal world model improves planning, alongside a way to represent possible futures.
10. The main warning is gradual power transfer, not a single alarm bell
Ruis sees agency as something observers naturally detect, recalling the 1940s Heider–Simmel animation in which moving triangles are immediately described as mean, scared, or escaping. That instinct may sometimes misfire, but it reflects why agents matter differently from thermometers: they can be useful or dangerous in different ways.
The “simulator” view treats an LLM as a superposition of many modeled agents, explaining both versatile role-play and failures of coherence. Agency can also exist at collective levels, such as a company or other group, where individual purposes bubble upward and institutional values then push back down.
On safety timelines, Ruis offers no precise forecast and says she does not see “this” happening in the next three years; reliability and adoption remain substantial barriers. She worries more about society slowly ceding control to “dumb agents” and only later recognizing the accumulated consequences, as with social-media externalities.
Her clearest stated social risk is “unequal access”: if AI raises productivity, existing politics and economic structures may distribute the gains poorly. She still sees possible benefits, including making doctors more productive as care demands rise, but argues that policy and institutional preparation—not capability alone—will help determine whether those gains spread.
Ruis’s own update is instructive. She once saw LLMs as less-general retrieval systems and treated zero-shot performance as crucial; now she views them as multitask learners requiring the right interaction. Andrew Lampinen’s analogy changed her mind: zero-shot prompting can resemble shouting “what is 15 times 32?” at a stranger, while few-shot prompting may supply context without licensing test-set overengineering.