Topics Pioneers Insight Method Author
Back

The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering

·The Cognitive Revolution

Rendering the long image…

Topics Model Interpretability and Internal Representations
Pioneers Curt Tigges

Summary

  • Goodfire’s Curt Tigges argues that modern LLM neurons are more interpretable than the field assumed. Better agents and SwiGLU’s gating architecture reveal functions that earlier methods missed; even neurons with multiple meanings often express “a compact set of values.” This supports studying the model’s own components directly, while leaving whole-model understanding unresolved.
  • Heavy activation sparsity preserves substantial capability, with surprisingly little retraining needed to recover reasoning. Keeping 64 of each expert’s 512 neurons removes roughly 88% of activations; 100 million training tokens largely restored benchmark performance, with some results improving by 5–15 points. Tigges suspects a calibration problem: “Something was wrong with confidence.”
  • The interpretation results go beyond assigning plausible labels to activation examples. A separate model predicts activations from paragraph-long descriptions, and inserting those predictions recovers 74% of damage under the study’s weighted KL-divergence measure. Another test asks a model to “surpass this”: generate a stronger activation than anything found across 10 billion tokens, using only the description.
  • Knowledge appears more localized and structurally organized than a tangled-network account suggests. Examined experts generally concern one to three topics, while connections encode relationships among concepts. “A neuron is not a fact”: a community of a few dozen neurons can represent artists and their relationships to impressionist movements.
  • Monitoring is the clearest immediate application, while reliable prediction of unfamiliar behavior remains an open problem. Reward-hacking neurons may detect particular subtypes, and personality circuits offer debugging targets. But Tigges acknowledges that researchers still need higher-level explanatory laws: “we haven’t figured it out yet.”
  • Long context and double-negative activations are material limits on the result. Testing extends to 32,000 tokens, leaving Nathan Labenz’s concern about weak activations accumulating importance across million-token workflows unanswered. Some double-negative activations can repair reasoning even when borrowed from unrelated sequences; “we’re not sure what exactly they do.”
  • Interpretability’s economics favor selective effort rather than maximum agent intelligence for every neuron. The most expensive interpretation run cost $45 for one neuron, but Tigges expects most neurons to need much less work and suggests local GLM-5.3. Sparsity and knowledge localization might reduce monitoring costs without imposing a large architectural “tax,” although neither the total interpretation budget nor inference savings is established.

Deep dive

1. Modern gating makes individual neurons easier to interpret

  • Tigges’s central claim is narrower than declaring neural networks solved: individual MLP neurons are complex but “generally more interpretable than previously thought.” Earlier networks were harder to read. Today, stronger agents can identify patterns across activation examples; in the networks examined, SwiGLU also appears easier to interpret than ReLU or GELU.

  • The principal subject is Qwen3.6-35B-A3B: 35 billion parameters, approximately 3 billion active, and 40 layers. Each layer contains 256 experts with 512 neurons each, with eight experts selected per forward pass. That produces roughly 5 million MLP neurons. Positive and negative activations can carry different meanings, giving approximately 10 million distinguishable poles.

  • Each SwiGLU neuron receives separate up and gate projections, whose outputs are multiplied after the gate passes through SiLU. The two projections can recognize different conditions: cooking context through one path, garlic through the other. Their conjunction creates a strong activation. SiLU permits some negative gate values; ReLU would suppress those. Tigges’s architectural explanation is retrospective: “They just work better.”

  • Gating also helps separate meanings within one neuron. An oral-health signal and a marine-fauna signal can reach different downstream readers because context prepares different circuits in advance. The architecture therefore supports conditional routing rather than indiscriminate mixing. The model also combines ordinary attention in one-quarter of its attention layers with Gated DeltaNet in the remaining three-quarters.

2. The strongest activations carry most of the tested capability

  • Sparsity here means suppressing activations for each token, not permanently deleting the same neurons. Some experiments replace weaker activations with each neuron’s historical mean, usually near zero because activations are bipolar. Others retain only the strongest 64 of an expert’s 512 neurons and zero the rest—roughly 88% suppression. Moving down to 32 retained neurons causes substantially greater degradation.

  • Percentile filtering and top-K filtering answer different questions. Percentiles use each neuron’s own historical activation range; the researchers use 10 billion tokens to collect examples for analysis. Top-K lets each expert retain its strongest signals on that particular input. Coherent meaning may disappear below P95 for one neuron and P99 for another. These thresholds also depend on the dataset, which is far smaller than the model’s presumed training distribution.

  • Removing the middle 50% of activations barely changes the reported results, including reasoning. Removing 80–90% creates clearer losses, although some ARC-AGI results improve. Tigges suspects that fewer distracting signals make the model more confident and faster, helping where excessive deliberation counts against it. Initially, however, difficult reasoning can deteriorate into repeated checking and getting stuck; sparsity alone does not preserve every behavior.

3. Agentic experiments expose functions that static labels miss

  • The “neuroscientist” agent starts with thousands of activation examples, grouped using incoming connections, text similarity, or other neurons’ activity. It proposes a hypothesis, edits or generates sentences, runs controlled comparisons, and observes activation changes. A separate evaluator supplies feedback, allowing repeated revision. The objective is a description that predicts future activation, rather than merely sounding plausible when read beside familiar examples.

  • One neuron looked uninterpretable within a 50-token window: cooking instructions, legal documents, and unrelated prose shared no obvious subject. Agentic investigation found a structural rule instead. It activated on “the second item in the list” when that material had appeared in another format perhaps a thousand tokens earlier—for example, prose or comma-separated content subsequently reproduced as a numbered list.

  • Tigges contrasts this with generic labels such as activation during “medical discussions,” which explain little. More capable agents given more work produce better predictions in the reported comparisons. Labenz’s qualification matters: the experimental workflow could also improve SAE interpretation. Tool use, counterfactual testing, and longer context are methodological advances beyond the separate argument for studying neurons directly.

4. Activation replacement supplies a causal check on descriptions

  • The agent ultimately writes a paragraph describing a neuron’s triggering tokens, concepts, and context. Another model, distinct from the description-producing model, predicts activation on a zero-to-nine scale for every token in a passage. Researchers first ablate the neuron, then insert those predicted activations into the original model and measure how much of the resulting disturbance is repaired.

  • The reported 74% recovery concerns KL divergence between next-token prediction distributions. It measures restoration of the output distribution damaged by removing the neuron, aggregated across token positions. Importantly, results are weighted by the original damage: cases where the neuron mattered strongly count more than cases where its removal barely changed anything. This is not a claim that 74% of the entire model has been explained.

  • Another evaluation supplies only the description and asks a model to generate text activating the neuron more strongly than any example in the 10-billion-token corpus: “surpass this.” A simpler test asks whether a neuron should activate anywhere in a sequence, often reaching around 90%. Tigges distinguishes that easier relevance judgment from accurately predicting activation strength token by token.

5. Monosemanticity depends on how a concept is drawn

  • An insect-related neuron might respond specifically to preferred foods, habitats, and egg-laying locations, without responding to every discussion of insects or every insect preference. “So, are these three concepts or one concept?” Tigges asks. After examining thousands of descriptions, he finds the monosemantic-versus-polysemantic distinction less useful than asking whether the neuron’s actual range of functions can be explained.

  • His strongest illustration is a neuron responding to clothes, exercise equipment, dishes, and tablecloths. Those categories look unrelated until the organizing concept becomes things found at a garage sale. Apparent polysemanticity can therefore reflect an observer missing the model’s grouping. But Tigges does not insist that every collection hides one elegant concept; genuinely separate meanings can simply be listed.

  • The mechanistic evidence matters more than the naming dispute. Different subsets of incoming neurons activate for different meanings, allowing researchers to identify distinct subcircuits in the connections themselves. A seemingly unified concept may split under closer inspection; visibly unrelated functions may remain separate. The practical question becomes completeness of explanation, rather than forcing every neuron into one semantic category.

6. Neurons perform operations while connections encode relationships

  • The move from nouns to verbs changes what researchers look for. A neuron is not simply an entity detector; its signal makes the model do something downstream. Labenz calls this “flow control,” and Tigges agrees. Reasoning-related neurons can promote confidence, waiting, checking, continuing an approach, changing direction, or flagging a wrong number or unit.

  • Labenz’s factual-memory puzzle is that a model with 5 million neurons can know far more than 5 million facts. Tigges’s answer: “A neuron is not a fact,” because relationships reside in connections and paths among neurons. Ten connected neurons can express many more than ten facts. He does not claim every fact has been located, or that facts exist exclusively in MLPs.

  • One observed community contains only a few dozen neurons connecting individual artists with impressionism, neo-impressionism, and post-impressionism. Artist representations link to movement tokens, creating a compact mechanism for classification. This is a concrete step beyond isolated neuron labels: the relevant information resides in how components connect, rather than requiring one standalone neuron for every artist–movement fact.

  • Examined experts also appear more focused than their opaque reputation suggests, generally covering “one to three” topics rather than hundreds. Examples include mathematical notation, algebra, geometry, constitutional and jurisprudential text, elapsed time and time units, and medical subtopics. Labenz argues that meaningful specialization fits the router’s task of choosing eight experts from 256; Tigges agrees that expert-level understanding deserves updating.

7. A catalogue of circuits still does not predict unfamiliar behavior

  • Labenz presses the central objection: even billions of circuit descriptions could remain an inventory without explaining what a model will do in a new situation. Tigges concedes that the higher-level primitives remain undiscovered. Physics, chemistry, and biology provide his analogy: lower-level understanding supports higher-level science, but describing everything through fundamental components is too detailed to be practically useful.

  • Tigges proposes combining broad circuit discovery with targeted investigations. Millions of queries and attribution graphs can reveal recurring neuron communities, while datasets designed around reward hacking, scheming, or planning can identify behaviorally relevant circuits. Researchers must then test how those communities influence behavior. Understanding every individual neuron would still fall short of ambitious reverse engineering of the model.

  • Work must proceed from both directions: neurons and subcircuits upward, steering vectors and training interventions downward. Personality provides a debugging example. A steering vector may usually work, yet a rare input could activate circuitry that disrupts it—perhaps in one case out of 10,000. Neuron analysis might locate the failure, although Tigges does not guarantee that it will.

8. Direct neuron analysis simplifies MLP work without resolving the whole model

  • Monitoring is the most immediate application discussed. A researcher inspired by the earlier essay found a neuron activated by reward-hacking text, potentially useful “at least for this subtype.” Labenz explicitly warns against watching the first useful neuron and assuming all reward hacking is covered. Personality steering offers another practical avenue, but neither example establishes comprehensive behavioral control.

  • Labenz argues that analyzing neurons directly removes an additional reconstruction layer when debugging MLP behavior. Tigges agrees conditionally: transcoders often recover neurons or sparse combinations of coactivating neurons, and replacing an MLP with a transcoder loses more fidelity than retaining top-K neurons. Such learned representations may still reveal useful combinations; his claim concerns which basis most directly represents the underlying computation.

  • Attention and the residual stream remain major unfinished work. Tigges emphasizes residual-stream superposition and complicated geometry, where a neuron-based account is not yet available and SAEs may still help. He also declines to assume SAEs are the final answer there. Better residual-stream understanding is, in his view, one of the most important requirements for ambitious reverse engineering.

9. Small retraining restores reasoning, but long-context evidence has limits

  • Asked about the surprisingly small token count, Tigges clarifies that it was 100 million—small in this context—and says the run took about one day on multiple GPUs. They stopped when performance had mostly reached parity rather than pursuing a longer campaign. Tigges says it would be interesting to test whether more than 100 reinforcement-learning steps could improve the model further.

  • His interpretation is that sparsification disturbed calibration more than it erased reasoning mechanisms. Before retraining, allowing “pass@4” or “pass@8” attempts could already match original-model performance. The sparse model retained useful knowledge but struggled to deploy it consistently. Repeated checking and disorientation support that account, although the precise mechanism behind the confidence disturbance remains unknown.

  • Reported results span mathematics, programming, general knowledge, MedQA, GPQA Diamond, and SuperGPQA. Some benchmarks improved by 5–15 points; MedQA recovered to within one point of the original. Tigges also found the model comparably useful in his own interactions, while acknowledging that sustained use would be needed to assess practical equivalence. Further coding evaluation was still underway.

  • Long-context testing reaches 32,000 tokens, including programming. Reasoning and coding require more active neurons than storytelling—by roughly an order of magnitude. Labenz worries that weak activations could preserve conversational texture across million-token windows and repeated compression. Tigges says million-token behavior needs further research; his expectation that the 32,000-token result probably extends further is an expectation, not a demonstrated result.

10. Double-negative activations remain a mechanistic puzzle

  • In SwiGLU’s double-negative case, a negative gate output multiplied by a negative up projection produces a positive activation. These cases are less interpretable. Removing them sometimes harms reasoning or recall, yet reversing the order of the tokens or substituting activations from an entirely different sequence can restore reasoning. Tigges’s conclusion stays appropriately unfinished: “we’re not sure what exactly they do.”

  • Labenz asks whether simpler activations might justify an interpretability tax. Tigges is unconvinced that simplification is necessary, because SwiGLU’s context switching can itself improve interpretability. Gemma 4 models appear to use ReGLU, preventing this particular double-negative pathway, but he has not analyzed their neurons in detail. His inspection of Kimi K3 found almost no such activations; whether architecture or learned behavior explains that remains uncertain.

11. Interpretability’s scaling path combines selective compute with targeted design

  • Labenz reframes architectural efficiency around total safety costs. Citing Jensen Huang’s discussion of spending only 10–20% of resources creating a product and 80–90% making it safe and reliable, he asks whether slower training or inference might be worthwhile if monitoring becomes cheaper. He also raises hardware that skips multiplication by zero as a possible route to savings; the conversation establishes no realized speedup.

  • Tigges’s counterweight is that existing models may already be interpretable enough to avoid a large architectural tax. That is “quite likely, although not guaranteed.” Encouraging sparse activations during or after training could reduce the number of signals requiring monitoring and make circuits clearer. The sparse Qwen experiment motivates this direction without establishing the optimal training recipe.

  • Localization offers another possibility: encourage specific knowledge to occupy particular parts of the model, then remove those parts when appropriate. The discussion includes gradient routing and concentrating potentially harmful biological or weapons-related knowledge into separable regions. Neuron understanding and expert specialization make targeted organization conceivable; the claim remains a design opportunity rather than a demonstrated comprehensive removal method.

  • Costs vary substantially. Tigges’s most expensive neuroscientist run cost $45 for one neuron, while most were much cheaper and cost optimization was not a priority. He suggests a local model such as GLM-5.3, an inexpensive initial pass, and additional intelligence only for difficult neurons. Full-model interpretation therefore need not mean running the most elaborate agent on every neuron, but he supplies no aggregate budget.