Untangling Neural Network Mechanisms: Goodfire's Lee Sharkey on Parameter-based Interpretability
Summary
Goodfire’s core bet is that interpretability must identify the computations a neural network uses, not merely the concepts visible in its intermediate activations. Lee Sharkey argues that parameters, architecture, and nonlinearities implement the learned algorithm, while activations are partly “along for the ride.” If parameter decomposition scales, it could provide a more actionable map for monitoring, capability removal, and extracting model-discovered knowledge.
Sparse autoencoders can reveal represented concepts while missing the reusable functions that transform them. Days of the week may occupy a plane where one rotation implements “next day,” rather than requiring seven independent transformations; similarly, modular addition can be encoded through angles and trigonometric operations. A feature-centric system might spend enormous representational capacity describing every input-output pair when the model is applying one simple operation.
Attribution-based parameter decomposition, or APD, treats a dense neural network as if many specialized experts had been “smushed together” and then tries to unmerge them. Its loss balances faithfulness—the components sum to the original parameters—minimality—few components are needed per input—and simplicity—each component should be low-rank and span few layers. The approach worked on small problems, but its full-model parameter copies made it computationally and memory intensive.
APD’s deeper weakness was that its optimization target could misidentify the mechanisms that matter. Top-k selection creates discontinuous switches where a tiny parameter change activates an entirely different component, while gradient attribution measures local sensitivity rather than causal necessity. A saturated attention mechanism can have a near-zero gradient precisely because it is attending so strongly, despite being mechanistically indispensable.
Stochastic parameter decomposition, or SPD, replaces full-network components with rank-one subcomponents that each read one activation direction and write another. These primitives are far cheaper and can later be regrouped into higher-rank or cross-layer mechanisms; rotating a two-dimensional day-of-week representation, for example, would likely require two such components. Lee expects computation to be in superposition too, so a matrix may need more rank-one subcomponents than it has activation dimensions.
SPD learns causal importance by randomly masking every subcomponent and training a small function to predict how safely each one can be weakened. An importance value of 1 keeps a component fully on; a value of 0 lets its strength vary anywhere from zero to one, with reconstruction errors teaching the predictor when that freedom was misplaced. Because components remain stochastically active and receive gradients, SPD avoids APD’s silent components, top-k discontinuities, and much of the proposed “junk sink” pathology.
The method has not produced a uniquely correct dictionary of neural mechanisms, and Lee explicitly rejects that framing. Researchers still must determine how rank-one pieces group into semantically meaningful computations, where to stop decomposing, and whether causal importances become pathologically fractional. “We’re all trying to discretise a fundamentally continuous object,” so reconstruction quality, near-binary importance values, and known-ground-truth toy tasks are evidence—not proof of a canonical decomposition.
The commercial and safety upside is substantial, but scaling evidence remains decisive. Goodfire’s $50 million raise could give Lee’s team more private-sector compute resources than Apollo’s philanthropic model, and experiments have begun from million-parameter language models into the “low single-digit billions.” Potential payoffs include parameter-level unlearning, function-based monitoring of deception across diverse contexts, and interpreting scientific models in “the terms that the network itself is thinking in”; whether any beats simpler gradient-guided or activation-based tools remains open.
Deep dive
1. Goodfire gives parameter interpretability a better-funded home
Lee’s Apollo team pursued mechanistic interpretability as a route to detecting deception without relying solely on model outputs—trying to “read the thoughts, so to speak” of frontier systems. Apollo ultimately decided to double down on evaluations, leaving its mechanistic team without the same organizational fit.
The move was not a repudiation: Lee said he remained supportive and somewhat involved with Apollo. He had also connected future Goodfire co-founders Tom McGrath and Eric Ho roughly six months before the transition, giving him what he called a “B character role” in Goodfire’s founding story.
Three people made the move from Apollo to Goodfire: Lee, Dan Braun, and Lucius Bushnaq.
Nathan’s investor-relevant observation was that Goodfire’s $50 million raise could support both talent and the heavy compute these methods require, unlike Apollo’s philanthropic funding model. Lee’s response was categorical: “Yep, 100%.”
2. Activations expose variables, while parameters implement the algorithm
Traditional activation decomposition collects intermediate states and separates them into pieces associated with concepts. Parameter decomposition shares the philosophical goal but targets the weights, architecture, and nonlinearities that repeatedly transform inputs—what Lee called “the thing that we really want to understand.”
A network may use different subsets of its machinery for a cat image and an Eiffel Tower image. Parameter decomposition seeks those specialized modules: mechanisms performing particular computational jobs, rather than only representations such as fur, whiskers, eyes, or the eventual cat concept.
Lee’s clean distinction was between “identifying the variables used in computations” and identifying the computations themselves. Activation tools take a serious bite out of the black box, but every transformation between their labeled intermediate results remains another, smaller black box.
3. Distinguishability inside an activation does not prove model use
Lee’s cat example sharpens the philosophical problem: researchers might distinguish cats standing in several positions within activation space, yet the network’s downstream machinery may treat them all identically. The fine-grained distinction exists in the data without necessarily being a variable in the learned algorithm.
His proposed criterion is computational: a variable is fundamental to the model insofar as the model performs distinct computations over it. That means “putting computations first and representations second,” instead of treating every recoverable direction or cluster as something the network itself meaningfully uses.
Nathan’s synthesis was that understanding why models behave as they do requires both the intermediate results and the functions connecting them. The stronger intervention objective—predicting or changing behavior—makes those transformations especially difficult to ignore.
4. Multidimensional features point toward reusable operations
Sparse-coding work beginning around late 2022 and accelerating in 2023 revealed representations spread across layers and structures richer than single activation directions. Lee’s hypothesis is that a plane or higher-dimensional feature may exist because it is the input to a correspondingly multidimensional computation.
Days of the week provide the signature specimen: rather than seven unrelated vectors, the days can occupy seven roughly equal directions around a plane. One rotation can then implement a general “next day” function instead of requiring a separate learned mapping for every weekday.
Nathan connected this to grokking results for modular addition, where inputs are translated into angles, combined trigonometrically, and mapped back to the answer. An SAE looking for independent sparse directions might recover fragments without making the cyclic algorithm legible.
Lee preserved an important complication: both representations can coexist. A sentence-completion task may need an individual Monday direction, while “the day following Tuesday” benefits from the shared rotation; parameter decomposition should expose algorithmic steps that activate only where useful, not impose one universal basis.
5. Feature-centric tools can lavish capacity on a simple function
Lee’s identity-transformation thought experiment shows the mismatch. If a million input features pass unchanged into a million output features, an SAE or transcoder may need a million input-output correspondences even though the actual computation is one compact instruction: preserve the representation.
The same inefficiency applies to a rotation. A transcoder predicts the next layer’s activations and therefore may encode each transformed feature separately; parameter decomposition instead aims to recover the single object performing the shared transformation across many possible inputs.
Lee did not claim parameter space is the only route. Activation-based methods can also target computations, including multidimensional or cross-layer approaches; his narrower claim is that parameter decomposition makes reusable functions, rather than only sparsely occurring directions, the primary object.
6. Dense networks may be mixtures of experts in disguise
Nathan extended activation superposition into computational superposition: networks may contain more computations than neurons, spreading each across many units. Lee agreed that this distribution can suppress interference, allowing overlapping mechanisms to share machinery when their triggering inputs rarely coincide.
The most intuitive analogy is a mixture-of-experts model whose experts were merged into one weight set. Parameter decomposition attempts “model unmerging”—splitting a network that does many things into smaller networks that each do one thing, or “as small a number of things as possible.”
Nathan visualized this as adding height to architectural width and depth: vertical slices contain simple subnetworks, only a sparse subset of which participates in a given input. Lee accepted the picture while grounding it in a third question: which computation occupies a given layer and neuron dimension?
7. APD balances faithfulness, minimality, and simplicity
APD, the earlier approach published under the Apollo banner, creates parameter components shaped like the target model and trains their sum to reproduce its parameters. This faithfulness constraint tethers the enlarged decomposition to the original network even though the components are randomly initialized.
Minimality asks the decomposed model to reproduce the original behavior using as few components as possible on each input. It is the computation-space analogue of sparse activation: a cat prompt might require one subset of mechanisms, while an unrelated prompt uses another.
Simplicity prevents the trivial answer of assigning the entire original model to one component. APD penalizes components for high-rank matrices and for spanning many layers, pushing each toward “as little computational machinery as possible.”
Nathan’s scaling thought experiment exposed the cost: splitting a billion-parameter model into a million full-size components would imply a quadrillion parameter slots. The published work remained closer to the “toy models of superposition” stage, where controlled algorithms make recovery measurable.
8. High-dimensional optimization can satisfy surprising constraints
Lee assumes ordinary training already learns sparse computation, though not in the neuron basis humans can directly inspect. Parameter decomposition therefore searches for the basis in which those computations become sparse, much as other work makes a trained network’s algorithm visually simpler through sparsity constraints.
Why should one optimizer satisfy faithfulness, minimality, and simplicity together? Lee’s intuition is that a larger model has more ways to get an objective right because optimization searches “many different directions in parameter space” simultaneously—counterintuitive to low-dimensional human reasoning.
Channeling Ilya Sutskever, Lee said, “The networks, they just want to learn.” He immediately qualified the wonder: APD was “extremely janky,” difficult to tune, and evidence that feasible joint optimization does not mean the chosen algorithm is scalable or pleasant to use.
9. Faithful parameter sums need not guarantee faithful-looking mechanisms
Nathan challenged APD’s faithfulness criterion: one active component might learn a new computation while a rarely selected “junk sink” absorbs the offset required to keep every component summing to the original weights. The resulting behavior could reconstruct correctly without each recovered mechanism resembling the original model’s causal organization.
Lee distinguished this from genuine neural-network degeneracy. Scaling a weight before a ReLU and inversely scaling the next weight can preserve the same algorithm, while an overparameterized MNIST network may contain parameter directions orthogonal to every activation it encounters and therefore safely ablatable.
Language models may be underparameterized enough to contain less true junk, but Lee conceded Nathan’s APD concern was “fairly reasonable.” He did not know whether the pathology occurred or whether other losses implicitly discouraged it; SPD would provide the clearer structural answer.
10. Top-k selection and gradient attribution made APD unstable
APD selected the top-k attributed components per input, making optimization discontinuous. A minute parameter update could replace one selected component with another and abruptly implement a different function, an unfriendly landscape for gradient descent.
The value of k and the interactions among numerous loss weights became highly sensitive hyperparameters. Lee’s complaint was not merely that a valid range had to be found; changing a setting could move training in an unpredictable direction because attribution biases altered the apparent optimum.
Gradient attribution also confuses local sensitivity with causal importance. A component near a local performance maximum can have a near-zero gradient and be misclassified as unimportant. If attention is essentially 1 on one token and 0 elsewhere, the responsible mechanism is crucial, yet saturation means a small weight perturbation barely changes attention and produces an almost-zero gradient.
Those conceptual problems compounded APD’s brute-force burden: every component resembled a full model, demanding large parameter copies and memory. The follow-up needed cheaper primitives, continuous learning signals, and a measure closer to “what the overall network can’t do without.”
11. SPD reduces mechanisms to rank-one read-write operations
SPD, the newer Goodfire approach, replaces each full-size parameter component with randomly initialized rank-one subcomponents localized to individual matrices and layers. Higher-rank or cross-layer mechanisms are meant to be recovered later by grouping these atoms, preserving APD’s target while avoiding its “frankly ridiculous” component size.
A rank-one matrix reads only the activation direction defined by its right singular vector and writes only the direction defined by its left singular vector. It matters when activations overlap its read direction; Lee also noted that a direction with no downstream relevance can make such a component do nothing.
These atoms can express a simple rotation and scaling from one direction to another. Rotating a day-of-week variable within a plane likely requires two rank-one pieces—one for each dimension—making the unresolved grouping step essential rather than cosmetic.
SPD can use more rank-one subcomponents than the original matrix has dimensions, just as representations can outnumber neurons in superposition. Lee nevertheless expects potentially fewer subcomponents than SAE latents because all subcomponents must sum to the original parameters, creating “a kind of upper bound”; he stressed that this remains empirical.
12. Stochastic masks learn causal importance rather than sensitivity
SPD adds a learned causal-importance function that asks, for every input and subcomponent, “how ablatable is this?” Its output lies between 0 and 1: importance 1 forces the component fully on, while 0 permits its mask to range randomly anywhere from zero to one.
A mistaken value exposes itself causally. If an indispensable subcomponent receives importance 0.5, random weakening between 0.5 and 1 damages reconstruction; through a reparameterized random sample, gradients flow back into the importance predictor until it learns that the component must remain stronger.
The predictor need not be another large model. In the paper, a roughly 16-to-128-parameter thresholding network used the overlap between activations and each subcomponent’s right singular vector—effectively measuring how much that component had available to read.
After training, the importance network can be discarded and every subcomponent summed to recover something close to the original model. Alternatively, it can identify the subset used on a distribution of interest, potentially yielding a much smaller task-specific parameter set; Lee labeled that future work.
13. SPD is better behaved, but semantic grouping remains unresolved
Unlike APD, SPD has no permanently silent non-top-k components: everything is stochastically active, so gradients continually reach every subcomponent. An offsetting junk component would also be randomly exposed and perturb activations, making hidden cancellation costly and producing a more continuous optimization problem.
Computation-first decomposition may also resist pathological feature splitting. A larger SAE might replace “words beginning with E” with separate features for “elephant” and everything else beginning with E; Lee argues that this is less interpretable if the network’s algorithm never treats elephant separately.
He was careful not to declare victory over sparse dictionary learning. There might be an ideal SAE scale, other variants may better identify computational variables, and respected researchers see problems in SPD too: “I don’t want to give the impression” that this is interpretability’s final word.
One toy result split
W_outinto 50 subcomponents that appeared to form one rank-50 component encompassing the matrix. Researchers infer grouping by varying decomposition strength and watching reconstruction loss and causal importances—preferably near 0 or 1—but Lee’s deeper answer was that no grouping is uniquely correct: “We’re all trying to discretise a fundamentally continuous object.”
14. Scaling will decide whether unlearning, monitoring, and science pay off
Toy problems with known mechanisms provide encouraging recovery tests: on problems with known ground truth, SPD recovered the expected mechanisms. Language-model scaling is the decisive experiment. The team stepped down into million-parameter models for traction and has tried systems in the “low single-digit billions”; components may still prove too high-rank for immediate understanding even after initial isolation.
Unlearning is the most direct parameter-space application. If a recovered vector performs capability X, researchers can modify that vector rather than translate an activation feature back into weights—though Lee explicitly left open whether this would outperform gradient-descent-guided removal.
Monitoring may benefit from tracking functions rather than input directions. Deception could arise across many situations and activation vectors, while one shared mechanism performs the deceptive transformation; SPD might flag “whenever the network was using function X” instead of searching for a single deception direction.
Scientific models provide the most expansive possibility: extracting knowledge from systems that are superhuman on trained tasks by reasoning “in the terms that the network itself is thinking in.” Lee presented that as a hope, not a result; whether parameter mechanisms generalize better than activation tools “remains to be seen.”