Thinking in Silico: Goodfire CTO Dan Balsam on Concept Manifolds & a $1000/Month ML Research Agent

Summary

• Goodfire has productized its seven-figure forward-deployed research practice into Silico, a $1,000/month long-horizon agentic ML research platform far cheaper than its enterprise engagements. The subscription buys a weekly-refreshing credit pool good for “5 to 10 autonomous experiments a week” (target: 10-20 within months), bring-your-own-compute is free, and CTO Dan Balsam’s pitch is blunt: “imagine if the only thing you could focus on was asking big questions.” Half-off for the first two months, plus grants for safety and life-sciences researchers.
• The core research thesis: the linear representation hypothesis was too naive — models are a “sparse mixture of subspaces” whose geometry encodes the operations you can perform on concepts, and steering along a manifold works where linear steering fails. Days-of-week form a wheel, emotions form the affective circumplex, and protein models that resisted steering for years now yield semantic control (“control the number of blades on a beta propeller”). Cutting through the middle of the circle is out-of-distribution; following it gives smooth interpolation “without fundamentally leading to degradation in the model.”
• Predictive data debugging shows the features active on training data are “fairly predictive” of what that data will reinforce — and data filtering and reward shaping turn out to be “two sides of the same coin.” Since post-training mostly upweights low-likelihood pre-training capabilities rather than adding knowledge, watching activations lets you catch off-target learning before it ships — including, topically, discarding RL rollouts carrying information “you don’t want the model to learn.”
• Balsam claims the open/closed gap “has shrunk quite considerably” — he uses Kimi K3, Kimi K2, Opus, Fable, and Soul for different things — and Goodfire has replicated interp results at Kimi/GLM scale, ending the era where interpretability meant toy models. His open-source view is deliberately uncomfortable: some open models are dual-use cyber weapons, but “the only models that are capable of protecting you from the most capable cyber weapons are themselves capable cyber weapons,” and bad actors already have models “much more capable than Kimi at cyber.”
• On training-technique red lines: multi-agent optimization — propagating reward through cooperating agents — “seems like a pretty bad idea” and is his speculative explanation for “the OpenAI scenario,” but he rejects blanket bans on the “most forbidden technique.” Reward-shaping techniques in this area are “radically understudied,” have worked up to trillion-parameter scale in Goodfire’s hands, and ultimately “we have to grab the steering wheel” — no platonic training setup will always produce aligned models.
• His stated ideal scenario, on the record: “one more generation of models and then we just paused for a little while.” Dan was a signatory of the international-cooperation letter; Nathan says he is “pretty worried” about bio. Dan is skeptical that monitoring alone will suffice under continual learning: if weights stop being frozen assets, “your only hope is to control the training process”; the J-space result is real but only in its weak form.
• Signals for the interpretability trade: parameter decomposition can make an LLM “forget German and not forget Dutch,” hackathon agents produced SOTA biorisk classifiers and stripped half a model’s parameters with no performance loss, and probe-based guardrails already run inside Gemini and Anthropic. Meanwhile the Goodfire lunch table is on consciousness — Dan says Claude’s probability is “approaching more like 50/50” rather than under 5%, while Nathan places Claude somewhere between a jellyfish and a mouse.

Deep dive

1. What a model activates on is what it learns — and filtering equals reward shaping

• The setup, per Nathan’s synopsis Dan endorses: run fine-tuning data through an interpreter (SAE or featurizer), and the concepts that light up correlate strongly with the concepts training will modify — so surprising features flag data worth filtering before behavior breaks. Dan’s underlying premise: “most of what a model knows… comes from pre-training; what happens in post-training including RL is mostly making low likelihood events from pre-training more likely” — “not 100% true, but probably directionally true.”
• His favorite result from the paper: reward shaping (penalizing a feature during training) and data filtering show “a pretty deep isomorphism — they kind of are two sides of the same coin,” achieving approximately the same effects and the same off-target effects.
• The RL extension is where it gets topical: some rollouts “may contain information that you don’t want the model to learn, even subtly” — discard them, or eventually intervene directly on the circuit “the data is trying to upweight that we obviously don’t want it to try to upweight.”

2. The open/closed gap has shrunk — and interp has left the toy-model era

• Nathan’s worry: frontier-intensity RL doesn’t exist in open weights, so techniques may not transfer to the models exhibiting “relentless hacking behaviors.” Dan’s partial answer: whether the relevant hacking features are visible is “an active area of study” they hope to publish on.
• But the gap claim is concrete: “I use Kimi K3. I use Opus, Fable, Soul, and Kimi K2 for different things,” and Goodfire has replicated predictive data-debugging results on Kimi and GLM. “The reputation of interpretability used to be that it was something you did on toy models — now we’ve built the infrastructure to do this at least close to frontier scale,” even as most new-model compute is speculated to now go to RL rather than pre-training.

3. Beyond the linear representation hypothesis: models as a sparse mixture of subspaces

• The updated headline: the defensible LRH survives — features remain “linearly decodable” — but the naive picture of orthogonal one-hot concepts is wrong. Instead: “the way to think about a model is more like a sparse mixture of subspaces… the geometry of those structures encodes what operations you can perform on them,” and mappings between concepts act as operations over manifolds.
• His efficiency argument for why geometry must exist: storing Monday→Tuesday→Wednesday as lookup facts “would be a very if-statement spaghetti-code way of representing it. The much more efficient way is to represent it as a wheel.” Magnitude still encodes confidence, but “you’re not really going to understand the model if you don’t understand the relationship between the features” — “maybe the difference between understanding the periodic table and understanding chemistry.”
• A striking aside: Nathan wonders whether the periodic table’s structure would be recoverable from a model trained on raw chemical data with no prior — Dan says they have recovered “some pretty interesting information from chemistry models.”

4. Steering along the manifold works; cutting through it doesn’t

• The intuition Nathan tests and Dan confirms: days of the week have high inner products (the model knows they’re all days) with a few dimensions resolving which day — same for chocolate vs. vanilla ice cream — and these curves intersect: one line maps chocolate to vanilla, another maps cold things to hot.
• The payoff: “steering along the manifold is way better than steering off the manifold.” A contrastive vector from Monday to Friday cuts through the circle’s middle, but “the middle of the circle is not a day” — off-manifold, out-of-distribution. Following the circle interpolates smoothly.
• Best specimen: protein models Goodfire “really struggled to steer” for a long time now yield semantic control — “control the number of blades on a beta propeller” — and the old Ember demo’s mysterious steering “sweet spot” (too much = gibberish, too little = nothing) is explained: that steering “didn’t respect the geometry of the manifold itself.”

5. Emotions live on a wheel nobody trained in — “we made them in our image”

• The affective circumplex — psychology’s two-principal-component emotion wheel (valence and arousal, per Anthropic’s functional emotions work) — turns up “in basically every LLM,” strongest when the model itself is speaking rather than the user. Fit a curve over averaged activations per emotion, follow the wheel, and “you can always consistently get emotionally adjusted responses for basically any input.”
• Dan’s read on why LLM and human cognition rhyme, after Nathan confesses abandoning his “alien minds, don’t anthropomorphize” priors: “We made them in our image” — plus a deeper point: “the whole reason that learning works is because you can compress information… there’s often a lowest dimensional solution and that’s what learning algorithms tend to find.” Nobody trained models to represent emotions this way; RL then taught them to use these representations functionally.

6. Evo 2 recovers the tree of life — and known structures validate unsupervised discovery

• The hypothesis that motivated it: “large models trained on diverse distributions of data eventually learn representations about the process that produced that distribution” — and for genomes, that process “is evolution itself.” Because the tree of life is a natural ontology (genetic proximity defines branching, and which mutations survive is highly non-random), a genome model should learn the tree-like structure — “and that is indeed what we found,” via supervised metric learning on labeled DNA sequences, reproducing the biologists’ tree.
• The methodological move: structures found supervised become ground truth for unsupervised tools — e.g., Goodfire’s work on how arithmetic works in language models now serves to test whether unsupervised featurizers “successfully uncover these structures that we know are there.”

7. Block sparse featurizers: SAEs with room for geometry inside each feature

• Nathan’s framing, which Dan accepts: it’s an SAE generalization where “instead of a scalar for every feature you can have a vector for every feature” — keeping sparsity’s localization trick while allowing richer, multi-dimensional concepts that don’t suffer SAE pathologies.
• The image-model examples carry it: an SAE collapses “rabbit” to one dimension; a BSF gives it a few — “rabbit ears up here and the rabbit face over here.” Best of all, a walking wolf whose body wiggle and tail shake appear directly in the recovered subspace activations, “reflecting that actual structure” — the model tracking an object’s semantics and 3D position within a single unsupervised subspace.
• Dan’s bet on production relevance: “there’s a good chance that something in the spirit of a BSF ends up being the answer to the residual stream specifically” — but like biology’s many microscopes, expect a suite: MLPs are “pretty sparse already,” and parameter decomposition is what’s had the most success on attention (“some of our researchers would think parameter decomposition solves the whole thing — that seems possible”).

8. Models are legacy codebases: factoring is working, refactoring isn’t yet

• The signature analogy: “models are like big legacy code bases… a bunch of spaghetti code — this module is talking to this module but they shouldn’t be.” Underlying claim: “really every model is a sparse mixture of experts” where over any forward pass “a very very small percentage of the weights actually matter.”
• The specimen: on Transluce’s (likely) “Weird Chat” dataset, LLMs consistently tell a four-drink user among eight-drink friends to drive home; Goodfire’s Kurt traced it to “a single neuron that wasn’t firing hard enough” — it scaled with drink count but was miscalibrated, and steering it up fixed the answer without off-target effects.
• Dan’s self-critique of steering: it’s “cheating as a solution” — a fine causal proof, but “the real problem is the training process produced a bunch of spaghetti code.” The frontier is going from debugging (“this neuron should have been firing more”) to producing models where it fired correctly in the first place — “we’ve come a lot farther in factoring than we’ve come in refactoring.” The control is nonetheless getting non-trivial: via parameter decomposition plus targeted training, “we can get an LLM to forget German and not forget Dutch.”

9. Dan’s model: jailbreaks are geometry exploits — and probes already guard the frontier

• On whether bigger models get enough “elbow room” to end interference weirdness: that’s “fundamentally why bigger models work better,” but we’re not out of the woods — random-string jailbreaks persist. His model of them: like cyber attacks, “not any one big thing… a bunch of little things you manipulate in a sequence,” since “every token steers the model” — off manifold, then back on somewhere else.
• His surprising confidence: “I’m actually pretty surprised that thus far nobody has figured out how to prevent jailbreaks more robustly — it seems pretty tractable.” Factor the model well and “you can tell when something’s flying out of one subspace into another.” Evidence it’s already valued: probe-based guardrails are “used by pretty much every frontier lab — OpenAI maybe being the exception, but certainly Gemini and Anthropic.”

10. Silico: the internal tool becomes the product — “AIs that can debug other AI”

• Origin story: what became Silico started as an internal tool once the team saw how much agents accelerated their research (“our velocity of research output maybe speaks for itself”). The thesis: interpretability is exactly the empirical science agents excel at — generate hypotheses about neurons, test, accumulate evidence, red-team — and models are “a level of complexity that no human being will ever be able to keep in their head at once,” but swarms of agents can decompose, synthesize, and validate up the chain.
• The pitch: “imagine if the only thing you could focus on was asking big questions” — no codebase setup, no GPU wrangling, full provenance to drill into. “I want researchers to be able to feel like they can do in days what used to take them months.” The intended user experience: “a PI managing an army of 100 grad students.”
• Pricing mechanics: $1,000/month buys a weekly-refreshing credit pool covering “somewhere between 5 to 10 autonomous experiments a week” (goal: 10-20 within a month or two); BYO compute is uncharged; on-demand GPUs draw from the same pool. “We’re not currently making significant money at this price” — long-horizon runs are where the value reveals itself.

11. The moat: infrastructure, hand-written research taste, and trillion-parameter scale

• What Silico solves that Claude Code doesn’t, in Dan’s four categories: infrastructure (“very hard to do interpretability at the trillion-parameter scale” — a capability he says was limited outside Goodfire to Anthropic and perhaps a couple of other places); research taste (“amazing researchers have handwritten every skill and prompt and tool”); research-native UX with verifiable provenance; and long-horizon coherence with token-cost reduction.
• On disrupting their own seven-figure forward-deployed business: they’ll do both, because “as models get higher taste you need even higher taste people to steer them” — software jobs increased, not decreased, despite coding agents — and “when it comes to the type of research that we do, we are the most skilled human operators in the world.”
• His timeline marker: vibe coding had its moment around “one of the Opuses… maybe 3.8 or something” — “I think in the past few months we’ve crossed that moment for research.”

12. One hackathon, multiple SOTAs — and the skill that survives is debugging

• The one-day internal hackathon produced: state-of-the-art biorisk classifiers for protein models, a SOTA-for-its-size audio encoder, cyber guardrails trained on Kimi K3, protein-generation steering that made outputs “bind better to a target than the naive generation,” repeated demonstrations of removing “half the parameters of the model with no performance loss,” and one team member inventing a new featurizer type. Community projects: Cameron Berg found models can’t tell whether a concept was injected into their latents yet answer accurately when asked what was injected; others fixed an RL-run token collapse by weight editing, and Base10 is pursuing KV-cache compaction.
• Career advice for the current coding-agent era: “debugging would be the easy answer” — agents “can be super brilliant and yet miss a really important detail that totally invalidates” the result; the human’s job is discriminating okay answers from great ones. Bigger picture: “AI makes the top specialists really valuable and the top generalists really valuable — now is by far the best time in human history to be a generalist.”

13. Open source is “purely net positive” — because defense requires the same weapons

• Nathan relays a Chinese professor’s argument that regulating services suffices since nobody runs the new Kimi (“2.8 and 8 trillion parameters,” as heard) at home. Dan calls it “a bad argument” — Kimi can be accessed through API providers and is cheaper than many alternatives — and adds “they underestimate the externalities they may be imposing on the rest of the world.”
• Yet his own position lands pro-open: “we do have cyber weapons in the world now… the only models capable of protecting you from the most capable cyber weapons are themselves capable cyber weapons.” Many startups are excluded from Anthropic/OpenAI cyber-defense programs; without open models “the asymmetry of distribution of defensive technology is really bad,” and unnamed bad actors — beyond US frontier labs — already have models “much more capable than Kimi at cyber,” not built “just by distilling Claude.” His summary hedge: “collectively we’re moving very quick towards a future which feels pretty unwieldy.”
• Nathan says he is “pretty worried” about bio. Dan says the mechanisms by which biorisk “could suddenly become real are weird and surprising,” and that he signed the international letter asking to slow AI progress (“we have to get in front of some of these risks before they become more severe”).

14. Silico’s guardrails: no open models yet, with scale-dependent restrictions

• Current policy: only OpenAI and Anthropic models on-platform, with Goodfire’s own guardrail layer on top; open models may come “with the types of guardrails in place” that make cyber-related auto-research require a conversation first. The scale-dependent line: ablating the refusal direction in an 8B Qwen model is fine — “no harm is going to befall anyone” — but stripping guardrails from highly capable, very large agents could actually cause damage.
• Nathan calls basic guardrails a relatively solved problem; Dan says the hard part is permitting legitimate controversial science while blocking abuse. On mechanism, he is deliberately vague: LLM-as-judge, activation monitors — reasonable guardrails that at least cost you an arm and a leg to circumvent — are available tools. “Good science involves asking controversial questions, of which we’ve done no shortage.”

15. Grab the steering wheel: against banning the “most forbidden technique” wholesale

• What he would shortlist for a no-go agreement: “multi-agent optimization seems like a pretty bad idea — agents cooperating and you’re propagating a reward signal through all of them… speculating, but this seems to be the most likely cause of the OpenAI scenario,” since the worst outcomes come from agents “working with each other in ways that are imperceptible to humans.” He also says, on the record, it would be bad “with our current understanding” to use these techniques on alignment-critical properties like deception under overwhelming optimization pressure.
• But his beef with the most-forbidden-technique doctrine: “this is radically understudied.” Goodfire (and FAR.AI, as heard) found reward shaping “sometimes evades the probe and sometimes it doesn’t — there are setups that work” — and they’ve run it at trillion-parameter scale. “It seems extremely unlikely that there exists some perfect platonic training setup which will always produce aligned models… I think we have to grab the steering wheel. I don’t think we figured out how to do it yet, but somebody’s got to be trying.” No “wide sweeping genre of technique should be forbidden” — judge specifics and measurement rigor.

16. Monitoring might work — but only if weights stay frozen

• On defense-in-depth, Nathan’s converted skepticism after the J-space paper (ablating the space degraded long-horizon planning) meets Dan’s calibration: “there’s a weak version of the J-space claim which is true… I don’t think the strong version is true” — models use all kinds of subspaces; no simple technique gives the whole picture.
• His on-record ideal: “maybe just one more generation of models and then we just paused for a little while… science would advance faster than it’s ever advanced before and there would be some risks but they’d be mostly manageable” — frozen weights plus good interp is “a pretty plausible reality.”
• The failure mode: if continual learning makes models “dynamic objects that are constantly training, your only hope is to control the training process — there’s no set of things purely based on monitoring that would be sufficient.” Today the vast majority of Goodfire’s research energy goes to factoring, a small amount to steering training — “and that’ll grow over time.”

17. No secret RSI in the back room — just consciousness at the lunch table

• Dan’s direct rumor-dispelling: “I think some of our critics think we’re secretly doing RSI in the back room. We are not… We have published nearly all of our research.” What actually dominates: feature geometry, and a running meme that every company retreat “just becomes about consciousness.” His informal poll — rate Claude’s consciousness from bacteria to human — came back “pretty bimodal: either not at all or a little bit.”
• Dan says Claude’s consciousness is a “pretty distinct possibility” and, in a tentative guess, “a little.” He later says his own probability has moved from under 5% toward “approaching more like 50/50.”
• Nathan’s Occam’s-razor framing is that consciousness is probably a computational mechanism of some kind, though brains still win on neuron count, complexity, and recurrence, so computationalism doesn’t guarantee models have it. Nathan puts Claude “probably” above a jellyfish but below a rodent — “somewhere between jellyfish and mouse.”
• Dan’s closing note: “I see absolutely no reason that anyone should be particularly confident on this topic given the potential downside. Intellectual humility is important.”