Model Interpretability and Internal Representations
Investigates internal representations, chain-of-thought monitoring, sampling dynamics, and interpretability mechanisms in large models.
3 sources · ↑4.5×Voices in this topic
All paragraphs
Oct 26, 2026
Oct 24, 2026
Oct 23, 2026
Oct 17, 2026
Oct 10, 2026
- Eric Bigelow Bigelow’s central claim is that an LLM’s decisive act often occurs during token sampling, not inside a fully settled plan. In his 2024 Forking Paths work, 30 rollouts from every token revealed moments when a 50/30 answer distribution suddenly collapsed—and …
- Modern reasoning models have partly moved uncertainty into a “linearized tree search,” but they have not eliminated critical forks. Rather than cleanly deriving A→B→C, they enumerate alternatives, backtrack with repeated “wait” moments—DeepSeek R1 might say it 50 …
- Interpretability has a practical 7–8 billion-parameter sweet spot for many rich behaviors, while frontier-risk research currently depends on Chinese open weights. Smaller models can support toy or narrow tasks, but Bigelow finds roughly 8B sufficient for zero-shot belief updating; at the top end, Kimi …
- Chain-of-thought monitoring is deteriorating under the same optimization that makes agents more capable. Outcome-based RL gives models no inherent reason to remain human-readable, producing strange dialects, compressed agent-to-agent messages, …
- The “stochastic parrot” metaphor should be retired, but stochasticity itself remains essential. Structured internal world models make the lookup-table critique shallow—“whatever these parrots are doing, it’s very, very interesting”—yet …
- The clearest product opportunity is to surface semantic uncertainty rather than present one answer as inevitable. Bigelow wants interfaces that expose meaningful branch points, alternative paths, and hallucination risk—a “GUI world instead of the Bash …
