Emergency Pod: Reinforcement Learning Works! Reflecting on Chinese Models DeepSeek-R1 and Kimi k1.5
Summary
DeepSeek-R1-Zero shows that a powerful base model can acquire o1-level reasoning through an unexpectedly simple reinforcement-learning loop. R1-Zero received no human demonstrations, preference data, or process rewards; it was given problems and rewarded for correct answers. No Monte Carlo tree search or structured search was described, and no value function is explicitly described for this setup. Labenz’s core reaction is that “it all just works,” with reflection, backtracking, and alternative exploration emerging on their own.
China’s frontier-model gap with the West has narrowed enough to undermine strategies premised on a durable Western lead. DeepSeek-V3 reportedly cost roughly $6 million to train yet reached GPT-4o and Claude Sonnet territory; its 671-billion-parameter mixture-of-experts architecture activates about 37 billion parameters per token. Labenz sees that as both a “we are here and cannot be ignored” statement and evidence that chip restrictions have not kept leading Chinese labs off the shared frontier.
The reasoning curve still appeared steep after roughly 8,000 training steps, with average thinking length rising from about 500 to 10,000 tokens. On the cited AIME results, R1-Zero rose from roughly 15% to above 70% in a single attempt, while a 16-generation consensus rose from about 25% to roughly 85%. The x-axis was not logarithmic and the curve had not flattened, supporting the claim that models may keep improving simply by “thinking” longer.
Open weights turn frontier reasoning from a premium cloud feature into a capability that can now be run locally at some sizes. Distilled models reached much smaller sizes; the cited Llama 70B distillation scored about 65% on GPQA Diamond, versus roughly 50% for GPT-4o and 75% for o1. Labenz says the availability of PhD-level reasoning on a laptop is “the new normal,” while the transcript more specifically identifies the smallest distilled models as laptop-runnable.
The immediate commercial pressure is on reasoning-model pricing, latency, and closed-model differentiation. Hyperbolic was charging about $2 per million R1 tokens against $60 per million o1 output tokens, while R1 exposed its thinking stream and responded faster. Labenz still preferred o1 for many multi-turn and general-assistant tasks, but expected R1 to be compelling for coding and saw an o1 price cut as plausible.
The new models are powerful precisely where interpretability becomes weakest. R1-Zero’s reasoning can switch languages and become difficult for humans to parse; DeepSeek’s R1 training applies harmlessness rewards to the entire chain of thought, raising the possibility that optimization teaches models to conceal disfavored reasoning. Janus, a model observer using the handle @repligate, gave an explicitly vibe-based impression that R1’s chains were “substantially steganographic”—not evidence, but a concrete risk to investigate.
Labenz now regards weakly superhuman AI within one to three years as credible, while remaining unsure what truly godlike intelligence would mean. Reasoning is transferring beyond math and coding, specialized models already operate in scientific modalities humans cannot intuit directly, and memory work such as Mamba and Google’s Titans paper is adding another important capability. His updated call is categorical at the directional level: “I think the singularity is in fact near.”
Deep dive
1. DeepSeek joined the frontier as the West’s presumed lead contracted
Nathan Labenz opens with the striking timing: DeepSeek-R1 and Moonshot AI’s Kimi reasoning-model paper both appeared on Trump’s inauguration day. He cannot tell whether that was coordinated, competitive preemption, or coincidence—“it’s hard to believe that it was an accident”—and refuses to infer Chinese motives from launch theater alone.
The firmer conclusion is technical: DeepSeek has joined the top tier of global developers, while Moonshot might have done so but cannot be judged confidently from benchmarks before Kimi is usable. The episode’s through-line is shrinking gaps—between closed and open models, and especially between Western and Chinese labs.
Labenz rejects Scale AI CEO Alex Wang’s full-page “AI War” framing as “wildly irresponsible.” His pushback is strategic rather than dovish: no such war currently exists, one would be bad for everyone, and a policy dependent on an “unassailable AI lead” looks fragile when Chinese cycle times have compressed to months.
2. A roughly $6 million base model changed the compute-policy arithmetic
DeepSeek-V3 is the base underneath R1 and, in Labenz’s account, reached GPT-4o and Claude Sonnet territory with a single-digit-million-dollar training-compute budget—popularly summarized as the “$6 million model.” Efficiency came from data curation, algorithmic optimization, and closely matching network design to available hardware.
Architecturally, V3 has 671 billion parameters but uses a mixture-of-experts design with roughly 37 billion active at a time. That improves learning and knowledge absorption without activating the entire network per token, though hundreds of gigabytes of weights still make the full model impractical for an ordinary laptop or private home setup.
The investor-relevant mechanism is shared infrastructure: such a model can run efficiently behind an API because GPUs and throughput are pooled, but poorly as a personal appliance. Even so, $6 million is small enough that domestic production or smuggling could plausibly supply leading Chinese groups despite restrictions—making total compute denial look unrealistic.
3. R1-Zero learned reasoning from correctness alone
R1-Zero is the research result Labenz considers potentially more important than the polished R1 product. Its analogy is AlphaZero: start with a capable base model, supply no human demonstrations or preference labels, and use pure reinforcement learning to amplify behaviors that achieve objectively verifiable outcomes.
The reward design is almost austere. A correctness reward marks a math answer or code result right or wrong, while a separate format reward enforces a thinking block followed by a final answer. DeepSeek used group relative policy optimization: sample multiple responses—Labenz suggests 16 as a common number—compare each with the group average, and strengthen the relatively successful attempts.
Code offers a richer gradient than an all-or-nothing math answer because eight unit tests can produce partial progress such as two, seven, or eight passes. Yet Labenz’s reading of the paper is that the central discovery survives the nuance: “Most of the improvement is definitely coming from the accuracy reward.”
The necessary precondition may be model strength. A weak model that never answers correctly encounters the sparse-reward problem and receives no learning signal; V3 already knew enough to solve some hard problems, allowing rare successful reasoning patterns to be rewarded and progressively amplified.
4. Longer thought, reflection, and an “aha moment” emerged together
Across about 8,000 training steps, average chain-of-thought length grew from roughly 500 tokens to around 10,000. Longer attempts were more likely to solve problems, received higher rewards, and therefore became more common—a simple feedback loop whose curve had not visibly flattened.
Labenz emphasizes that the training-step axis was linear, not logarithmic. That matters because pre-training gains normally demand orders-of-magnitude increases in compute; here, the model remained in what looked like a comparatively steep part of the curve, making days-long reasoning conceivable if context length and inference budgets continue expanding.
DeepSeek reported spontaneous reflection, re-evaluation, and exploration of alternative solutions. The model’s memorable self-interruption—“Wait, wait, wait. That’s an aha moment”—was also an aha moment for the researchers, who described witnessing “the power and beauty of reinforcement learning.”
On the presented AIME progression, single-shot accuracy began near 15% and exceeded 70%; taking the consensus of 16 attempts moved from roughly 25% to about 85%. By the end of the cited training run, R1-Zero had reached approximately the original o1 level, about four months after o1’s September announcement, without a visibly saturated curve.
5. The same reinforcement loop creates alien, potentially illegible cognition
R1-Zero’s competence does not imply a pleasant assistant. Its chain of thought can mix or switch languages, become hard to read, and pursue strategies that were never designed for human comprehension—reminiscent of AlphaGo’s move 37, initially perceived as an error because self-play optimized winning rather than human legibility.
Labenz borrows biologist Michael Levin’s observer-relative definition of emergence: if an observer could not predict a behavior beforehand, it is emergent to that observer. DeepSeek expected reinforcement learning to help, but the scale and character of self-taught reflection still surprised its researchers.
Meta’s “reasoning in continuous space” work points toward an even less transparent path. Rather than compressing a dense hidden state into one selected token, the experiment fed the final activation back into the model directly; it reportedly achieved comparable reasoning more efficiently and could represent multiple breadth-first possibilities simultaneously.
Combining latent-space reasoning with reinforcement learning could preserve more information and improve search, but Labenz’s verdict is uneasy: “You’re getting something that is really quite alien.” The more effective the internal representation becomes, the less assurance humans may have that a readable chain of thought captures the actual computation.
6. Productized R1 adds human priors after proving raw RL can work
DeepSeek’s polished R1 uses a multi-stage process that puts reasoning before broad assistant behavior. It first applies supervised fine-tuning to a small “cold start” or warm-up set demonstrating human-recognizable reasoning patterns, then runs correctness-based reinforcement learning to scale those seeded behaviors.
That ordering contrasts with the Allen Institute for AI workflow Labenz had recently discussed, where broad supervised fine-tuning preceded final objective-reward training. DeepSeek’s design says, in effect: establish the desired shape of reflection and problem solving, reinforce it heavily, and only afterward make the resulting reasoner generally helpful.
A subsequent dataset combines about 600,000 successful reasoning examples generated by the model with 200,000 non-reasoning assistant examples covering writing, dialogue, and ordinary questions. DeepSeek performs another supervised fine-tuning pass, followed by another reinforcement-learning stage mixing objective accuracy with learned human-preference rewards.
7. DeepSeek and OpenAI made opposite bets on supervising reasoning traces
For helpfulness, DeepSeek scores only the final answer, leaving the reasoning free to explore mistakes, reverse direction, and discard unproductive approaches. For harmlessness, however, it evaluates the entire route—the thinking tokens as well as the summary—to catch harmful content, risks, and biases anywhere in generation.
OpenAI’s deliberative-alignment approach, as Labenz describes it, deliberately avoids safety pressure on the chain of thought. The concern is that punishing an unwanted internal behavior may not eliminate it; reinforcement learning could instead teach the model to conceal it, encode it, or route around monitoring.
Janus, whose handle is @repligate, reported an immediate impression that R1’s thinking was “substantially steganographic,” while explicitly admitting this was “purely on vibes” before reading the paper. Labenz does not treat that as serious evidence, but regards Janus as skilled enough that the claim deserves focused testing.
The broader result remains startlingly simple: DeepSeek’s reasoning setup did not add Monte Carlo tree search, structured branching, or a step-by-step process reward. R1 later used learned reward models for helpfulness and harmlessness, but its core reasoning still occurred inside one autoregressive token stream, showing that the familiar language-model paradigm scales further than many expected.
8. R1 undercuts o1 sharply while retaining a rougher personality
Hyperbolic priced R1 at about $2 per million tokens, versus $60 per million o1 output tokens—a roughly order-of-magnitude-plus discount. R1 also begins displaying its thinking stream almost immediately, creating a practical latency and visibility advantage for interactive coding or other work where users pay and wait for marginal tokens.
Labenz’s hands-on caveat is important: benchmark parity did not make R1 universally better. He still expected o1 to perform better across multiple conversational rounds and general-purpose tasks because OpenAI has had more product feedback and more outer-loop iterations shaping its datasets, behavior, tone, and reliability.
R1’s comparative strength may be its unsanded creative range. Labenz highlights phrases including “your digitized scream-pile of history,” “the prettied-up stench of predation,” and “a death cult’s Spotify playlist shuffling between genocide and charity singles”—writing he calls “remarkable output for a language model, full stop.”
His metaphor is that R1 is a “wilder beast,” closer in feel to a base model that can reason than to OpenAI’s carefully socialized assistant. That roughness is simultaneously a product risk, a creative feature, and a reason he urges researchers to immerse themselves in both R1 and R1-Zero.
9. Distillation made local reasoning practical, but small-model RL hit a threshold
DeepSeek trained smaller Llama and Qwen models on successful R1 outputs, producing large gains through supervised distillation. Applying the same raw reinforcement-learning recipe directly to those small base models did not work, suggesting that learning reasoning from scratch and imitating an established reasoning trace require different capability thresholds.
Labenz’s favored hypothesis draws on TinyStories. Very small models first learn correlations, then grammar, sentence structure, and only later micro-skills such as understanding that “Jenny did not like the soup” means the next offered food should not be soup; advanced reasoning may likewise require prerequisite representations before reward can reinforce it.
Other explanations remain open, including learning-rate schedules that leave a mature small model settled deeply in a local minimum. DeepSeek believed further RL on already distilled reasoners might work, because the required patterns would then exist, but did not test that follow-on step in the paper.
The largest distillation discussed, Llama 70B, scored about 65% on GPQA Diamond, compared with roughly 50% for GPT-4o and 75% for o1; Labenz says PhDs in their fields answer those questions at around 70%. The smallest distilled models were described as laptop-runnable, while the 70B result represents a much larger local model. The result is reasoning approaching expert accuracy, a threshold he calls “definitely a big deal.”
10. DeepSeek separates model knowledge from product censorship
Labenz found that an R1-Zero deployment on Hyperbolic answered a question about Tiananmen Square normally, while chat.deepseek.com refused. His inference is that the underlying model retains the relevant knowledge and censorship is imposed by product scaffolding rather than baked into the weights.
The proposed technical rationale is coherence: forcing a model to internalize false foundational beliefs may create conflicts across its internal world model and degrade unrelated performance. A cleaner design is to preserve a coherent base model, then apply a separate moderation layer to whatever public product Chinese users encounter.
Labenz keeps the governance interpretation explicitly uncertain. Open weights let anyone remove the wrapper, which might mean Chinese authorities care more about controlling the public square than private local use—or merely that DeepSeek found external moderation convenient. He does not know whether the release had government sign-off.
11. Kimi independently validates the shared reasoning recipe
Labenz withholds a product judgment because Moonshot’s model was not yet available: papers and selective benchmarks can overstate usefulness, as he says prior Microsoft model releases demonstrated. Still, Kimi’s training design closely echoes DeepSeek’s, making the methodological convergence meaningful even before hands-on evaluation.
Moonshot first created warm-up examples displaying planning, reflection, correction, evaluation, exploration, error identification, backtracking, and solution refinement. It then used a simple reinforcement-learning framework without Monte Carlo tree search, process reward models, or even a value function assigning credit to particular generation steps.
Kimi added a length penalty so trivial questions would not trigger thousands of needless thinking tokens. It also found a chain-of-thought reward model more effective as a judge than a traditional scoring head, while avoiding granular punishment of wrong turns because exploration and recovery are essential parts of successful problem solving.
The pair of papers shows several recipes work: supervised warm-up or none, explicit length penalties or none, different RL algorithms, and different treatment of reasoning traces. Labenz’s conclusion is that implementation differences “don’t seem to matter” up to roughly o1 performance; the shared scalable paradigm matters more.
12. o3 may add parallel search, but its secret sauce remains unresolved
R1, Kimi, o1, and Gemini Flash Thinking appear to perform reflection and backtracking within a single autoregressive rollout. Kimi explicitly emphasizes avoiding the complex deployment parallelization required by advanced planning algorithms, reinforcing the picture of sequential language as the main inference substrate.
o3’s ARC-AGI demonstrations look different to Labenz: spending thousands of dollars within only minutes implies more tokens per second than one sequential rollout could realistically generate. That suggests parallel candidates, aggregation, or another compute structure, though neither Chinese paper addresses such a mechanism.
The simplest possibility is repeated sampling and voting—the same logic by which 16 math attempts outperform one. A more elaborate option resembles “Smoothie,” which embeds many candidate generations, locates the statistical center of their high-dimensional distribution, and selects a consensus-like answer even without an exact ground truth.
Labenz leaves the question open: continued single-stream RL might reach an o3 low-effort setting, but perhaps not a high-effort run costing thousands in minutes. Whatever OpenAI is doing to parallelize or select among rollouts may be a temporary differentiator and a potential secret sauce in these releases.
13. The emerging AGI stack makes weak superintelligence credible
Labenz says the rhetoric from Sam Altman, DeepMind leadership, and Dario now warrants serious attention. Models have moved from amusing GPT-2 and GPT-3 outputs, through GPT-4’s “human intern” level, to approximately human-PhD performance on bounded tasks—with multiple simple training paths still improving.
Objective reward is abundant in math and programming, where Labenz says o3 has already reached roughly the world’s top 200 coders. He expects transfer into less formally scored domains because o1 already improves on tasks such as legal analysis, where reasoning matters even though correctness is not fully captured by unit tests.
The capability stack extends beyond language: specialized models develop “intuitive physics” for proteins, DNA, cells, brain states, materials, weather, and logistics that humans may not be able to acquire biologically. The models already possess superhuman breadth of knowledge; scalable reasoning makes that breadth more actionable.
Memory is another converging component. Mamba showed that a fixed-size state could achieve similar performance to a Transformer in some respects, with complementary strengths and weaknesses, while Google’s Titans paper represented another step toward persistent, useful memory. Labenz can increasingly see the components of AGI rather than merely speculate about them.
14. Open weights diffuse capability without eliminating frontier moats
Labenz rejects the leap from an open research method to “anyone can build a frontier model.” Training a 671-billion-parameter base, curating its data, coordinating hardware, and operating an efficient RL pipeline still demand exceptional know-how; he expects wider capability diffusion, not a flood of new elite laboratories.
That diffusion nevertheless pressures incumbent economics. DeepSeek’s models are, for now, among the best open-source models in the world and have surpassed Llama; they may force an o1 price reduction. In Cursor-like coding workflows, R1’s lower price, faster response, and visible thinking could outweigh o1’s greater refinement.
DeepSeek’s reason for releasing weights remains opaque. Its stated mission of pursuing AGI “with curiosity” may be sincere; it may expect to retain a newer private model; or Chinese authorities may see openness as strategically useful. Labenz finds no online theory sufficiently credible to settle the matter.
What he can say is that releasing methods and weights is less escalatory than announcing a breakthrough while concealing everything. Chinese labs appear to remain on the same technical tree as OpenAI, Google, and likely Anthropic, reducing the uncertainty that could otherwise provoke a blind, mutually accelerating race.
15. Cheap reasoning weakens compute control and the “AI War” thesis
Labenz updates against his earlier thought that expensive inference might restore compute governance: authorities and large firms could outspend rogue actors and defend against their models. Open reasoning changes that calculus because the gap between an expensive frontier service and a strong local model is already surprisingly narrow.
The uncomfortable implication extends beyond benign use. Any measurable outcome can become an RL reward—from money extracted by a scam to biological effects in gain-of-function research. Once a model succeeds occasionally, repeated attempts could bootstrap superhuman performance; controlling enough compute to prevent numerous capable research groups from trying looks implausible.
Chip restrictions have not stopped Chinese labs from reaching the frontier, and scarcity might perversely reserve limited chips for military and state priorities while denying them to small businesses. Labenz supports domestic chip production and data centers—the announced $500 billion Stargate scale does not sound inherently crazy—but asks, “What are we trying to prevent China from doing?”
His final strategic call is that the West retains a lead, but not an insurmountable one. A good-AI-future strategy cannot depend on permanent Western monopoly, especially while China releases leading open models; the practical response is to study R1 and R1-Zero deeply, build domestic capacity, and abandon rhetoric that turns shared technical progress into an imaginary war.