Researchers
2 Curated Dialogues
Nathan Lambert
Frontier Insights
Core Frontier Thesis: Frontier AI value is decoupling from brute pre-training scale toward architectural efficiency (MoE/MLA), test-time compute (RLVR), and post-training distillation, radically resetting cost curves and triggering Jevons paradox.
Strategic Playbook: With open weights expanding geopolitical leverage, defensibility shifts from raw model size to proprietary data, CUDA-level systems engineering, reliable distribution interfaces, and private evaluation harnesses as public benchmarks degrade.
Risks & Warnings: Export controls choke inference deployment over training; model distillation weaponizes frontier APIs into regulatory flashpoints; and severe benchmark contamination obscures true capability gains amid soaring reliability and legal hurdles.
Key Views & Dialogues
Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
Lambert estimates distillation may give Chinese labs a few months, but its value in RL-heavy post-training remains unclear. Closed labs integrate inference and lead coding, where revenue has gone from zero to $100 billion, while open-model economics fragment builders, inference firms and post-training companies. Open models may still win buyers seeking owned weights and the cheapest adequate model.
View summary (8)
- Nathan Lambert’s organizing framework is that model progress reduces to three forms of capital — compute, data, and talent — and China is compute-starved with a comparable talent pool. He allows there might be a slight talent edge depending on who was where when AI got going. His compute anecdote: tell a top Chinese pre-training researcher that “an average OpenAI researcher has 5,000 H100 equivalents that they can access at any time” and “their jaw just falls to the floor.” Distillation can help compensate because “outputs from a language model are a function of compute” — it can be a way to get something compute-equivalent.
- Lambert’s literature-based estimate is that distillation helps Chinese labs by a few months, but its role in today’s RL-heavy post-training is hard to establish. His evidence: open post-training recipes are “stuck using a really outdated Qwen model” because it is hard to get the teacher right, and he doubts Claude Opus reasoning traces would have made OLMo much better. He says 20–30 Chinese companies are building good models, so the idea that “only we are good at building models” is “a bit of cope.”
- He opposes government action on distillation, while supporting labs securing their own products if they choose. He questions government involvement in what he calls a terms-of-service violation; labs can cancel access, and restricting models to products or enterprise agreements would make distillation harder. Jensen Huang’s quoted line anchors the segment: “People distill my products every single day… That’s called competition.” Lambert argues the Chinese labs “are not gonna stop,” and that action against distillation could do more harm than good.
- The open-model economy runs on a slower exponential than closed labs but can end in real ownership — “model weights, no terms and conditions,” and the ability to run them on your own hardware. Closed labs have integrated inference and reach high-value markets like coding first; the revenue there has gone from zero to $100 billion and is likely to grow. Open models involve builders, inference companies, and post-training companies, so customer information and revenue do not flow back to model builders easily. Lambert is “less concerned about this being kind of bubbly”: open models could gain ground as buyers figure out the cheapest model that gets a task done.
- A lot of frontier progress in less-verifiable domains comes from high-quality data and environments, not simply “poof generalization.” Lambert says it is probably some of both, but a lot comes down to data. He balks at the scale of research budgets for environments, while arguing labs buy them because models improve from them. The gold rush also brings corner-cutting and reward hacking downstream of the data supply chain. Investable wrinkle: a Chinese company may buy an RL environment and use it six months later.
- Lambert is skeptical of fast takeoff on current evidence: “all the scaling laws are an exponential increase in compute for a linear increase in performance.” He cites Claude system cards as saying, roughly, that it would take “10× more progress to make this a fast takeoff.” His stated worry is the capability curve alongside “pretty much nonexistent preparation” for public cybersecurity — “not fearing fast takeoff does not mean that you don’t worry about risks.”
- His new nonprofit Trillium will build open-source environments and share its research and model-use process — tasks, configs, trajectories, and failures — to demystify RSI. The name is the thesis: a trillium is “a spring ephemeral flower” that blooms before the canopy and helps spread nutrients. Lambert wants open infrastructure in an industry where “all CS research… the benefit accrues to the tech industry in about 10 years,” while researchers can make “nine-figure returns… within five months of founding a startup.”
- Entry barriers are lower than the discourse implies: “it’s not that you need $10 billion to have a relevant model,” and frontier-lab ideas can travel over “one beer” (with SSI raised as an apparent exception). The durable closed-lab advantage, in Lambert’s view, is serving “the cheapest intelligence per watt.” He poses an illustrative price-drop scenario — roughly 2X every eight months — and asks whether the result would be higher margins from the Jevons paradox or a market that fills up. The open question is whether coding is the main profitable market (favoring frontier integration) or whether the market is all knowledge work (“they can’t do everything”).
[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka
Anthropic’s distillation warning turns frontier API access into a geopolitical and capability-control issue. Nathan Lambert argues GPU-constrained Chinese labs “obviously should do this”: buying Claude outputs is easier than generating comparable synthetic data internally, while Anthropic labels distributed collection an “attack.” Terms of service mainly let providers terminate access, but renewed enforcement raises the possibility…
View summary (7)
- Anthropic’s distillation warning turns frontier API access into a geopolitical and capability-control issue. Nathan Lambert argues GPU-constrained Chinese labs “obviously should do this”: buying Claude outputs is easier than generating comparable synthetic data internally, while Anthropic labels distributed collection an “attack.” Terms of service mainly let providers terminate access, but renewed enforcement raises the possibility that providers keep their strongest models in products or restrict premium API access.
- The enforcement problem is epistemic: evaluation and distillation look identical while the API is running. Sebastian Raschka uses essentially the same loop for MATH-500, a 12,000-example math set, and book experiments with DeepSeek V3.2; only later would someone save the outputs and train. Volume, repetition, and topic distribution might expose collection at tens or hundreds of thousands of requests, but legitimate benchmarks run questions three to five times and production chatbots also generate huge traffic—creating an uncomfortable privacy trade-off.
- Anthropic’s evidence looks highly timing-sensitive, particularly the gap between MiniMax and DeepSeek traffic. swyx highlighted MiniMax redirecting nearly half its traffic when Opus 4.6 launched, consistent with active MiniMax 2.5 training; DeepSeek’s roughly 150,000 exchanges could reflect a different training window. Without knowing whether requests span four weeks or six months, Nathan sees Anthropic’s DeepSeek emphasis partly as messaging: it is “the only Chinese AI name that people in the US know.”
- Distilled data is not automatically valuable—the teacher must fit the student. Nathan says Qwen-dense models often teach Qwen-like open models better than stronger frontier systems, while GPT-OSS is also a surprisingly good teacher: “Just because you have this data does not mean it’s actually gonna make your model better.” A lab needs evidence that Claude data produces signal before paying to extract 100 billion tokens, especially when large models generate at roughly 40 tokens per second.
- The API may be a weaker moat than the products built around it. Nathan calls the API market “brutally competitive” and suggests a distillation-worried Anthropic could release its best model directly into Claude Code without offering it broadly; Sebastian counters that chatbots, OpenClaw workloads, and users exceeding subscription caps create a large unsubsidized-token business. swyx thinks Anthropic has “Apple vibes,” while describing OpenAI’s leadership as less likely to lock models behind products out of idealism and principles—even though the last three GPT-5 Codex variants reportedly received two-to-four-week product-exclusive windows.
- SWE-bench Verified has stopped discriminating among frontier coding models because both saturation and task defects overwhelm the signal. Scores cluster in the 80s while run-to-run noise is roughly 0.5–1 point, making tiny release-to-release gains “obviously bullshit.” OpenAI’s later audit found 59% of the stubborn remainder could not be solved as written; one task secretly required the exact string
get_annotation, making memorization the only reliable route. - The next evaluation cycle becomes materially more expensive and more private. GPT-5 used knowledge from later Django versions, while Gemini Flash and Opus could reconstruct a task and solution from its ID—evidence that public GitHub benchmarks contaminate models even without deliberate cheating. SWE-bench Pro adds newer tasks, private/public splits, more repositories and languages, but the panel expects frontier evals to cost millions, then potentially “tens of millions and hundreds of millions of dollars,” advantaging labs and data providers with scale.