Sebastian Raschka
Frontier Insights
Frontier Thesis: AI’s value driver has shifted from sheer pre-training scale to hybrid optimization—RLVR, test-time compute, and specialized post-training. Coding is the premier monetizable frontier, while distillation turns API access into a geopolitical and IP chokepoint.
Strategic Moat: Durable advantage no longer relies on public benchmarks. It requires proprietary data, CUDA-level infrastructure, private evaluation pipelines, and integrated distribution.
Risks & Warnings: Open-weight ecosystems—particularly China’s—erode model margins; SWE-bench contamination renders public metrics unreliable; and high inference costs, legal exposure, and severe reliability gaps still choke real enterprise deployment.
Key Views & Dialogues
[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka
Anthropic’s distillation warning turns frontier API access into a geopolitical and capability-control issue. Nathan Lambert argues GPU-constrained Chinese labs “obviously should do this”: buying Claude outputs is easier than generating comparable synthetic data internally, while Anthropic labels distributed collection an “attack.” Terms of service mainly let providers terminate access, but renewed enforcement raises the possibility…
View summary (7)
- Anthropic’s distillation warning turns frontier API access into a geopolitical and capability-control issue. Nathan Lambert argues GPU-constrained Chinese labs “obviously should do this”: buying Claude outputs is easier than generating comparable synthetic data internally, while Anthropic labels distributed collection an “attack.” Terms of service mainly let providers terminate access, but renewed enforcement raises the possibility that providers keep their strongest models in products or restrict premium API access.
- The enforcement problem is epistemic: evaluation and distillation look identical while the API is running. Sebastian Raschka uses essentially the same loop for MATH-500, a 12,000-example math set, and book experiments with DeepSeek V3.2; only later would someone save the outputs and train. Volume, repetition, and topic distribution might expose collection at tens or hundreds of thousands of requests, but legitimate benchmarks run questions three to five times and production chatbots also generate huge traffic—creating an uncomfortable privacy trade-off.
- Anthropic’s evidence looks highly timing-sensitive, particularly the gap between MiniMax and DeepSeek traffic. swyx highlighted MiniMax redirecting nearly half its traffic when Opus 4.6 launched, consistent with active MiniMax 2.5 training; DeepSeek’s roughly 150,000 exchanges could reflect a different training window. Without knowing whether requests span four weeks or six months, Nathan sees Anthropic’s DeepSeek emphasis partly as messaging: it is “the only Chinese AI name that people in the US know.”
- Distilled data is not automatically valuable—the teacher must fit the student. Nathan says Qwen-dense models often teach Qwen-like open models better than stronger frontier systems, while GPT-OSS is also a surprisingly good teacher: “Just because you have this data does not mean it’s actually gonna make your model better.” A lab needs evidence that Claude data produces signal before paying to extract 100 billion tokens, especially when large models generate at roughly 40 tokens per second.
- The API may be a weaker moat than the products built around it. Nathan calls the API market “brutally competitive” and suggests a distillation-worried Anthropic could release its best model directly into Claude Code without offering it broadly; Sebastian counters that chatbots, OpenClaw workloads, and users exceeding subscription caps create a large unsubsidized-token business. swyx thinks Anthropic has “Apple vibes,” while describing OpenAI’s leadership as less likely to lock models behind products out of idealism and principles—even though the last three GPT-5 Codex variants reportedly received two-to-four-week product-exclusive windows.
- SWE-bench Verified has stopped discriminating among frontier coding models because both saturation and task defects overwhelm the signal. Scores cluster in the 80s while run-to-run noise is roughly 0.5–1 point, making tiny release-to-release gains “obviously bullshit.” OpenAI’s later audit found 59% of the stubborn remainder could not be solved as written; one task secretly required the exact string
get_annotation, making memorization the only reliable route. - The next evaluation cycle becomes materially more expensive and more private. GPT-5 used knowledge from later Django versions, while Gemini Flash and Opus could reconstruct a task and solution from its ID—evidence that public GitHub benchmarks contaminate models even without deliberate cheating. SWE-bench Pro adds newer tasks, private/public splits, more repositories and languages, but the panel expects frontier evals to cost millions, then potentially “tens of millions and hundreds of millions of dollars,” advantaging labs and data providers with scale.