117. An Open-Source Journey Through AI Papers and Model Evolution

2025/10/28 · Source link · AI Summary archive

Summary

The most important judgment for the AI industry is that model progress has never been a single-architecture miracle, but the combined lift of four curves: chips, infra, data, and model structure. Transformer won because it could model long-range relationships directly and “hit the hardware lottery,” making it well suited to GPU parallelism and scale; since 2017, its backbone has changed little, yet it has unified language, images, video, and Diffusion. Xie Qingchi’s summary: compute, data, and model structure must scale together.
Scaling Law has yet to reveal a clear ceiling, but the growth axis has expanded from pretraining alone to data quality, post-training, and test-time computing. OpenAI went from roughly 0.1B parameters and 5GB of text for GPT-1 to 175B and 570GB for GPT-3; Chinchilla then argued that parameters and tokens should grow proportionally, and that an overtrained small model may offer better lifetime economics. Xie remains optimistic: “There are still many places where we’re doing things very crudely,” leaving room to scale synthetic data, search, and inference-time compute.
The real compute moat is not how many GPUs you buy, but whether you can organize memory, bandwidth, and compute units into a stable system. Brook, CUDA, ZeRO, DeepSpeed, and MegaScale trace an infra evolution from “programmable GPU” to single-task training across tens of thousands of GPUs; current best-in-class training may approach 50% MFU, meaning a substantial share of compute is still lost to communication and waiting. DeepSeek illustrates the value of algorithm-infra co-design: “compute doesn’t need to wait for communication,” and scarce hardware can force efficiency innovation.
The data moat is not simply exclusive access to corpora; the ability to automatically clean, combine, and understand data matters just as much. LAION-5B used CLIP to build 5B image-text pairs from Common Crawl, several orders of magnitude larger than ImageNet’s roughly 14M; RefinedWeb showed that internet data plus rigorous cleaning can approach or exceed curated human datasets. One counterintuitive detail: LAION labeled NSFW content instead of deleting it, because removing it entirely could damage the model’s understanding of human anatomy—capability training and output safety are separate problems.
Post-training has already shown that model value cannot be measured by parameter count alone. CoT activates latent reasoning with “Please think step by step,” while ReAct connects reasoning, action, and observation into an Agent loop; InstructGPT even let a 1.3B model beat GPT-3’s 175B model on instruction following. The key shift is that a model may already be powerful after pretraining, with product differentiation coming from “how you bring that power out”; prompt engineering is therefore evolving into context engineering.
The central multimodal strategy is to compress different modalities into a jointly computable representation space, then keep scaling through the Transformer ecosystem. CLIP learned a joint vision-language representation from 400M image-text pairs; Stable Diffusion performs diffusion in latent space, reducing pixel-level compute while adding text control; ViT and DiT follow the strategy of “if you can’t make the model fit the data, make the data fit the model.” Xie expects Transformer to continue moving into image generation, while autoregressive models may once again challenge Diffusion.
For products and organizations, understanding model boundaries sometimes means investing decisively, and sometimes means being “perfectly comfortable waiting.” In one customer-service project, GPT-4o was capable but too slow, while DeepSeek-V3 and Qwen 2.5 were still roughly 20% short of full coverage; instead of spending 3–4 months rebuilding SFT, the team waited 2–3 months, and Qwen 3’s largest model covered the requirement. The corresponding career shift is from narrow roles toward becoming a builder: “owning an end-to-end outcome.” Xie’s personal view is that OpenAI is using MCP, a super app, and a model entry point to pursue the next operating system.

Deep dive

1. Reading Papers as a Product Manager’s Tool for Exploring Technical Boundaries

Although Xie Qingchi says he is “not from a technical background,” both his undergraduate and graduate degrees were in computer science. He then spent 9 years doing product work and entrepreneurship at Douban, including O2O and consumer businesses, until GPT emerged in 2022 and convinced him that the technology industry had entered a major transition.
His definition of a product manager is: “A product is about finding the optimal solution within a given set of boundaries.” Understanding those boundaries can change the optimum and sometimes push the boundaries themselves, so papers are not just for engineers.
He read papers in the internet era too, but they were on urban planning and architecture: while building Douban’s “Alpha City,” he systematically studied Jan Gehl and the history of urban development. The difference is that mobile internet technology was already relatively mature, while AI remains early-stage and products need to track its evolution continuously.

2. Two Hundred-Plus Papers as a Process of Rebuilding the Fundamentals

Xie read more than 200 but fewer than 300 papers, ultimately retaining more than 100; the effort consumed almost all of his evenings, weekends, and hotel stays on business trips over the past 2 years.
To re-enter the theoretical context, he used overseas materials to review linear algebra, probability theory, stochastic processes, and partial differential equations. Many formulas are not necessarily difficult; the real barrier is often the notation system: “You just need to be able to read it.”
After spending a week reading directly in English, he realized: “I wasn’t really learning AI. I was learning English.” He switched to immersive translation, removing the language barrier from the primary task.

3. AI Learning AI Has Already Lowered the Cost of Entering the Literature

Immersive translation handles not only papers but also YouTube subtitles. As language models improve, Xie has seen translation quality steadily rise, sharply reducing language-level friction.
He sees ChatGPT, Claude, and Doubao as “the best teachers around”: difficult questions can be cross-checked across multiple models, helping identify hallucinations while making the gap between frontier and ordinary models on academic questions immediately visible.
Claude, Artifacts, and similar tools can turn abstract principles into visual web pages. With an AI browser such as Dia, asking about highlighted text also keeps paper reading, concept lookup, and background follow-ups in one context.
Recommended materials include Andrew Ng’s courses, Li Hongyi’s 2025 generative AI course, Andrej Karpathy, Li Mu’s paper walkthroughs, 3Blue1Brown, and structured resources such as Dive into Deep Learning and Mathematics for Deep Learning.

4. Understanding Papers Requires Tracking History, People, and Paradigms Together

Xie found that the hardest part is not a particular formula but “not knowing why they did it this way.” Paper authors and product teams face concrete problems in specific eras; once the context disappears, methods can look arbitrary.
He therefore organizes the material along 3 lines: the historical line reconstructs the crisis of the time, the people line follows how researchers moved into industry, and the paradigm line asks whether a paper improves the mainstream or brings a fringe approach to center stage.
He further breaks the history of deep learning into 4 shared variables: chips carry compute, infra is the software system that controls the chips, and the other 2 are data and model structure. “We are not discovering the world from scratch”; innovation always accumulates on existing frameworks.

5. Brook Turned the GPU from a Graphics Device into a Programmable Compute Platform

The program treats the GeForce 256 in 1999 as the starting point for the modern GPU. In 2001, the GeForce 3 introduced programmable vertex and pixel shaders, allowing developers to control the graphics-rendering process.
Around 2003, GPU FLOPS surged past CPU FLOPS because of the GPU’s highly parallel structure. Scientific computing, data processing, and early neural networks similarly involved processing large numbers of pixels at once, creating a new question: how do you map tasks onto a GPU?
Brook for GPUs, released in 2004, abstracted away low-level graphics details into streams, kernels, and reductions, with a compiler and runtime to match. Developers could program in a C-like high-level language, with the paper’s authors claiming performance close to hand-written GPU code.
Ian Buck later joined Nvidia and became a founding member of CUDA. His adviser, Pat Hanrahan, worked across academia, Pixar, and Tableau. The team even linked multiple GPUs to run Doom at 4K, an early and intuitive demonstration of GPU computing.

6. AlexNet Broke the Rule of Hand-Crafted Features Through Triple Scaling

Fei-Fei Li’s team built ImageNet with roughly 14M labeled images across more than 20,000 categories. Yet the number of competition entrants fell from roughly 13 in 2010 to 7 in the second year and 6 in the third, suggesting the traditional approach was nearing its limit.
The prevailing view was that computer vision required humans to understand the task and hand-design the features; simply giving a neural network images and labels would “never solve image classification.” In 2010, the NEC team that included Kai-Fu Yu won precisely with traditional feature methods.
AlexNet scaled data, compute, and model size at the same time: roughly 62.5M parameters and 650,000 neurons, with Alex Krizhevsky solving the engineering problem of using 2 GPUs. By today’s standards it was only about 0.06B parameters; at the time it was enormous.
It beat existing approaches by more than 10 percentage points, a gap large enough to push the industry into a new paradigm. Alex, Ilya Sutskever, and Geoffrey Hinton later founded a company that Google acquired, and the team became an important predecessor to Google Brain.

7. Seq2Seq Freed Language Tasks from Fixed-Length Outputs

In 2014, machine translation was still dominated by SMT, relying on complex feature engineering and multiple subsystems. Vision, language, and speech each had their own communities and paradigms and had not yet been unified as the same kind of deep-learning problem.
Translation is fundamentally a variable-length sequence-to-variable-length sequence problem, but existing network structures struggled with that input-output relationship. Seq2Seq used an encoder to compress the input into a hidden state and a decoder to generate a sequence of another length, decoupling the 2 ends.
Ilya, Oriol Vinyals, Quoc Le, and others brought pure neural networks into machine translation. The same structure also covered speech recognition and question answering, while encoder-decoder became a familiar pattern in Transformer, Diffusion, and other models.

8. Attention Solved the Long-Sequence Bottleneck in Hidden States

A fixed hidden vector has limited capacity: it may hold a sentence, but a long passage will lose information. The longer the sequence, the harder it becomes for the model to reconnect an earlier “Xiaozhang” with a later “he.”
Bahdanau Attention no longer required the decoder to rely on a single compressed vector. It generated a representation for every input word and calculated the relevant weights when producing each target word, creating a context-aware vector.
The mechanism gave the network room to learn relationships between words; the specific relationships were discovered through the process of reducing loss. Two papers published in 2014 both outperformed statistical translation systems and laid the groundwork for “Attention Is All You Need.”

9. Knowledge Distillation Transferred Large-Model Capability to Low-Cost Models

As deep networks grew, training costs could be amortized over one run, but inference latency, deployment difficulty, and per-call costs accumulated. Distillation, introduced in 2015, became an auxiliary technology for the large-model paradigm.
The teacher model gives the student not only the final hard labels but the full probability distribution. The lower-probability “soft labels” encode subtle relationships between classes, allowing a smaller model to imitate the teacher’s generalization ability.
Xie stresses that collecting model outputs through an API is not complete distillation as defined by the paper because the probability distribution is unavailable. But incomplete fitting can still work, like repeatedly approximating a complex function with a rule-based function: “It gets more and more similar, and eventually almost the same.”

10. ResNet Made It Possible to Go Reliably “Deeper Than Deep”

The 2015 lesson was that bigger models were not always better. Once networks exceeded roughly 100 layers, performance began to decline, a phenomenon known as degradation; vanishing and exploding gradients also limited further scaling.
ResNet stopped learning the full function from X to Y directly. It learned the increment relative to X, then used a shortcut connection to produce F(X)+X. Reframing the task from “learn a complex target” to “learn how much to add or subtract from the existing result” made training easier.
The work by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun made networks with hundreds or even thousands of layers possible. The program says its citations are approaching 300,000; Transformer, Diffusion, and other models still widely use residual structures.

11. Transformer Was Selected by the RNN Bottleneck and GPU Parallelism

RNNs have an inherent sequential dependency: each step must wait for the previous one, making parallelism difficult. CNNs suit GPUs but require many convolutional layers to connect distant information, with long dependencies easily lost along the way.
This was the “hardware lottery”: algorithms naturally suited to the dominant hardware of the time were more likely to scale and become the mainline. RNN did not win the lottery; Transformer’s matrix operations matched GPUs extremely well.
In 2017, 8 Google authors asked an extremely simple question: if Attention works, can we keep only Attention and remove the RNN? The paper retained the encoder-decoder structure, while subsequent mainstream models evolved further into decoder-only architectures.
A structure that began with translation ultimately became the common foundation for language, vision, video, and even diffusion models. That unification was the result of modeling capability and compute efficiency being selected together.

12. Self-Attention Reduced Relationships at Arbitrary Positions to a Fixed Number of Steps

Self-Attention lets any token within the window calculate directly with any other token, without passing through multiple layers of propagation. In “The cat lies on the mat, and it is cute,” “it” can directly compare its relationship with “cat” and “mat.”
In implementation, embeddings are mapped to Q, K, and V; dot products and softmax produce attention weights, which are then used to weight V. The outputs of multiple heads are concatenated and passed to the decoder. Since the multiplications are independent, GPUs can execute them simultaneously.
Attention alone loses sequence order, so positional encoding and masks are required. Later, Su Jianlin improved rotary positional encoding. Modeling capacity, data scalability, and GPU parallelism became possible at the same time, clearing the way for “brute force works.”

13. Transformer Worked Before Humans Could Fully Explain Why

When Noam Shazeer was asked why the architecture worked, his answer was: “We provide no explanation; if it works, we attribute it to the mercy of God.”
Xie was initially puzzled by why certain latent spaces happened to be 64×64. When he asked researchers, the practical answer was often: “We tried it.” At this stage, neural networks looked like a black-box science that first recorded the movement of the stars and only later searched for a theory.
Since 2017, the architecture and hyperparameters of mainstream Transformers have changed surprisingly little. The industry can apply them at scale, yet still cannot fully explain why this particular combination works best.

14. AlphaGo Zero Opened Both Pure Reinforcement Learning and Test-Time Scaling

AlphaGo, which defeated Lee Sedol, still relied on human game records and hand-crafted concepts such as liberties and eyes. That imitation route had 2 ceilings: AI would struggle to exceed its teacher and could not solve new problems for which humans had no examples.
AlphaGo Zero knew only the board and the rules. It played itself from scratch without game records or Go concepts. Xie believes closed domains with explicit rules and clearly representable objectives are suitable for reinforcement learning without pretraining.
The program says it used fewer GPUs and surpassed AlphaGo Lee after 36 hours of training. Improving both performance and training efficiency made it a paradigm shift.
It also ran roughly 1,600 MCTS searches for every move. Without thinking, a Go model could not reliably beat top humans. This line of work later inspired o1’s test-time computing and echoed DeepSeek R1 Zero’s exploration of “what happens if you skip SFT?”

15. MoE Rewrote the Inference-Cost Curve Through Sparse Activation

The idea behind MoE dates back to the last century, but 2017 work pushed it back into focus: train a model with an enormous total parameter count, while activating only a small number of task-relevant experts for each inference.
The intuition comes from the brain’s specialization by region—thinking does not activate every area. The approach can absorb patterns from more data without making inference costs rise fully with total parameters.
The paper reported “more than a 1,000-fold increase in model capacity with only a slight loss in computational efficiency.” That explains why cost-sensitive teams choose MoE.
The trade-off is major training and engineering complexity. The program cites DeepSeek’s engineering advantage, says OpenAI may have moved to MoE relatively early, notes that Llama’s transition has not gone smoothly, and suggests MiniMax may have needed several attempts. Structural gains depend on infra to realize them.

16. CoT Showed That Much Reasoning Ability Already Exists but Has Not Been Activated

By 2022, scaling models produced inconsistent gains in arithmetic, common-sense, and symbolic reasoning. SFT on PhD-level human data was expensive, while conventional fine-tuning could not scale across every reasoning task.
Work by Jason Wei and others showed that simply displaying intermediate derivations, or prompting “Please think step by step,” could materially improve performance. The work looked like a simple blog post, yet made the industry realize that post-pretraining capability was not being fully used.
Xie sees this as an important starting point for post-training, shifting research away from endlessly expanding pretraining. The input side also began moving from prompt engineering toward context engineering.
Xie’s assessment of Jason Wei is that he is good at “finding the most important question of the era.”

17. LoRA Turned Fine-Tuning into a Pluggable Parameter Increment

Full fine-tuning of a large model is expensive and difficult, while storing a complete copy for every task consumes space. Small adapter layers save parameters but can add serial inference latency.
LoRA freezes the original model and trains 2 low-rank matrices alongside it to learn the parameter increment required by a task. At deployment, the increment can be merged into the original matrix, keeping performance close to full fine-tuning without adding inference latency.
Its shared philosophy with ResNet is “learn only the delta.” Although the paper was designed for Transformer, LoRA became more widely popular in Diffusion: it can train styles and characters, and even periodically write a person’s memories into a LoRA.

18. ReAct Added a Closed-Loop Interaction with the Real World

Russell and Norvig’s classic definition of an Agent is a system that senses its environment through sensors and changes it through actions. A language model can reason through CoT but remains disconnected from reality and prone to hallucinating inside closed text.
ReAct, published in October 2022, cycles through thought, action, and observation: think first, call a tool or affect the environment, observe the result, then revise the reasoning.
The work predates ChatGPT, showing that the core ideas behind Agents and context engineering were not a reaction to the later hype. Yao Shunyu chose a relatively obscure problem at the time, yet defined the direction the entire industry would soon follow.

19. Finding the Right Question Is Sometimes Scarcer Than Solving a Complex Problem

Xie distinguishes between 2 types of researchers: those who solve recognized difficult problems and those who identify early “the most important question of the era.” The formulas in CoT and ReAct are less complicated than PPO, but their impact is no smaller for that reason.
When Zhang Xiaojun asked whether research achievements are concentrated in youth, Xie said that is often the case because young researchers carry less historical baggage. Noam Shazeer is an important exception, having remained strong across multiple generations of technology, algorithms, and engineering.
Papers also provide a path to the authors themselves. Once you understand one roughly, you can watch talks by Danny Zhou, Jason Wei, and others and engage directly with “the smartest minds in the world.”

20. Full-Stack Culture Shortens the Lossy Chain from Idea to Code

Xie observes that many Bay Area managers still write code hands-on, and roles are less rigidly divided into frontend, backend, and algorithms. Chinese internet companies are more inclined toward fine-grained specialization in management and business.
He compares a PRD and prototype to lossy compression: the product manager compresses an idea into a document, then designers and engineers decompress it, inevitably losing information. If the originator writes the frontend directly, many intermediary steps can be removed.
Silicon Valley has a stronger full-stack culture, and some current AI startups are also hiring full-stack people. Xie believes teams can cultivate more full-stack engineers and product managers.

21. The Bitter Lesson Reduces Long-Term Winners to Search, Learning, and Compute Scaling

In a 2018 lecture, Richard Sutton drew a compute-per-dollar curve: the amount of compute available per unit cost has grown exponentially over the long term, the underlying variable that has recurred throughout 70 years of AI.
His “bitter lesson” is that general methods that exploit computation ultimately work best. Encoding human knowledge into systems is usually effective and satisfying in the short term, but over time it tends to plateau and block paths that scale better.
The most jarring line is: “Building systems in the way we think is right will ultimately fail in the long run.” Breakthroughs often come from the opposite direction—letting search and learning expand with available compute.
What should be built in is the primitive method that discovers patterns, not conclusions humans already possess about space, objects, or multi-agent systems: “We need agents that are capable of discovering things like we are, not agents loaded with things we have already discovered.”

22. Hand-Crafted Features Work in the Short Term, Until Compute Crosses Another Order of Magnitude

Xie does not interpret the Bitter Lesson as a reason to stop engineering today: “There are times when hand-crafted features are useful, until compute grows by another order of magnitude and completely defeats them.”
Agent workflows, RAG trigger rules, and the engineering orchestration in Cursor or Claude Code all involve injecting human-designed structure. Claude Code’s system prompt even tells the model to reason directly for calculations under 6 digits and call Python or another tool for more complex computation.
Zhang Xiaojun’s question points to the practical contradiction: users care only whether something works, not whether the method is “pure.” Industrial teams must balance current performance with long-term generality rather than treating research aesthetics as a product objective.

23. ZeRO Let Model Size Scale with Total Cluster Memory

Around 2019, language models moved from the hundred-million range toward hundreds of billions of parameters, while per-GPU memory rose only from roughly 16GB to 80GB. Parameters grew nearly 1,000-fold, hardware capacity did not keep pace, and a single card could no longer hold the model.
ZeRO optimized and dynamically managed training within a data-parallel framework, removing as much memory redundancy as possible and making trainable model size approximately proportional to total cluster memory.
The work became part of Microsoft’s open-source DeepSpeed and supported OpenAI’s early expansion of large models. It represents the third generation of training infra: the question was no longer simply “can we use more GPUs?” but whether thousands of GPUs could complete one training task together.

24. Scaling Law Turned Large-Model Training from a Gamble into an Extrapolatable Experiment

OpenAI’s experiments found stable log-linear relationships between test loss and compute, data, and parameter count. As long as the other 2 variables are not bottlenecks, increasing any one of them can predictably reduce loss.
Teams could therefore test data, architecture, and training recipes on small models before extrapolating to the largest scale, rather than spending months and enormous budgets on every idea.
The real value was reducing the risk of opening a “mystery box” only after a large training run ended. Scaling Law is not merely “bigger models are better,” but a method for experimental trade-offs and final training decisions.

25. Chinchilla Corrected Scaling from “Maximum Parameters” to “Compute Optimal”

OpenAI’s early conclusion favored training the largest possible model under a fixed compute budget, even if that meant stopping early. DeepMind pushed back that this left many models underfed on data.
The Chinchilla approach argues that parameter count and training-token count should scale proportionally: double the parameters and double the data. A smaller model trained on more data may achieve lower loss than a larger model trained on less.
A small model can also be deliberately “overtrained,” going past the best training cost-efficiency point in exchange for cheaper long-term inference. Training cost is paid once; inference cost is paid every time, so their optima differ.

26. Stable Loss Curves Reveal a Training Team’s Real Engineering Capability

Large-model training can suffer a “loss explosion,” so teams must save checkpoints continuously and roll back when something breaks. The more GPUs involved, the more important stability and fault recovery become.
The program uses Llama’s training as an example, saying failures may occur once every 2 hours during training, with researchers keeping their phones on and sometimes waking up in the middle of the night to restore the job.
By contrast, it says DeepSeek’s training curve declined steadily, with few crashes over several months. For investors, GPU count is only nominal capacity; stable training determines actual throughput.

27. LAION-5B Showed That the Open-Source Community Could Build Foundation-Level Multimodal Data

OpenAI released the CLIP model but not its 400M image-text pairs. Christoph Schuhmann, the German high-school physics and computer-science teacher who initiated LAION, believed that if foundational data were monopolized by a few giants, the direction of AI could also be controlled by a few companies.
The community used CLIP to filter Common Crawl and build roughly 5B image-text pairs, several orders of magnitude beyond ImageNet’s roughly 14M images.
LAION-5B became an important foundation for Diffusion training. Its significance was not only scale, but proof that open-source organizations, general-purpose crawlers, and model-based filters could be combined into a scalable data-production line.

28. NSFW Data Exposed the Conflict Between Capability Training and Output Governance

LAION did not simply delete NSFW images; it labeled them and let trainers decide. Xie explains that removing them entirely could weaken the model’s understanding of human anatomy.
The analogy is a painter learning anatomy: to draw clothing and posture well, the model still needs to understand the body. The program argues that a multimodal model can retain the capability during training and restrict its use at output; “cannot generate” and “never learned” are not the same safety strategy.

29. RefinedWeb Challenged the Monopoly of Human-Curated Corpora with Automated Cleaning

After GPT emerged, high-quality books, papers, and social-media corpora became increasingly concentrated in large companies. The industry briefly assumed that carefully curated datasets were an irreplaceable prerequisite for strong models.
RefinedWeb used only Common Crawl, applying cleaning and quality rules to produce a large dataset that could still train models matching or exceeding those built on curated human corpora.
The key shift is that data production can scale. People do not need to organize every piece of knowledge manually; cleaning experience can be encoded into a compute pipeline. Xie also recommends Hugging Face’s series on data cleaning and infra as a practical starting point.

30. MegaScale Publicly Described the First Systematic Method for 10,000-GPU Single-Job Training

ByteDance released MegaScale in 2024. According to the program, it was likely the first public paper explaining how to use roughly 10,000 GPUs for a single training job; previously, most of that experience was held internally by a few companies.
Ten thousand GPUs cannot simply be connected and expected to work. More cards do not necessarily mean faster training, and a physical failure or electromagnetic bit flip on one card can contaminate the global computation.
The system therefore needs deep observability: monitor training, visualize metrics, diagnose failures automatically, identify bad cards, remove and replace them, then decide whether to restart or repair. Both efficiency and stability must be designed in from the start.

31. DeepSeek Demonstrated Co-Design Across Algorithms, Bandwidth, and Parallelism

Xie uses DeepSeek-V3 and the H800 as an example: with hardware bandwidth constrained, the team designed tensor-parallel splits around the bandwidth boundary so model communication would remain within tolerable limits.
The ideal is for data transfer to happen in parallel with computation and arrive exactly as computation finishes. Otherwise compute units wait for communication, or communication units wait for compute, leaving expensive GPUs idle.
MFU in current training may be only 50% or lower, and 50% is already strong in practice. Xie uses this to suggest that roughly 3,000 DeepSeek GPUs may have achieved something comparable to Meta’s roughly 20,000, illustrating how co-design can amplify capital efficiency.

32. Word2Vec First Gave Words Semantics in Continuous Space

Words had previously been treated as isolated atoms, making it difficult to distinguish “apple” as a phone brand from “apple” as a fruit. Word2Vec used machine learning to find continuous vectors for words, allowing semantic relationships to be expressed through distance and direction.
The classic example is king−man+woman landing near queen; the relationship between Italy and Rome is similarly close to that between France and Paris. Vectors represented not only words but latent directions such as gender and capital-city relationships.
These representations improved search and downstream NLP tasks and became the foundation for embeddings. But Xie notes that treating words as the smallest unit still carried a human prior; tokenization later moved toward more abstract machine representations.

33. Tokens Explain Many Counterintuitive Capability Boundaries in Language Models

Models such as GPT-4 have roughly 100,000 tokens; during training, the model sees tokens and their vectors, not the letters, words, or Chinese characters understood by humans.
That is why “How many r’s are in strawberry?” can stump a model: the word may be split into several composite tokens, so the model never observes it letter by letter. The internal codename for o1 was Strawberry.
Today’s embeddings are no longer pre-specified by Word2Vec but learned jointly by the model and the task. The same “apple” can occupy many more semantic dimensions, although humans increasingly struggle to interpret the coordinates directly.

34. Google Neural Machine Translation Brought the Academic Paradigm into Production

Neural translation methods had appeared in academia by 2014, but Google’s online system still relied mainly on statistical and phrase-based methods because NMT was slow, weak on rare words, could omit translations, and was not yet reliable enough for production.
Systems by Yonghui Wu, Quoc Le, Oriol Vinyals, Jeff Dean, and others integrated ResNet, encoder-decoder, Seq2Seq, Attention, low-precision computation, data parallelism, and model parallelism, becoming a synthesis of engineering advances.

35. GPT-1 Established the New Paradigm of Generative Pretraining Plus Fine-Tuning

In 2018, NLP models usually depended on task-specific labels and dedicated training, limiting their coverage. Vision already had a general path through pretrained transfer, but language lacked an equivalent route.
GPT-1 learned language patterns through unsupervised pretraining and then adapted to tasks through supervised fine-tuning. Architecturally, it used a decoder-only Transformer with next-token prediction as its objective.
It had roughly 0.1B parameters, 12 layers, and about 5GB of BookCorpus training data. Its scale was tiny; what mattered was the “Generative Pre-training” methodology that later came to dominate LLMs.

36. BERT Once Beat GPT Across the Board and Temporarily Dominated NLP

BERT used bidirectional masked-language modeling to incorporate context from both sides of a word. The base model had roughly 0.11B parameters and the larger version roughly 0.34B; after release, it swept multiple benchmarks and performed better while models were still relatively small.
Its criticism of GPT was direct: left-to-right models could attend only to previous tokens, which might hurt tasks such as question answering that require bidirectional context.
BERT dominated NLP at the time. Xie believes BERT was better suited to language modeling and clearly defined tasks, while GPT was more like open-ended, implicit multitask learning. GPT’s generality became visible only after parameters, data, and task diversity expanded further.

37. GPT-2 Used Zero-Shot Signals to Show That the Next-Token Path Was Worth Pursuing

After BERT became mainstream, OpenAI did not switch to masked language modeling. It hypothesized that GPT-1’s shortcomings came from data that was too small and too homogeneous.
GPT-2 grew to roughly 1.5B parameters, 48 layers, and about 40GB of WebText. WebText was built by crawling Reddit and retaining content with relatively high upvote counts, in the hope of obtaining more diverse and higher-quality web text.
Without changing its architecture or fine-tuning separately for each NLP task, it already showed zero-shot generalization. The paper therefore defined the language model as an “unsupervised multitask learner,” giving OpenAI an extremely strong signal to continue.

38. GPT-3 Was an Organized, Financed, and Conviction-Driven All-In

About 1 month after GPT-2 was released, OpenAI restructured the company in March 2019. In July, Microsoft invested $1B and provided cloud-computing support, showing that the team was preparing for an order-of-magnitude expansion.
OpenAI subsequently cut or scaled back other directions, concentrating more people, money, and compute on GPT. Xie believes the team must have observed an important signal in GPT-2 scaling; otherwise it would be difficult to explain expanding the model by roughly 10x and then toward 100x while restructuring the organization.
The author count rose from roughly 6–8 people on GPT-2 to more than 30 on GPT-3, adding roles in data, infra, evaluation, and other areas. This was no longer a small research effort but a large engineering project.

39. GPT-3 Turned Context into the Task Interface for General-Purpose Models

GPT-3 reached 175B parameters and 96 layers, with roughly 570GB of training data including cleaned Common Crawl, books, and Wikipedia. Its parameter count was roughly 100x GPT-2’s.
It demonstrated in-context learning: users could provide rules, task descriptions, and a few examples in a prompt, and the model would perform a new task without updating its weights.
By Chinchilla standards, GPT-3 may have been undertrained. But its interface has continued to this day: models no longer need to be separately deployed for each task; the task itself is written into context.

40. InstructGPT Proved Alignment Could Overcome a Hundredfold Parameter Gap

Original GPT-3 generated false, toxic, or unhelpful content. It also failed to proactively refuse harmful tasks or clarify ambiguous intent; these problems did not disappear automatically with pretraining scale.
InstructGPT first performed SFT, then trained a reward model on human rankings, and finally optimized with RLHF. The team hired more than 40 contractors to build the data, listing their names in the acknowledgments; rankings could then be combined into many relative comparisons, amplifying the reward signal.
The result was a 1.3B model that outperformed the 175B GPT-3 on instruction following, despite a roughly 100x parameter gap. The industry saw that the leverage from post-training methods could exceed the leverage from simply adding parameters.
The model began shifting from “powerful but difficult to use” toward something closer to a standard assistant.

41. Tülu 3 Opened Up the Previously Closed Post-Training Recipe

By 2024, the industry knew that post-training mattered, but almost nobody had published a complete process. InstructGPT was one of the few references, while details on data, evaluation, and reinforcement learning remained highly scarce.
The Allen Institute for AI built Tülu 3 on Llama 3.1 and published the full post-training pipeline, including data, code, and training recipes, allowing outside teams to see the entire chain.
According to the program, it beat the original Llama 3 as well as Qwen2.5 and Mistral, and could be compared with GPT-4o. Its value was not only its leaderboard position but the reproducible post-training “anatomy” it provided.

42. Video Deep Learning Proved the Path but Exposed the Missing Temporal Model

In 2014, Andrej Karpathy and Fei-Fei Li’s team built a dataset of roughly 1M YouTube videos and studied video classification starting from an ImageNet-pretrained CNN.
Video contains at least roughly 24 frames per second, requiring far more compute than a single image, while adjacent frames differ little. When to fuse spatial and temporal information therefore became a long-running unresolved problem in multimodality.
The final multi-frame model improved accuracy only from 59.3% for a single frame to 60.9%, a gain of 1.6 points. It showed that the path was viable without fully exploiting video information, leaving a clear problem for later work.

43. Two-Stream Networks Added “Motion” as a Primitive Method for Video Models

Karen Simonyan and Andrew Zisserman split video into spatial and temporal streams: one viewed static images, while the other used differences between adjacent frames to form optical flow and explicitly provide information about how objects move.
Optical flow supplied motion information unavailable from a single frame. The model could recognize not only “what is in the scene” but also “how it is moving.”
The structure was the first to beat the best hand-crafted features on video understanding, which Xie calls the field’s “AlexNet moment.” The debate over early versus late fusion continues today, showing that multimodal unification remains unfinished.

44. GAN Used an Easy Discrimination Task to Force Difficult Generative Capability

At the time, people believed generation required understanding first, making image generation harder than judging “is this a flower?” GAN’s breakthrough was to rewrite the difficult task as a zero-sum game between a generator and a discriminator.
The generator acted like a counterfeit painter trying to fool the police; the discriminator only had to distinguish real from fake. Its discrimination loss became the generator’s training signal, using a solvable task to drive one that could not yet be solved directly.
Ian Goodfellow’s method dominated image generation for roughly 5–6 years, producing sharp samples with small, fast models. Its weakness was unstable training and a tendency toward mode collapse.

45. Early Diffusion Approximated the Complex Image World with a Computable Noise Distribution

The original Diffusion work in 2015 drew inspiration from physical diffusion: gradually add Gaussian noise to an image until its structure disappears completely, then learn the reverse process and restore the data step by step.
The goal was to combine flexibility with computability. Complex image distributions are difficult to handle directly, while Gaussian distributions are easy to calculate; many simple steps can be composed into a complex result.
The approach initially performed poorly and spent roughly 5 years in GAN’s shadow. Its inventor, Jascha Sohl-Dickstein, called it his “least-known work”; his background across physics, neuroscience, and Mars exploration also made for an unusually varied path.

46. DDPM Returned Diffusion to Center Stage by Predicting Noise

By 2020, GANs worked well but were unstable, VAEs were stable but blurry, and autoregressive generation was reasonably good but slow. Diffusion was still not mainstream.
DDPM stopped trying to learn the full reverse image and distribution parameters directly. It predicted only the noise added at each step, then subtracted that noise from the current image. Like ResNet, it converted a full target into an incremental target.
Combined with the image-friendly U-Net, Diffusion’s quality improved quickly and training became more stable. Over the next 2 years, extensive work reduced denoising from roughly 1,000 steps to roughly 10 while improving high-resolution efficiency.

47. ViT Made 2D Images Adapt to Sequence Models

In 2020, CNNs still dominated vision. Early Attention-CNN hybrids were neither standard enough nor able to fully exploit the GPU, libraries, and training ecosystem that Transformer had built.
ViT’s strategy was to cut an image into 16×16-pixel patches, number and flatten them into a sequence, add positional information, and feed the result into a standard Transformer. The paper’s title said it plainly: “An Image Is Worth 16×16 Words.”
These tokens are highly abstract; a small patch does not naturally correspond to an eye or petal. But the model receives pixel content and position and can discover compositional relationships on its own.
On small datasets, ViT underperformed CNNs. Only after large-scale pretraining did it surpass convolutional networks, becoming easier to scale by reusing the Transformer ecosystem. Its advantage was not a small-sample prior but a higher scaling ceiling.

48. CLIP Used Natural-Language Supervision to Build a Shared Vision-Text Space

OpenAI trained on roughly 400M image-text pairs through contrastive learning, eliminating reliance on expensive, narrow human labels such as ImageNet’s. Images and descriptions from the internet became the supervision signal.
The model mapped the text vector for “bird” and bird images to nearby locations, enabling text-to-image and image-to-text retrieval, as well as zero-shot classification, OCR, geolocation, and action recognition.
CLIP’s core output was not a single-purpose classifier but a general multimodal representation. Diffusion later used that representation to move from generating random images to generating images controlled by text.

49. Stable Diffusion Cut Costs and Increased Abstraction by Working in Latent Space

Early Diffusion operated directly in pixel space at resolutions such as 1080×720. The higher the resolution, the more expensive training and inference became; generating a single image could initially take as long as a day.
Stable Diffusion first compressed images into a lower-dimensional latent space such as 128×128, performed diffusion there, and decoded back into high-resolution pixels, reducing compute by at least 2 orders of magnitude.
Counterintuitively, compression did more than save compute; it could improve results. Xie compares it to the Feynman technique: “Reading a book thin” forces you to extract the essence. A low-dimensional space can similarly filter pixel noise while retaining the structure that “this is a rose.”

50. Cross-Attention and DiT Connected Text Control and Transformer Scaling to Generative Models

Stable Diffusion used CLIP’s text encoder as a condition, feeding it into the U-Net’s noise-prediction process through cross-attention. This made the model both efficient and controllable through prompts.
DiT replaced the U-Net backbone with a Transformer, directly inheriting parallel computation, training recipes, and software infrastructure. Its title, “Scalable Diffusion Models with Transformers,” makes the priority clear: scalability.
Among the authors, William Peebles later joined OpenAI with another co-author and worked on Sora. The program also mentions outside speculation that Sora uses many ideas and capabilities associated with DiT.
Zhang Xiaojun draws a broader lesson: architectural innovation needs hardware compatibility and may even have to wait for a new compute platform. Xie expects Transformer to continue entering image generation over the next 1–2 years, while the autoregressive route may return.

51. The History of Deep Learning Is a Symbiotic Chain of More Than 30 Works

Yao Shunyu once said it was difficult to name a single most important paper because every step depended on earlier tools, data, and questions. Xie initially wanted 10 papers to reconstruct the history, then found that at least more than 30 were needed to avoid breaking the chain.
AlexNet depended on ImageNet, GPUs, and CUDA; Transformer depended on Seq2Seq and Attention; Stable Diffusion depended on DDPM, latent space, and CLIP. Remove any of these links and the later “sudden breakthroughs” become impossible to explain.
Long-term bets mattered just as much. Years before Transformer appeared, Ilya believed in unsupervised pretraining and could only wait for the right architecture and compute to mature. Great researchers do not necessarily predict every detail; they hold their conviction until the critical conditions converge.

52. Once You Understand the Boundary, Waiting for Model Progress Can Also Be a Product Strategy

Xie went through a 2–3 month “plateau” during which he often fell asleep reading papers. Once he built the historical connections, he began to enjoy asking “why” and could tell whether a problem came from tokens, the training distribution, model capability, or product engineering.
He once gave Gemini an entire meeting ASR transcript full of errors, supplied domain context, and asked it to revise the text like a simultaneous interpreter. What previously took a full day of manual cleanup now took roughly 30 seconds at negligible cost, showing that capability often hides inside usage patterns nobody has tried.
A customer-service project showed the value of waiting. GPT-4o was capable but too slow, while DeepSeek-V3 and Qwen 2.5 fell short in roughly 20% of cases; building custom SFT would have taken 3–4 months, so the team waited 2–3 months and Qwen 3’s largest model solved the problem.
He has therefore made peace with imperfect products today and expects research such as sparse attention to expand context windows by 10x or even 100x while cutting inference costs by an order of magnitude. “I can be perfectly comfortable waiting” is not passivity; it is aligning the product-development cycle with the model cycle.

53. AI Is Reorganizing Careers Around Builders and Pushing the Model Interface Toward an Operating System

Xie’s 3-part advice to newcomers is to use AI frequently, learn engineering skills such as Python and JavaScript, and then read papers to master principles that remain stable over time. The more important identity of the future is the builder: “owning an end-to-end outcome.”
For functional specialists at large companies, he recommends crossing role boundaries deliberately: product people should do engineering, and engineers should understand products. Teams can also cultivate more full-stack engineers and product managers. A culture once concentrated mainly in Bay Area companies such as Google may now be easier to practice.
Xie’s personal view is that OpenAI is trying to build the next operating system: GPT and MCP could connect real-world applications and APIs, allowing users to access compute and operate the physical world through conversation. If it can form a super app or operating system, OpenAI may have a chance to stand alongside Google.
Google’s talent, engineering, and infra depth still exceeds what many expected. The issue may be that too much talent creates internal friction, though the company is gradually reorganizing and integrating. Small-team opportunities are also growing, but depend on the business model: Telegram can serve 1B users with roughly 40 people, while Meituan-style complex fulfillment is harder to compress in the same way. The industry remains extremely early, perhaps still in the minicomputer era.

Nvidia CTO Michael Kagan: Scaling Beyond Moore's Law to Million-GPU Clusters Market Overview October 28, 2025