01. Taking on 李彦宏 as Llama 3 Launches: What Is the Open-vs.-Closed-Source Debate in Foundation Models?

Summary

• Wang Tiezhen’s core view is not that open source must win: closed source sets open source’s capability ceiling, but the gap will not steadily widen. He agrees with the first half of 李彦宏’s argument: if a closed-source company has no product better than a free open-source model, its pricing model cannot last; but once both open- and closed-source models clear the GPT-4-level usability threshold, competition will shift across niches such as coding and writing, making it difficult for any single leaderboard to prove that one side has left the other “far behind.”

• Llama 3’s significance is that it puts near-GPT-4 capability in the community’s hands, not that enterprises will immediately deploy the largest 400B model. The released 8B and 70B models and the then-unreleased 400B model sit on different cost curves: within the same family, with the same training method and architecture, larger models are necessarily more capable and more knowledgeable, but inference is more expensive and slower, and ordinary developers may not even be able to run 70B. Wang’s view is that the industry will “scale up first, then make models smaller”; mass deployment is more likely to rely on models below 10B, including 2B, 4B and 8B models on edge devices.

• Open and closed source are not binary camps but a broad spectrum running from fully open training to open weights to closed-source services. A company can layer private data on an open-source model and charge for a closed-source product at the “last mile”; it can also release a smaller model for user acquisition while commercializing a more powerful version. Wang argues that sustainable open source usually needs a path to self-funding: “Only open-source projects that can ultimately convert into a business model can go further.”

• Open source’s commercial value can come from distribution, ecosystems and complementary assets, and it can also support licensing or service revenue. Alibaba can use open source to drive cloud compute and GPU consumption, hardware companies can use software releases to expand device demand, and Meta may use model ecosystems to supplement content and interaction in the metaverse. Open source can also function as a market strategy, technology brand and recruiting tool, so “free” does not mean there is no business model.

• Open source’s real contribution is to compress the cost of experimentation and collaboration: one person need not understand all the math, own a large GPU fleet or even work for a particular company to improve one part of the stack. Wang cites an analysis showing that large-model inference costs fell by roughly 100x over the past year, with much of the work driven by the open-source community. Projects can also be copied, dissected and developed into new projects, allowing A and B technical paths that a company would have to choose between to evolve in parallel in the community. “Everyone stands on someone else’s shoulders” (「每个人都站在别人的肩膀上」) is why he believes open science accelerates innovation.

• Open weights are not the same as full open source, and Llama is closer to open access. Full open source would disclose the dataset, cleaning methods, training and inference code, model architecture, logs and weights; releasing only the weights is like providing an executable .exe without the source code. Developers can still fine-tune, merge models, quantize them or extend the context window, but they cannot fully reproduce how the model was trained.

• The safest path for a founder is to validate demand with the strongest closed-source model first, then decide whether to migrate to open source. Wang recommends testing prompting with models such as Claude 3 Opus; if prompting solves the problem, there is no need to rush into fine-tuning, since prompt costs are far lower and a product may launch within a week. Once real queries accumulate, the team can organize the data for SFT and private deployment to reduce costs and protect privacy. Most startups may be able to afford SFT, while continuous pretraining can cost more than 10x as much and requires knowledge of the original developer’s training-data mix and other details.

• Compute remains the hard constraint behind the open-source ecosystem, and distributed collaboration will not eliminate talent hubs. Llama 3 was trained, according to the episode, on 24,000 GPUs and 15T tokens; purely community-funded grants may reach only the hundreds-of-GPUs scale, with thousand-GPU support rare. At larger scales, fault replacement, checkpointing and automatic recovery become systems-engineering problems. Hugging Face can operate globally through remote work and asynchronous Slack collaboration, but Beijing, Silicon Valley and Paris still form hubs through talent density: “Talent hubs will always exist” (「人才高地一定会存在」).

Deep dive

1. Wang Tiezhen Found Open Source’s Appeal in Big-Company Reorgs

• Wang Tiezhen spent nearly nine years at Google, working on TensorFlow early in his tenure before moving to Hugging Face in 2022. What he wanted to recover was the collaborative experience of the previous AI cycle, when “doing open source really well was incredibly satisfying.”

• Engineers at large companies are rigorously selected but remain constrained by organizational boundaries: an employee in Group A cannot freely help Group B because the manager may object, while the other team may feel that someone is “taking its business.” Repeated reorgs can also force people to hand over projects they love—“it felt like I had a treasure”—only for those projects to die afterward.

• After an open-source project is released, users report bugs, reproduce issues and say thanks in GitHub issues. For Wang, that feedback makes it tangible how a developer’s work helps other people: “You feel that you are constantly needed.”

2. Transformers Showed How a Problem Can Become a Public Production Line

• After Hugging Face saw Google’s paper and its TensorFlow implementation, Thomas Wolf built a PyTorch version. It gradually absorbed more models and evolved into Transformers, turning a limited initial implementation into a public foundation that the community could keep extending.

• The standard workflow starts with an issue: someone requests a model or reproduces a bug; a community member turns the idea into a fuller PR; maintainers inspect the code, discuss the issue and merge it into the main branch; the next release then delivers it to every user.

• The difference from closed-source software is not just whether it is free to use. A closed-source user facing a problem can only file a support ticket, and if the issue does not affect the vendor, it may sit in the queue indefinitely; an open-source user can inspect the source code and submit a fix directly.

• Hugging Face itself is more like a Hub, with each repo maintained by its own group. Libraries such as Transformers and Diffusers may have hundreds or even thousands of contributors, but only a dozen or so may be actively contributing on any given day.

3. Subtitle Groups and BBSs Had Already Demonstrated Open-Source Governance

• Wang argues that the kernels behind Android, Chrome, Firefox and Edge, along with Web3 projects, have long brought open source into ordinary people’s lives. Chinese internet subtitle groups and BBSs can likewise be viewed as early open-source collaboration that simply did not take place on GitHub.

• He cites his own participation in translating subtitles for The Intouchables: a forum recruited volunteers; members claimed tasks based on interest and availability; stronger contributors reviewed and consolidated the work; an encoder burned the SRT subtitles into the film before handing the release to distribution channels. The key was how the subtitle files were produced, modified and reviewed; the video files themselves could raise copyright issues.

• Mapped onto a software community, ordinary viewers are users, translators are contributors, reviewers are maintainers and the encoder corresponds to the release role. 卫诗婕 described a maintainer as “the team lead who checks and marks homework,” a comparison Wang endorsed.

4. Public, Forkable Code Lets Communities Bet on A and B Simultaneously

• 卫诗婕 asked why ego and self-interest seemed to disappear in open-source communities. Wang’s correction was that they had not disappeared; rather, the work was transparent, and early-stage projects treated every contributor as a scarce resource, making participation easier.

• A contributor does not have to give code only to the original project. They can copy it, take it apart and build a project of their own. Companies generally have to allocate resources and choose between technology A and technology B; community members acting out of interest can advance both paths at the same time.

• GitHub also creates a layer of identity separation: unless developers disclose it, others may not know their real names or employers. Mature projects are not detached from corporations, however; Linux contributors and maintainers include employees of Google, Microsoft, Tencent and Alibaba.

5. Once Infrastructure Matured, Open Source Was No Longer Just for Geeks

• Wang agrees that early open source often lagged the most advanced closed-source technology, but the gap also reflected the era. In Linux’s early days, few people could access the code; high-speed home internet, Git and GitHub-style version control did not exist; and the infrastructure for efficient large-scale collaboration was not yet in place.

• Once networks, code hosting and ready-made scaffolding became routine, open and closed source could both develop quickly if they started at the same time. Developers no longer need to build the entire system from scratch; they can continue iterating on existing components.

• Wang even predicts that “everyone may be a developer in the future.” Someone who does not know a programming language can ask ChatGPT to generate one-off code. The best text-to-image practitioners may come from art and design, while the best large-model novelists may come from the humanities; domain expertise will matter more than the traditional identity of a programmer.

6. Open Science Spreads the Cost of Innovation Across the World

• 卫诗婕 cited the Hugging Face founders’ view that if Google had not shared Attention Is All You Need and BERT, and if the original diffusion paper had not been released openly, the industry might have needed another 30-50 years to reach the level it had reached at the time. Wang believes the basic argument holds.

• He offered an even more extreme aerospace analogy: without patent barriers around aerospace knowledge, and with information and supply-chain materials made public, it might take only 7-8 years to land on the Moon, colonize it, extract resources and then land on Mars. This is his belief in “one plus one being greater than two,” not a firm forecast.

• The experience supporting that belief is an analysis showing that large-model inference costs fell by roughly 100x over the past year. Contributors do not need to understand all the mathematics of deep learning, own a large GPU fleet or belong to a designated company; they only need to improve the small piece they understand best.

7. Open Source’s Catch-Up Was a Product of Corporate Strategy, Not Technical Destiny

• Wang pushes back on the implied premise that open source only began closing the gap with closed source in the past eight months. Before GPT-3, OpenAI released GPT-2 on day one as the most advanced text-generation model of its time; the later shift to closed source was primarily a change in business model and corporate strategy.

• His competitive framework is straightforward: the leader does not need to open-source because users will come naturally; the second-place player may believe it can become number one; and the third player, unable to win head-on, may use open source to flank and leapfrog the leaders in exchange for influence and visibility.

• Companies may also choose openness for reasons of control, security and privacy. A company may not want to send internal information to ChatGPT, but it can deploy a model with controlled code and weights in a private environment. Meta’s, Alibaba’s and other companies’ open-source choices therefore cannot be explained by model scores alone.

8. The Li彦宏–傅盛 Debate Mixed Values With Commercial Choices

• 李彦宏 said at Baidu’s Create conference that open-source models would be left “far behind” by closed source, drawing public pushback from 傅盛 and other founders. Llama 3 was released soon afterward, and 傅盛 used its performance—near or above that of some closed-source models—to make the long-distance argument look like an instant “gotcha.”

• Wang did not reduce the outcome to one side winning. Before ChatGPT appeared, Baidu had built substantial open-source influence through PaddlePaddle and PaddleNLP; its later reduction in investment looked more like a decision that closed source could give Baidu an advantage than a sudden discovery that open source was technically ineffective.

• OpenAI moved from openness toward closed source while still releasing open-source models such as Whisper; Meta has made extensive work public while retaining internal models. Wang’s conclusion is that the two mechanisms “can convert into each other,” so companies do not need to make a permanent, pure-camp choice.

9. Open Source to Closed Source Is a Spectrum, With Monetization in the Last Mile

• At the most open end of the spectrum, everything from data, cleaning, architecture and training through inference is public. Wang cited Hugging Face’s recent cleaning and open-sourcing of a 15T English corpus from Common Crawl; some training logs, code and final weights are also to be made public.

• Moving toward the closed end, a company can release only the weights while restricting commercial use and requiring a separately purchased license. It can also release a smaller model for users to try before selling access to a larger, higher-performing model. Open source is both the product and the customer-acquisition funnel.

• Universities and research institutions such as BAAI may prioritize open research and public welfare rather than near-term profit; companies generally need a commercial loop. Alibaba can use the model ecosystem to drive cloud compute and GPU demand, hardware companies can use open software to sell more devices, and Meta may use it to fill gaps in metaverse content and interaction.

• In the common models Wang has observed, open source can be a market strategy, a technology brand, a recruiting tool for top talent, or a way to support hardware and professional-services sales. The key question is not whether open source can make money, but which layer ultimately captures the value.

10. Closed Source May Hold the Ceiling Without Widening the Usability Gap

• Responding to 李彦宏’s specific claim, Wang drew a clear distinction: “I agree with the first sentence, but not the second.” He accepts that “closed source is open source’s ceiling,” but rejects the idea that the gap must widen over time.

• The commercial logic is survival. If a free open-source model is already better than a company’s paid model, customers have no reason to keep paying and the closed-source business becomes hard to sustain. Wang sees the normal strategy as releasing a second-best model while reserving the stronger version; otherwise, when customers ask for better performance, the company can only say, “This is already our best model.”

• The GPT-4 being used as the open-source benchmark at the time was no longer a newly trained model. The episode cited one account that GPT-4 had completed training two years earlier and had spent the intervening period in safety testing. Wang therefore leaves the question open: outsiders do not know whether OpenAI already has a better successor, while 卫诗婕 also listed the possibility of GPT-5 through GPT-8.

• The more important turning point is the “usability threshold.” Once both categories reach GPT-4-level capability, model marginal costs can keep rising while marginal benefits may not keep pace. Applications can also add knowledge through repeated calls and RAG, so competition may fragment into coding, fiction writing and other capabilities rather than resolve through a single overall leaderboard.

11. Open Source and Open Access Are Not the Same Thing

• The English-speaking community has increasingly distinguished open source from open access: the former opens the complete chain of model training and inference, while the latter provides access only to the trained weights. By Wang’s definition, BLOOM can be called open source, while Llama is closer to open access.

• The .exe analogy is the clearest one. An executable can be used, but it is difficult to add a button, change the text or replace the logo; source code allows a developer to recompile and redesign the system. Weights correspond to the executable output, while the training code and data pipeline are closer to source code.

• Fully open models do exist, but they serve a smaller audience. Even training a model below 10B may require tens to hundreds of GPUs and several T of training data; most hobbyist developers cannot afford that, while publishing every piece of “black magic” can also give competitors the playbook.

• Open access is not the same as being impossible to modify. Developers can import Transformers through the Hugging Face Hub to run inference and then fine-tune; they can use model merge to combine same-architecture models that specialize in coding and fiction, or quantize BF16 to INT4 for 4x gains in performance and storage efficiency, while expanding an 8K context window to 32K.

12. Founders Should Validate With Closed Source Before Encoding Real Demand Into a Model

• Hugging Face’s advice to founders without an AI background is to start with the strongest closed-source model available at the time, such as Claude 3 Opus. If prompting already solves the problem, its cost is far lower and there is no reason to fine-tune immediately: a closed-source model may allow the product to launch in “one week,” while an early-stage team still does not know what real users will ask.

• The queries accumulated after launch are the more valuable training material. The team can use them to build question-answer pairs, fine-tune an open-source model and deploy it in a private environment, reducing long-term costs and protecting data privacy instead of training a capability that no one ultimately needs.

• SFT primarily adjusts a model’s behavior and identity. When a customer-service model is asked “Who are you?”, “Who is your father?” or “Who is the company’s founder?”, it should answer with the company’s prescribed social identity rather than “I am Llama 3.” Depending on model size, training may require at least 1 A100 or H100, and in some cases 8 H100s.

• Wang believes most startups should be able to afford SFT on a small model; LLaMA Factory has put the process behind a UI, making it nearly as easy as using Word or Excel. The truly scarce resource is domain experts’ judgment: knowing whether a model’s answer is good and which question-answer pairs to prepare.

13. Continuous Pretraining and 400B Push Capability and Cost Higher Together

• If SFT is insufficient, a team can consider continuous pretraining. Wang says it may cost 10x as much or more and requires an understanding of the original developer’s training-data mix. His analogy is that SFT teaches a college student to remember a work identity, while continuous pretraining turns a mathematics student into a chemistry expert—roughly equivalent to earning another degree.

• The released Llama 3 models, 8B and 70B, and the 400B model that had not yet been released at the time serve different resource profiles. Within the same family, trained the same way and with the same architecture, the larger model is necessarily more capable and knowledgeable, but inference and fine-tuning cost more, and generating the next token is slower.

• An ordinary developer may not even be able to run 70B; Wang says his own computer can run only models below 10B. 70B already requires more GPUs and potentially a multi-machine cluster. Even if 400B approaches GPT-4, it will not be an immediate deployment choice for most companies; enterprises are more likely to consider models below 8B or around 10B.

• The episode says Llama 3 was trained on 24,000 GPUs and 15T tokens. Purely community-funded grants may reach only the hundreds-of-GPUs scale, with thousand-GPU support rare; once the fleet passes 1,000 GPUs, card failures, replacements, checkpointing, restarts and automatic fault avoidance become core infrastructure problems. At 10,000 or even 100,000 GPUs, the work becomes company-scale systems engineering.

14. 400B Is a Research Watershed, but Small Models Are Closer to the Production End State

• 卫诗婕 relayed NVIDIA scientist Jim Fan’s view that if the 400B Llama 3 reaches GPT-4-level performance, the community will have access to a heavyweight, usable model, potentially changing both research methods and the compute strategies of bootstrapped startups.

• Wang agrees on the potential significance. Even when companies primarily used open-source models in the past, they might still hand complex task decomposition to GPT-4; if a usable open-source model can take over that step, information control and privacy would improve. But he repeatedly cautions that “400B is, frankly, still too big”—a research watershed is not the same thing as a deployment wave.

• His view of technical evolution is to solve zero-to-one first and productionization second: initially relax constraints and scale the model to push capability higher, then make the model smaller to reduce cost.

• The edge case makes this even clearer. A fully offline phone cannot readily carry 400B; realistic candidates are closer to 8B, or even 4B and 2B. The near-term trend is not a blind race for parameters, but lower inference costs and faster responses while preserving capability.

15. Distributed Collaboration Will Not Flatten Talent Density in Beijing, Silicon Valley or Paris

• Hugging Face is a textbook distributed company: most employees and all 3 founders often work from home, communicating mainly through asynchronous Slack. For early AI companies, this structure saves on Silicon Valley office costs while allowing them to recruit globally for people who share their values and can build cohesion.

• Large enterprises face different constraints. Remote collaboration depends heavily on self-direction, while in organizations of 10,000 or even 100,000 people, many employees are accustomed to taking assigned tasks rather than choosing projects autonomously. Wang therefore sees Meta’s decision to bring core employees back together after the pandemic as understandable from a management perspective.

• Decentralization has not eliminated geographic clusters. 卫诗婕 noted that 10 of Llama’s 13 authors are based in Paris; Wang observes that China’s foundation-model talent is highly concentrated in Beijing and identifies Beijing, Silicon Valley and Paris as high-density nodes. His conclusion is categorical: “Talent hubs will always exist” (「人才高地一定会存在」).