游凯超 on vLLM, Open-Source Infra and Model Co-design
Summary
vLLM has grown from a paper prototype into a leading candidate for the de facto standard in AI inference. InfraRact is not a concept financed on a slide deck; it is the corporate vehicle for an open-source asset maintained over nearly 3 years by more than 2,000 contributors, and it has raised a $150M seed round. The 4 core founders turned down an offer from a top-tier tech company that would have paid each of them $20M a year, because they shared the same answer: “If the vLLM project fails, we’ll regret it for the rest of our lives.”
The company’s key asset is not proprietary code, but community leadership and the ability to coordinate. vLLM’s trademark belongs to the PyTorch Foundation, which also provides institutional safeguards for continued open sourcing; InfraRact supplies stable headcount, clusters, a legal entity able to sign NDAs, and technical direction. Its commercial experiments include endpoint service, BYOC, and ecosystem partnerships. The pricing logic is not selling engineers’ time, but charging for how many more tokens customers can produce and how much machine cost they can save.
游凯超’s core bet is that open-source models will win in the end. Models have to be used at scale, and “every token carries the model’s imprint”; once users accumulate enough interaction data, they may be able to train their own models. This is especially true for Coding Agent applications, where he estimates a company-level organization could build a usable data flywheel in about 1 month. Models may become commodities, with the real moat shifting to iteration speed, inference efficiency, and ecosystem position.
In the post-Moore’s Law era, AI’s ceiling is set not by algorithms alone but by the co-design of models, hardware, inference engines, and harnesses. 游凯超 considers DeepSeek the strongest team on this front: its algorithm researchers understand infra, while its infra team can extract the maximum from architectures such as MoE, FP8, NSA, and speculative decoding. But its self-built data centers and full-stack control are difficult for cloud-based teams to replicate.
Inference demand is shifting from short conversations to extended reasoning and multi-turn Agent interaction, with training and inference costs potentially reaching 1:1 and inference’s share still rising. On the supply side, PagedAttention, continuous batching, speculative decoding, and cluster optimization are needed to increase token output; on the demand side, stable prefixes, prefix caching, and better harnesses are needed to reduce compute. A demand explosion is not a reason to stop optimizing—it makes efficiency more urgent.
Tokens are not homogeneous commodities like electricity. DeepSeek tokens cannot be converted into Kimi tokens; models, inference implementations, and Agent harnesses all leave their imprint. vLLM’s ability to support 200-300 model architectures is valuable precisely because it absorbs this heterogeneity. The cost is having to continuously delete obsolete models and low-value features, concentrating resources on mainstream models, hardware, and workloads.
Coding Agent has made code nearly free and is beginning to erode open-source governance. Citing GitHub statistics, 游凯超 says vLLM became the project with the most active contributor base on GitHub in 2025, while also being flooded with résumé-padding junk PRs from training institutions and bots submitting 20 PRs in 10 minutes—what the team calls AI slop. The scarce resource in future communities will shift from code to trustworthy feedback, historical context, and maintainers with the authority to carry a roadmap over the long term.
Geopolitical specialization is shaping the technology path: 游凯超 observes that European and US models prioritize capability while Chinese models prioritize efficiency. China lacks high-end chips but has advantages in land and power; the US has more high-end compute, but the next bottleneck is power infrastructure. Chinese models carrying the open-source banner does not mean vLLM’s commercial center must remain in China: models can be open-sourced in China and deployed in North America, where InfraRact’s business is still primarily based.
Deep dive
1. The gaokao is only the admission gate; the real trajectory is determined by self-directed learning
游凯超 did not seriously engage with programming until 2015. He had no background in algorithm competitions in secondary school; his main track remained the high-school and college entrance exams, along with mathematics and physics competitions. Even a provincial first prize in physics did not materially change his admissions outcome.
His first major shock came from Geometer’s Sketchpad. Function graphs and physics simulations showed him that computers could turn abstract knowledge into observable objects. He then studied college physics and calculus on XuetangX, followed by Python through courses co-branded with edX.
He disliked repetitive, drill-based problem solving and preferred to spend exam-prep time learning new material. The gaokao happened to be relatively easy that year, so the accumulation did not show up fully in his score; he ultimately cleared Tsinghua’s cutoff by only 2-3 points. His conclusion was blunt: “Once you get into Tsinghua, you should forget that score.”
2. A hundred emails and 36 perfect grades were not résumé optimization
Outsiders often summarize his undergraduate years as “sending more than 100 emails in freshman year, reading more than 10,000 pages, and earning a 4.0 across 36 courses.” 游凯超 does not see this as résumé competition: with more than 10 classes, asking a professor a serious question once a week naturally adds up to more than 100 emails.
His notes being exhibited by the university archives was not a pre-set objective either. He says he never optimized for word count or volume; he simply enjoyed learning and needed to write down what he understood. The result was a by-product of learning, not a KPI worked backward from.
He still remembers a line from that period: “Tsinghua is a stage, but don’t be just a spectator.” But taking the stage did not mean beating other people; it meant pursuing one’s own growth. University life should not be reduced to watching “immortals battle it out.”
3. Tsinghua’s top scholarship brought recognition, not a ranking of the 10 best students
When he received Tsinghua’s top scholarship at the end of 2019, he was naturally excited. The practical benefits also included a broader network and exposure to more outstanding peers. But he objects to the mythology the media later attached to the award.
游凯超 notes that recipients are usually students in the first semester of senior year, roughly 3 years into college. Keeping an “immature young person” under the spotlight for too long can interfere with normal study and research. He approved when the university later stopped livestreaming the on-site defense and toned down the publicity: “That’s better.”
When the host asked whether the award represented the 10 strongest students in the university, he rejected the premise outright: each person pursues a different direction, and there is no single standard for who is strongest. An award can be recognition, but it should not become the single coordinate used to measure everyone.
4. Algorithmic academia slid from constructive debate to “submitting papers feels like drawing a lottery ticket”
游凯超 entered AI research around 2018. At the time, a conference received only a few hundred papers per cycle, and reviewers could still explain what was good and what was not. Authors and reviewers were engaged in a genuine discussion about how to improve the work.
After 2019 and 2020, submissions grew exponentially, while the number and quality of reviewers failed to keep pace with the community. “Whether you get in or not may no longer have much to do with the quality of the work itself.” He describes the condition as a collapse of norms.
It was not a single rupture, but a gradual loss of interest during his PhD. He accepts that “when the whole building is coming down, it isn’t something I can change by myself.” He stopped treating publication as a career objective and moved toward machine-learning systems, where progress was more verifiable.
5. He learned from 何恺明’s success that scale experiments are first and foremost a systems problem
In 游凯超’s mind, 何恺明 is the ceiling of algorithmic research. Whenever a new paper appeared on 何恺明’s homepage, his classmates reacted as if they had received “a new oracle.” After reading it, they often felt the design was inevitable—something resembling ideas they had discussed in group meetings.
The real gap was not just the idea, but the ability to run experiments at scale. Ordinary labs were “building a palace inside a snail shell” on toy datasets; 何恺明 could use systems and data to validate natural ideas at sufficient scale.
Once experiments are expanded, the decisive variables are usually no longer 5 different ways to weight cross entropy. They are the training system, compute, and data. The host summarized it as the end of the algorithmic blue ocean; 游凯超 agreed that the low-hanging fruit had been picked and scaling laws had reinforced the path of piling on compute and data.
6. In industry, the hardest problems are hardware caches and kernels—not a new loss function
During WACV 2022, 游凯超 pulled 张祥宇, then at Megvii, aside for a conversation lasting more than 1 hour. He was surprised that the object-detection specialist spent most of the time discussing how complex hardware constrains software in return, rather than proposing a new loss function.
Chips designed for vision workloads have multiple levels of cache, requiring engineers to manually plan how data moves between them. The team’s daily headaches were cuDNN, hardware adaptation, and a range of kernels—not another round of algorithmic parameter tweaks.
That conversation, the deterioration of academic review, and the need for large-scale experiments formed 3 strands in his career shift: algorithmic results increasingly depend on systems, while systems problems can be measured clearly and improved continuously.
7. A 1% algorithmic gain can be wiped out by the next generation of pretraining models
As an undergraduate, he worked on transfer learning, domain adaptation, and efficient learning in data-scarce settings, aiming to improve model generalization. The problem was that these methods were often icing on the cake, improving the accuracy of an existing task by only 1%-2%.
When he began his PhD in 2020, around the time GPT-3 was released—he qualified this as “it should have been around the time GPT-3 was released”—his lab lacked the compute for large-scale pretraining, so he shifted to large-model fine-tuning. After a year of algorithm work produced a 1%-2% gain, the market could switch to a larger pretrained model and generate more value than the entire year of research.
He realized that research dependent on someone else first defining the task, followed by a small incremental improvement, was difficult to scale. In the second half of his PhD, he gradually shifted toward machine-learning systems engineering; during a return to Berkeley in 2024, he moved into large-model inference.
8. Two Berkeley stints planted the shift from theory to real-world impact
In summer 2019, 游凯超 spent time at RISELab, where Michael I. Jordan was based, working around both machine-learning theory and systems research led by Ion Stoica. The former focused more on theoretical insight, game theory, and economics; the latter had a direct objective: real-world impact.
He once worried to Stoica: “If I build a machine-learning system and nobody uses it, have I wasted my time?” Stoica’s answer became a long-term anchor: “As long as you build good software, someone will use it.”
When Stoica returned to Tsinghua in 2023, 游凯超 brought up the advice from 4 years earlier and showed that he had been following it all along. He subsequently applied to join the lab, by then evolved into SkyLab. For him, the move reflected both circumstance and the broader rise of systems value.
9. 天授, LogME, and PyTorch contributions were successive exercises in moving toward systems
During the 2020 pandemic, he and his undergraduate classmate 翁嘉义 built the open-source reinforcement-learning system 天授, training agents on games including Atari. In 2022, he contributed algorithms and implementations to communities including OpenMMLab and PyTorch.
Work such as LogME already had a clear systems orientation. Rather than debating novelty, it could show how much it accelerated a method relative to alternatives and analyze the source of the speedup. “You can say my work isn’t novel, but you can’t say it’s useless.”
After ChatGPT launched, 翁嘉义, with whom he had once “played in the mud,” was already involved in training models. Their conclusion from talking was that data and algorithms depend heavily on the resources of a few companies, while systems can be studied at small scale in a lab and then expanded to industrial scale.
10. Systems research delivers more direct feedback than paper citations
After the shift, he judged work by how many people actually used it and whether it generated returns. GitHub stars, thank-you notes in issues, and users reporting how much efficiency improved were all closer to the objective than paper citations.
A systems optimization that delivers a 2x speedup can usually be decomposed and verified, and its transferability to other users assessed with reasonable confidence. An algorithmic improvement of 2%-3% in accuracy may fail to reproduce when the setting changes, or even reduce the metric in the opposite direction.
Papers therefore became a by-product: once a system had been used widely, he would write up the generalizable designs. At that point, reviewers were often users themselves, making publication easier.
11. PagedAttention had limited academic novelty, but vLLM’s value goes far beyond a paper
vLLM originated with PagedAttention, which borrowed virtual memory from operating systems to manage KV Cache for autoregressive generation through paging. Together with continuous batching, it became one of the core optimizations in early large-model inference.
游凯超 is candid that most systems researchers who studied inference carefully could have arrived at the idea; its novelty was “even on the low side.” When the paper was submitted to SOSP 2023, reviewers criticized it as too simple. It eventually scraped through with a low score because it was early and the experiments were thorough.
Today, vLLM is not a synonym for PagedAttention. It is a complete inference ecosystem spanning models, hardware, and service scheduling. The paper provided the starting point; the moat was built through years of subsequent adaptation and productionization.
12. Berkeley’s “5-year lab” tradition waits for research cycles and industry demand to resonate
游凯超 places vLLM in Berkeley’s lineage of heavy-duty software: BSD, RISC-V, RAID, and later Spark and Ray all began as student research projects before maintainers expanded them into industrial infrastructure.
He cites David Patterson’s paper How to Build a Bad Research Center: 5 years is enough for a research center, conveniently covering 1 PhD cohort and allowing strong students to focus on pushing a direction to usability.
AMP Lab, RISELab, and SkyLab share the same lineage. When a 5-year research cycle resonates with an industrial demand cycle, Spark or Ray can emerge. After seeing vLLM’s growth rate, he concluded: “The next project is this one.”
13. When 游凯超 joined, vLLM needed long-term engineers more than new ideas
When he applied for a Berkeley exchange in late 2023, he proactively proposed working on large-model inference. After joining in 2024, he became roughly the fourth major full-time maintainer. In his view, once a model is trained it must be inferred, and vLLM occupied a position with “a very promising future.”
The early core members included 李卓翰 and Woosuk, the main authors of the PagedAttention paper; Simon, who had open-source startup experience and later became CEO; and 游凯超, who was willing to devote himself fully to pure engineering.
The window for a suitable maintainer is narrow. Junior PhD students need to lead new projects and publish; people nearing graduation need jobs. He happened to have enough work to graduate, while also wanting hands-on system experience, allowing him to spend several consecutive months doing nothing but engineering.
14. Open-source teams are held together by meaning—and filtered by hardship
Asked why he would not work on a project unless he had initiated it himself, 游凯超 said he had no founder complex. The project was desperately short of people; as long as someone could contribute and had enthusiasm, “sincerity is enough.”
He divides human motivation into money, power, and meaning. His own criterion is whether the work is meaningful. The basic conditions for a computer professional are also relatively simple: food, housing, a computer, and an internet connection are enough to sustain long-term commitment.
He uses Song Lian’s Essay in Sending Yang to Dongyang to describe the team’s condition: “When one has sufficient joy within, one does not notice that one’s material comforts are inferior to others’.” Maintainers filtered through several years of shared hardship later became InfraRact’s founding team naturally.
15. v0 to v1 in 2024 turned an experimental prototype into a production-grade power grid
The main theme of 2024 was restructuring. The 2023 prototype had to become production-ready, and after surveying other inference engines, the team pushed through a comprehensive overhaul from v0 to v1 at year-end.
游凯超 uses the power system as an analogy. Users still simply “plug in,” but underneath, the system has shifted from thermal power to renewables. The operating behavior of different model-hardware combinations varies radically; the inference engine is responsible for hiding all that complexity.
The biggest challenge is compatibility. Models and hardware keep changing, while users still expect to change only 1 model parameter and run the system on different devices. v0 to v1 was merely the visible version number; vLLM has in reality been under continuous restructuring ever since.
16. China’s 2025 open-source explosion forced vLLM from single-machine optimization to cluster deployment
In 2024, open source in Europe and the US was still led mainly by Meta, Mistral, and others, with relatively limited coupling between model architectures and hardware. The main theme of 2025 was large-scale deployment, as models, hardware, and clusters all expanded rapidly.
游凯超 returned to China after his exchange ended in December 2024, just as DeepSeek-V3 launched. R1 arrived in early 2025 and went viral during the Spring Festival. The surge in Chinese open-source models exceeded the team’s expectations and visibly shifted the ecosystem’s center of gravity from Europe and the US toward China.
vLLM’s collaboration with DeepSeek actually began earlier, around DeepSeek-MoE, DeepSeek-V2, and DeepSeek-V3, with DeepSeek internally using vLLM and optimized variants as a base. The real shock after R1 was that DeepSeek’s inference team had already moved further ahead in engine optimization, forcing vLLM to learn from it in return.
17. China’s community was not a translated Slack; it was a new local network
vLLM was born in the US, with English-language Slack as its primary communication channel. Participants in China faced both language differences and the friction of using foreign software. Simply saying “the community is welcoming—join Slack” did not solve the problem.
游凯超 built a Chinese-language community from scratch, distributing information through WeChat public accounts, Xiaohongshu, Zhihu, and other channels, while establishing direct communication with Chinese companies to fill the original community’s gap in China.
DeepSeek, Kimi, and other companies have publicly said they use vLLM, but time zones and distance had kept communication limited. After returning to China, he visited them one by one, asking how they used it, where it hurt, and what they needed, then connected model companies, hardware vendors, and maintainers into a network.
18. Every new model launch has become a live pressure test for vLLM
游凯超 remembers July 2025 particularly vividly. With WAIC approaching, many companies released models at once, forcing maintainers to complete architecture adaptation, performance validation, and external support in an extremely short window.
By the time of the program, he cited DeepSeek-V4 and MiniMax-M3 as complex launches that had already taken place, while judging that another wave could come before WAIC. He explicitly retained the uncertainty: “I don’t know whether it will happen this year.”
For an open-source model, launching without support from a mainstream inference engine effectively means being unusable. Vendors therefore typically work with vLLM before launch; inference adaptation has become a critical path in model releases.
19. The PyTorch Foundation provides open-source continuity insurance, not an engineering team
By the end of 2024, the team was discussing long-term maintenance. If it could not find sustained support, the project might cool off. Before a company had been established, Ion Stoica pushed to donate vLLM to the PyTorch Foundation.
As an independent legal entity, the foundation can hold the trademark, accept donations, and provide legal support. It also institutionally guarantees that vLLM cannot be closed-source by a single company. Neither a university nor a 5-year lab has this kind of long-term legal continuity.
The foundation does not participate in day-to-day technical governance; code continues to be written by the community and the company. InfraRact reports progress roughly once a quarter. 游凯超 calls the foundation a “bumper”: the trademark belongs to the community, while technical direction and commercialization remain open to participants.
20. vLLM uses a “benevolent dictator,” not one-person-one-vote governance
The top decision-making group consists of Simon, Woosuk, 游凯超, 李卓翰, and some maintainers from Red Hat. They call themselves “benevolent dictators”: daily matters are handled through consensus where possible, but major directions require someone to make the final call.
Beneath them are more than 10 core maintainers responsible for key modules, followed by dozens of committers and a broader contributor base. NVIDIA, AMD, Red Hat, Amazon, Google, and other institutions contribute as hardware, software, or cloud vendors.
Maintainers do more than write code; they also coordinate large companies whose interests may conflict. The foundation provides the public governance framework, while the core team bears responsibility for technical continuity and prioritization.
21. Deleting beam search is the kind of unpopular decision governance requires
One of 游凯超’s tougher calls was removing beam search from the engine. Recommendation-system users continued to complain, but mainstream large-model inference had moved away from that path. Continuing to support it would only make model adaptation harder to maintain.
Similar trade-offs apply to small models. Models of a few hundred MB can still run, but vLLM no longer prioritizes extreme performance optimization for them; resources are being shifted decisively toward large models and large clusters.
The greatest responsibility created by more than 2,000 contributors is not accepting more requests, but deciding “what to do—and, more importantly, what not to do.” The larger the project, the higher the long-term cost of getting priorities wrong.
22. Coding Agent made code cheap and broke the open-source community’s assumption of goodwill
Citing GitHub’s 2025 statistics, 游凯超 says vLLM was the most active project on the platform by contributor activity. That was both an honor and a source of enormous review burden.
In 2025, in May, the team discovered training institutions using junk PRs to pad students’ résumés: merge 1 useless change and you could call yourself a vLLM contributor. More extreme bot accounts submitted 20 PRs in 10 minutes, content the team labeled AI slop.
Maintainers were forced to introduce authentication, favor contributions from recognized institutions, and ignore or block obvious bots. The old default assumption that contributors acted in good faith has been thoroughly broken.
23. The scarce resource in future open-source communities will be high-quality feedback, not code
As Coding Agent can rewrite implementations quickly, the cost of understanding an unfamiliar PR may exceed simply generating it again with complete context. “Code has become so cheap,” and the importance of any individual’s code contribution is falling.
游凯超 predicts that users will mainly submit bug reports and feature requests, while maintainers handle implementation. Value will shift toward problem descriptions, use cases, roadmap judgment, and long-term responsibility—not who wrote the specific lines of code.
He cites a similar view from Edward Yang, a core PyTorch maintainer: there is no longer enough capacity to review every PR; rather than read someone else’s generated implementation, it may be better to rewrite it oneself. OpenClaw-style “AI submits, AI reviews, AI merges” has appeared, but its long-term trajectory remains unknown.
24. AI can write 90% of the code, but not yet the final 10% of roadmap judgment
vLLM often maintains not the problems that exist today, but new models expected in the next 3 months and new hardware expected in 6-12 months. Code review must therefore carry unpublished roadmaps and the historical trade-offs behind them.
游凯超 believes AI can help cover 90% of the distance, but the top-level systems design and the final 10%—the “finishing touch”—still require a person. Those judgments depend on long-term context, not the text of a single issue.
When a founding team changes, partners, technical direction, and community trust all go through a period of dislocation. He even believes many projects struggle to continue developing well after their core leadership is replaced.
25. An open-source project must solve a real problem first, then find an organization to sustain it
游凯超’s first condition is straightforward: the project must solve a problem that enough people genuinely care about. If the demand is not important enough, even the most elegant community mechanisms will not make it grow.
The second condition is someone who can continuously organize resources; the third is a founding team that keeps pushing forward. He summarizes it as timing, place, and people: foundation, organization, and continuity are all essential.
OpenSSH is a cautionary example. Critical infrastructure used worldwide was maintained for years by very few people under severe strain. Only after a serious vulnerability threatened to leave the world “running naked” did people establish a foundation and commit resources. Infrastructure is often noticed only after it breaks.
26. The best vehicle for a neutral open-source project is a new company founded by its original maintainers
Ion Stoica’s logic is that Linux needed Red Hat, Kubernetes needed Google, PyTorch needed Meta, and Spark needed Databricks. vLLM, already of comparable scale, also needed a stable company behind it.
But allowing an existing tech giant to lead would import its interests. Meta, NVIDIA, and AMD cooperate while competing commercially, and other partners might not accept any one of them making the final call. Neutrality and convening power come from the team that maintained the project from the beginning.
A foundation can preserve open sourcing but does not manage day-to-day technology. A startup can organize people, machines, legal support, and commercialization while leaving the trademark and public assets with the community.
27. The company was forced into existence by 3 walls: NDAs, clusters, and long-term feature ownership
By 2025, the team needed to support new models and hardware before launch, but an open-source community is not a legal entity and cannot sign NDAs. Partners’ legal teams would also ask whether the participants changed each time and whether any competitor was involved.
The cluster problem was equally severe. In 2024, a single GPU or machine was enough for many optimizations; in 2025, much of the work had to be validated on clusters. Donating 1 machine was already a stretch for partners, let alone reliably providing an entire cluster.
Maintainers could only “beg everywhere”: spend 1 month persuading someone to lend machines, then fill the experiment schedule before the month expired. The partner might reclaim the machines early or reserve them for its own use on Mondays, Wednesdays, and Fridays, leaving Tuesdays, Thursdays, and Saturdays open. The team therefore acknowledged that it “didn’t do a good job in 2025.”
The coming and going of volunteers also could not guarantee that an important feature would be advanced continuously. 游凯超 repeatedly heard the same refrain: “The other side needs to sign an NDA again, and we can’t sign it. If only there were a company.” The company ultimately became infrastructure they had no choice but to build.
28. 游凯超 accepted a financially lean postdoc to wait for the team to start a company
After observing founders, investors, and the choice between entrepreneurship and employment in Silicon Valley in 2024, he concluded that the worst outcome of failure was merely earning less for a few years; not starting a company could leave vLLM unable to develop over the long term.
Before leaving Berkeley, he had already decided to participate in the startup. But the other core members were not yet ready, so he accepted a SkyLab postdoc offer and promised to wait for the team for another 2 years.
He also asked Michael I. Jordan how to choose between an outside job offering substantial money and the financially austere life of a postdoc. Jordan, drawing on his own experience, advised him to support vLLM over the long term, further reinforcing his decision.
29. Simon was suited to be CEO because he had already been acting as one in the community
Simon had helped build Anyscale and had also worked at Character.AI. He recognized early that vLLM needed a company, and had already demonstrated the community management, cross-party communication, and resource coordination required of a CEO.
He tried to secure people and machines for vLLM from inside Character.AI, but ultimately found that “you can’t rely on others”: a company with its own core business cannot indefinitely deploy employees to power an outside open-source project through goodwill.
游凯超 had no desire to compete for the CEO role. He considered Simon clearly better suited, and did not want to start a company alone while the core team and roles were incomplete.
30. Woosuk was not turning down an ordinary offer, but wealth that could have changed his life
The most hesitant was Woosuk, the principal author of the PagedAttention paper and later the CTO. He had faced an opportunity to lead infrastructure at xAI, spent time at Google DeepMind, and been invited by Thinking Machines Lab to lead infrastructure.
The team eventually put the question to him bluntly: “If you don’t start this company with us and make a lot of money, but vLLM fails 10 years from now, will you be happy or unhappy?” He did not answer immediately. After thinking for several days, he chose to return.
Just before the company was registered, a top-tier tech company’s top executive directly invited all 4 founders to join, offering each $20M a year. This time, the 4 founders rejected it without hesitation.
李卓翰 did not join, mainly because he had graduated in 2024 and subsequently worked at OpenAI and Meta, where he already had his own team and projects. The team judged that pulling him away would impose too much disruption on the organization he had built.
31. The $150M seed round was the result of investors waiting 2-3 years
The fundraising went more smoothly than the founders expected. In 2023, a16z had already provided vLLM with funding in the hundreds of thousands of dollars; in 2024, Sequoia and ZhenFund also made donations through the open-source community.
These institutions had already concluded that vLLM was important infrastructure and repeatedly said they would fully support the team if it started a company. Once the founders announced their decision, the investors’ response was essentially: “The money is ready. You’ve finally decided.”
The figure given at the start of the program was a $150M seed round. 游凯超 was unsure whether it was the largest in the history of global AI infra, but acknowledged that it was very large. The investment thesis is that every open-source model and every chip vendor is moving closer to vLLM, against a market measured in the tens of billions of dollars.
Even after the round closed, people were willing to reserve space with SAFE investments worth tens of millions of dollars. The team’s position was: “We’re not raising for the sake of raising.” The pace of funding must match the company’s growth and clearly defined uses.
32. Incorporation turned “please get this done” into plannable quarterly delivery
An open-source community can offer only best efforts. Participants have their own jobs, and plans often degrade into “please, please get this done this week.” A company has full-time staff and dedicated resources, allowing it to set quarterly goals and break them down into team and weekly tasks.
At the time of recording, InfraRact had more than 30 people, close to 40, and was still hiring. The 4 core founders have no rigid boundaries: whoever holds the relevant customer or technical context takes responsibility. They describe themselves as “4 bodies, 1 brain.”
Ion Stoica, as co-founder and advisor, participates in major strategic decisions. His experience building companies including Databricks and LMArena helps the team assess industry temperature and company stage, but he is not involved in daily execution.
33. The company did not spend the community’s trust; it supplied the resources the community lacked
In the past, partners were blocked from signing NDAs or providing compute and funding because an open-source community had no legal entity. After the company was formed, those legal and procurement barriers eased significantly.
For ordinary users, the key question is not who owns how much code or holds how much influence, but whether new models, new hardware, and new inference features will be supported on time. Stable hiring and cluster resources make those deliveries more reliable.
If a commercial customer’s requirements conflict with the community’s direction, 游凯超’s answer is: “We don’t take the project.” Demand currently far exceeds the team’s capacity, allowing it to reject work it does not want and select only opportunities aligned with the vLLM ecosystem.
34. InfraRact wants to sell token capacity, not engineers’ hours
The first model is endpoint service: the company owns the machines and software, and customers pay to use the service by token. The second is BYOC: customers provide the machines, while InfraRact provides the software and produces tokens inside the customer’s environment.
The third category is ecosystem-building partnerships with strategic customers. The principle remains value-based pricing: “We are not a technology outsourcing company.” Revenue should depend on how many more tokens are produced and how much machine cost is saved.
The 10-year vision is to make large-model inference “basically solved,” so that people think of vLLM and InfraRact whenever they think about inference. The internal milestone for the next financing is to prove out a commercial model that can scale.
Open sourcing by Chinese models does not mean the business must be based only in China. Models can run in North America, and InfraRact remains primarily focused on North American business, connecting Chinese model supply with overseas deployment demand.
35. Co-design is about adapting the generator to local hydrology
游凯超 compares hardware to natural resources such as wind, water, and sunlight; the model to a generator; and the inference engine to the power system. The end goal is to convert resources into tokens and distribute them to users.
The same flow of water calls for different engines in a turbulent river and a calm one. Model-infra co-design determines how many useful tokens can be produced from the same underlying natural resources.
While Moore’s Law was working, software could wait for general-purpose hardware to become faster every 2 years. Now that the general-purpose performance dividend is fading, new compute is increasingly specialized. An algorithm that fails to use those specialized units has missed the “hardware lottery.”
36. Transformer won the GPU lottery; Capsule Network did not
Transformer contains extensive parallelizable matrix multiplication and therefore matches GPU characteristics closely. 游凯超 agrees with the description that it “won the GPU lottery.”
Hinton’s Capsule Network may have been conceptually sound, but it was not GPU-friendly and did not see broad adoption. To become popular, it might have needed specialized hardware. He retains the conditional nature of that judgment rather than declaring the algorithm invalid.
Model architecture determines the upper bound of inference efficiency. If an architecture is inherently mismatched to hardware, no amount of systems engineering can save it. In practice, the hardware has already been built, so models usually move toward hardware rather than the reverse.
37. RoPE won not only on mathematical properties, but because it did not require changing FlashAttention
More than 100 positional-encoding schemes appeared after Transformer, but many intruded into the attention kernel. Methods such as ALiBi that require changes to the core implementation are difficult for ordinary teams to support as high-performance kernels evolve rapidly.
After FlashAttention solved the efficiency problem for long-sequence training, it became a key training primitive. Most users lack the ability to modify its internals, naturally filtering out many positional encodings that required changes inside attention.
RoPE combines strong relative-position properties with the ability to be injected into query and key independently, outside the attention kernel. It therefore became FlashAttention’s “ideal partner”—a textbook case of model design and infra resonating with each other.
The counterexample is MoE’s Expert Choice. It would make load balancing and fast execution easier for the infra team, but was unacceptable algorithmically. Co-design does not mean letting infra overpower algorithms; it means jointly constraining the feasible design space.
38. DeepSeek’s advantage is that its algorithm researchers can also write efficient infra
游凯超 rates DeepSeek’s infra as world-class. Whether it is absolutely number 1 is debatable, but he considers its model-infra co-design the best.
During the early exploration of DeepSeek MoE in 2024, an algorithm researcher wrote a relatively efficient coarse-grained MoE implementation. Fine-grained MoE was later deployed in its inference system ahead of others. Because the algorithm team understood some infra, it could explore a much larger design space.
NSA and DeepSpark also have a strong infra flavor. In the other direction, a strong inference team can quickly tell whether an algorithm can be implemented efficiently. The best arrangement is to have both teams work together, letting daily discussion and even casual meals create sustained exposure.
He speculates that this capability is linked to the extreme performance optimization developed during Phantom Quant’s quantization work. With its own data centers, the team could control every machine detail and optimize the full stack. The benefits are control and lower total cost; the drawback is limited short-term scalability, since a cloud-based team cannot simply replicate the setup.
39. DeepSpark’s value is not maximum novelty, but making speculative decoding work in practice
Autoregressive decoding generates 1 token at a time. Speculative decoding first guesses a sequence and then verifies it in a batch, turning several decoding steps into a short prefill. Routes such as EAGLE and MTP typically guess 3-5 tokens: short sequences with relatively high acceptance rates.
The DFlash route discussed on the program guesses many tokens at once—for example, 16 tokens. Its advantage is parallelism; its drawback is more errors and wasted verification compute. vLLM and NVIDIA have achieved more than 1,000 tokens per second for some models.
DeepSpark improves the process by estimating which guesses are more likely to be correct. If only the first 8 of 16 candidates have sufficient confidence, it verifies only those 8. 游凯超 considers the idea unsurprising; Tencent Hunyuan’s D-Cut and a team at Shanghai Jiao Tong University’s Domino have taken similar directions.
But novelty and the ability to execute robustly are different things. DeepSeek’s value lies in pushing inference optimization to the limit. Any speculative-decoding algorithm will struggle to see broad use without a high-quality engine implementation.
40. AI infra is not the same as having the money to build a large cluster
Asked why some highly visible teams with enormous clusters had not developed equivalent capabilities, 游凯超 distinguishes between “infra’s infra” and AI infra. The former means finding land, power, and capital, then turning on and delivering the machines.
Building a cluster is a heavy undertaking, but technical capability is not equivalent to capex. AI infra also requires understanding model workloads, communication, kernels, scheduling, and inference behavior. If the top executive does not understand these things, even expensive hires may not help if they are directed poorly.
Hence “good infra is not added; it is designed.” Resources determine whether a team can get started, but model-system co-design determines how many usable tokens those resources ultimately produce.
41. The more inference demand explodes, the less efficiency can be abandoned
2 years ago, model economics were still dominated by training costs. Today, training and inference may reach 1:1, with inference continuing to grow. Training is like developing a generator; inference is the continuous power generation that follows, so even a small efficiency gap compounds over long-term usage.
When compute supply is far below token demand, land, power, and chips cannot expand quickly. The only option is to improve generation efficiency. Designing a model for known hardware conditions may make R&D more complicated, but can materially reduce long-term operating costs.
游凯超 uses an attention head size of 1,024 as an extreme example. If an algorithm team suddenly adopts that structure without regard for hardware, the infra team may “pass out.” Teams working in isolation have, in his view, “no future at all.”
To the objection that test-time scaling is growing faster than optimization, he replies that power plants cannot stop optimizing because summer electricity demand is surging. Supply-side capacity increases and demand-side token savings are not in conflict; efficiency and scaling are compatible.
42. Tokens carry the model’s imprint, so inference will not become a simple power grid
Electricity can be converted into standardized voltage and frequency, but DeepSeek’s tokens cannot be converted into Kimi’s tokens. Model capability, behavioral style, inference implementation, and even the harness shape the final output.
“Every token carries the model’s imprint.” Some models are better at directing, others at execution; some fit Claude Code, others OpenClaw. Users are not consuming a standardized commodity, but heterogeneous intelligence supply.
vLLM supports roughly 200-300 model architectures, and its value lies in absorbing that complexity. The maintenance cost is continuously deleting unused architectures and features while focusing optimization on mainstream models, hardware, and workloads.
43. Differences between Chinese and US model architectures first reflect compute constraints
游凯超’s broad observation is that European and US models prioritize capability, while Chinese models prioritize efficiency. Compute is more abundant in Europe and the US, while high-end compute is scarce in China, so architecture choices naturally reflect different resource constraints.
For inference engines, the most important difference is often attention. Full attention has a smaller per-token state that only grows, making it suitable for PagedAttention; linear attention has a larger state that changes over time and requires different management.
Hybrid and cross-attention states are more complex, but have not yet become mainstream. From an infra perspective alone, a simple Llama-like structure is easiest to support. But generating “useless electricity” efficiently is still pointless; capability remains a constraint that cannot be discarded.
44. MoE creates 3 systems challenges, and DeepSeek has supplied an end-to-end engineering answer
The first is fine-grained experts. Finer experts may make a model more effective, but shrink the matrix dimensions and make it harder for GPUs to stay fully utilized. Algorithmic benefit must be traded off against kernel efficiency.
The second is dynamic routing: each token selects experts on the fly, while GPUs prefer a statically predetermined data flow. The third is scaling to 1T parameters while activating only a small fraction of them, requiring expert parallelism, data parallelism, and efficient communication to work together.
游凯超 cites DeepSeek’s 2025 suite of solutions: DeepEP handles expert-parallel communication, while DeepGEMM handles expert-parallel computation and selection. His conclusion is that “DeepSeek is indeed leading this MoE cycle.”
The main battleground after MoE remains attention. It is not yet clear whether linear attention or sparse attention is superior; both may work, and there is no need to force a choice between them.
45. Test-time scaling has evolved from “answer a few more times” into long-term Agent interaction
Early ensemble methods had multiple models, or the same model multiple times, answer a question and then aggregated the results. They scaled the number of attempts. o1-style extended reasoning scales the token count of a single output, with extreme tasks producing hundreds of thousands of tokens.
The more significant form this year has been Coding Agent interaction with an environment. Each round may involve only a few hundred tokens of reasoning, but the Agent continuously reads the environment, executes tools, and adjusts its strategy. What scales is the number of interactions and the runtime of the task.
Extended reasoning occupies resources for a long time within a single request and is expensive, serving mainly specialized needs such as scientific work. Multi-turn Agent interaction can enter everyday workflows, but requires historical state to be reused efficiently.
That makes prefix caching central: preserve as much existing state as possible so the next request can reuse it directly. If the model actively deletes the previous round’s reasoning, the model and inference engine will need to be optimized jointly.
46. A small harness change can invalidate every prefix cache
The Agent ecosystem is highly fragmented. Different harnesses create different workloads, while model architectures require different optimizations. The next phase will require harness-infra co-design.
Some frameworks fine-tune the set of callable tools on every turn, believing they are reducing the model’s burden. But tool definitions sit at the front of the request; once they change, every subsequent prefix becomes unusable, creating far greater inference cost.
An older version of ChatGPT wrote the current date into the system prompt, causing all caches to expire across the date boundary. A better approach is to expose the date as a tool and let the model query it only when needed. The rule is: frequently changing content does not belong at the front of the request.
More extreme frameworks write the time to the exact second into the request and schedule jobs on the hour, causing traffic to surge simultaneously. The team’s metaphor is “a group of little lobsters setting out together every hour to attack the moon.” Randomized staggering avoids the problem.
47. DeepSeek’s next lesson may be bringing the Agent harness into co-design
游凯超 notes that DeepSeek recently completed a large financing round and is hiring aggressively, with Agent framework design among its first priority roles. He therefore expects it to extend model-infra collaboration into the harness.
This is still an expectation, not a claim about a product already delivered: “I also believe their Agent framework and harness framework can reach the same level of co-design as their infra, and teach the community another lesson.”
For investors and product teams, the implication is that the competitive unit will no longer be a single-model benchmark, but the end-to-end efficiency of the model-inference-tool orchestration system.
48. FP8 shows how model architecture can actively capture the benefits of new chips
From A100’s Ampere to H100’s Hopper, one of the flagship selling points has been FP8 compute. 8-bit floating-point operations are 2x faster than 16-bit operations, but applying quantization globally can easily damage precision.
游凯超 says DeepSeek was probably the first team to make FP8 training work at scale. It uses block quantization for weights and per-channel quantization for activations, with a design that also tracks the next generation of MX quantization formats.
Block quantization still creates precision issues in matrix computation. DeepSeek’s solution is to use vector units to accumulate after the matrix multiplication, allowing the 2 types of on-chip units to work in parallel and balancing precision with efficiency.
The deeper trend is that Moore’s Law no longer provides general-purpose gains, so chips must specialize for domain workloads. Matrix multiplication has been proven simple and effective through more than 20 years of practice, leaving models under pressure to move toward matrix computation.
49. First principles help identify the main line amid 10 new technologies every month
游凯超 believes a credible team must first have people who truly understand the basics: why continuous batching works, how GPU benchmark results are affected by thermal power and clock frequency, how attention is computed under different batching schemes, and how FlashAttention works.
If someone can only patch existing code and measure a speed number without knowing where the gain comes from, performance optimization becomes a lottery ticket. Someone who knows only textbook formulas also cannot decide which of more than 100 context-extension methods is worth pursuing.
When studying GPU kernels, he started with the programming model and inferred that the driver layer should have a tool for debugging illegal memory accesses. After repeatedly pressing Nvidia, he found CUDA core dump, a long-existing capability little known in the AI infra community.
Applying this method more broadly helped the community solve many difficult problems. His methodology is: “Make bold conjectures, verify them carefully.” First principles clarify the main line, filter out noise, and show where to keep investing.
50. A million-token context may be enough; long-term memory may not require ever larger windows
游凯超’s only explicit technical hot take is that models designed for human-facing use may plateau at the million-token level rather than continuing to expand dramatically.
Specialized scientific tasks in biology and chemistry may require tens of millions or even hundreds of millions of tokens of context, but he qualifies this as applying to “specific fields” and says it “may” be the case. He does not generalize the forecast to every workload.
Long-horizon tasks and lifelong learning can be delegated to external memory, skill modules, sub-agents, and other tools. If the view is correct, today’s model architectures and infra paradigms could remain viable for a long time.
51. Systems repeatedly rewrite “algorithms cannot do this” as “this can be implemented precisely and efficiently”
In long-context research, many approximate Softmax Attention algorithms once appeared. FlashAttention showed that exact attention could also be computed efficiently, causing many approximate approaches to lose their value.
Early autoregressive decoding was considered too slow because it generated 1 token at a time. Continuous batching and PagedAttention lowered the cost, while speculative decoding transformed single-token decoding into segmented verification.
The usability of linear attention likewise depends on systems implementations such as chunk-parallel. 游凯超 agrees with David Patterson’s view that “in the post-Moore’s Law era, AI’s ceiling depends on systems.” Algorithms that systems can implement efficiently are the ones that have a chance to survive.
But co-design can solve only the problems of a given period. Whether 1T parameters should expand to 10T, whether context really needs to reach more than 10M tokens, and how AGI evolves are ultimately not only technical questions, but also questions of history.
52. China has carried the open-source-model banner, while the US still controls more high-end compute
游凯超 believes the leading US models—OpenAI, Anthropic, and Google Gemini—are primarily closed source. Mistral and others remain open source, but currently lack an open model at the same level. After the US shifted toward closed source, China “picked up the open-source banner.”
The infrastructure constraints are not a simple win-loss contest. China’s shortage of high-end chips makes domestic-card adaptation and extreme efficiency priorities, but land and power supply are relatively abundant. The US has more high-end chips, while its next infrastructure bottleneck is land and power.
He does not want to describe China and the US as simply competing, preferring to emphasize that each faces its own limitations and constraints. Those constraints will continue to shape model architectures, deployment methods, and engineering priorities.
53. Open-source models will win because models used at scale cannot stay hidden forever
游凯超’s core bet is: “I think open-source models will win in the end.” A nuclear plant can output electricity without teaching users how to build one; a model interacts continuously with users through tokens carrying its imprint and leaves behind training data.
This is especially true for Coding Agent applications, where companies can quickly accumulate tasks, feedback, and execution traces. He estimates that a company-level organization might collect enough data to start a training flywheel in about 1 month, but does not present that estimate as a universal rule.
Using GLM-4.5’s coding capability as an example, he believes its lead is likely related to accumulated data, although he does not know exactly how it was achieved. Leaders have an advantage, but it is difficult to defend that advantage indefinitely.
Models may ultimately become commodities. The moat is not a single capability lead, but whether a team can iterate quickly: “Nothing can be guarded for a year; you have to keep replacing the old with the new.”
54. Model vendors will build their own inference stacks, but still need a portable public layer
DeepSeek’s internal engine is powerful, but depends on complete control over data centers, clusters, and the software stack. Suddenly adding 10x more machines does not mean immediately producing 10x more tokens.
vLLM’s role is to provide a more general inference layer. Even when cloud resources impose constraints and prevent the most extreme full-stack control, it can deliver “pretty good” inference efficiency across a broad range of hardware.
This is the division between public infrastructure and internal vertical stacks: model companies optimize their own ceiling, while vLLM spreads models across more hardware, clouds, and customer environments.
55. vLLM is aiming to become the Linux of AI inference
游凯超 believes vLLM’s influence can already be described as “the Linux of AI inference.” Red Hat maintains both Linux and vLLM and packages the latter into its enterprise distribution, which he cites as industry validation.
Future enterprise deployments will involve not merely a “cold operating system,” but a system environment capable of running AI directly. The foundation’s public assets and InfraRact’s role as a key commercial maintainer together create a sustainable structure.
This positioning also explains why the team rejected a closed-source path. If they only wanted to build a closed product, the core members could simply have joined a large tech company. The premise of the startup is that the open-source inference engine itself has irreplaceable historical significance.
56. 游凯超’s ultimate bet is not on fate, but on verifiable long-term meaning
He does not claim to believe in fate, because “you first need a way to observe fate.” The 2 Berkeley experiences that changed his direction were, in his view, the result of both coincidence and the combined force of years of accumulation and broader historical trends.
Writing is another form of open source: putting technical insight into the public community also forces him to explain it clearly. “If you cannot write it down clearly, you probably don’t understand it well enough.” His training in mathematics and physics likewise makes him favor formal and verifiable expression.
He offers newcomers no shortcut, only the advice to start from first principles, clarify what they want, and accumulate over the long term. The hotter the industry and the more feverish the mindset, the rarer that choice becomes.
His ideal division of labor between people and AI still leaves humans in the loop: silicon intelligence provides large-scale synthesis and execution, while humans provide insight, inspiration, and fuller context. For now, his most important personal bet remains the unvarnished line: “Open-source models will win in the end.”