From Compute to Scale: The Future of AI Infrastructure | A Conversation with Google VP of Engineering Bill Jia

Summary

• Bill Jia’s core view is that AI is still at a very early stage, with demand set to grow 10x, 100x, and 1,000x, making it necessary to build infrastructure now. His investment logic: if demand surges and you haven’t invested while others can meet it, you’ll have a problem.
• In response to the host’s concern that Google’s balance sheet is negative and CapEx is out of step with revenue, Bill pointed to 3 ways to cut costs: smarter models that use fewer FLOPs, software optimization across networking, memory caching and compute, and cheaper hardware. He believes continued optimization can ultimately deliver positive cash flow; as long as optimization continues, there is no need to fear temporarily negative cash flow.
• On startup opportunities, he highlighted devices and agents that can work efficiently at scale, and said he would specifically invest in agents that optimize inference. There are many N×M combinations of models and hardware; optimizing them one by one with people is “very, very unscalable,” while agents can improve performance, reliability, and scalability more cheaply and at scale. Big tech will work on this too, but startups can sometimes move faster and do a better job.
• Power is the supply-side bottleneck: the host cited 40 GW of additional compute capacity next year, a figure Bill did not confirm, though he expects many gigawatts of data-center capacity to come online. Power generation currently relies mainly on natural gas, coal, and crude oil; solar and wind are not yet scalable or efficient enough, and locations able to supply this kind of power are limited. Hydrogen and nuclear are more realistic options, with space data centers as a moonshot; sites should be chosen for abundant energy.
• Bill expects inference to account for a larger share of compute over time; whether the split is 50/50 today is “hard to say,” and inference may already exceed training. If the split is 50/50 now, it could move to 40/60, 30/70, or 20/80 over time. Memory will remain scarce in the agentic era, while demand for CPUs, flash, and hard-drive storage will also rise.
• His key calls: TPUs and Nvidia are not in all-out competition but are learning from each other under different business models; unifying training and inference remains premature, and there is no need to combine pre-training, post-training, and inference in one system. The model landscape will include both frontier models and vertical models, with room for many winners.
• Safety risks are not confined to inference: Bill said models are black boxes whose behavior is hard to control because we do not understand why they work as they do. After training, debugging, and addressing safety issues, inference is “not too bad” for now; greater concern lies with newly trained models that may behave unpredictably during testing, evaluation, and in sandbox environments.

Deep dive

1. Demand growth supports continued investment; the key is lowering costs and reaching cash flow

• Bill’s fireside-chat recap set the tone: AI is “still very early,” and the technology will continue to advance rapidly over the next 1, 2, 3, 5, 10, and 20 years, with demand continuing to grow. The consensus is that building infrastructure for AI now is necessary.
• Host 曹卿云 noted that Google’s balance sheet is negative, CapEx is high, and the revenue has yet to materialize, and asked whether the issue was one of pacing or timing. Bill’s investment logic is that if demand will be enormous and surge in the future, investment should happen now; otherwise, when demand arrives, being unable to meet it while others can will be a major problem.
• There are 3 ways to lower costs: models can become more intelligent while using fewer FLOPs; software can be optimized across networking, memory caching, compute, and storage, reducing costs even when the model and hardware stay the same; and hardware itself can get cheaper. Over the past 1–2 years, performance per watt, performance per dollar, and dollars per unit of intelligence have all improved, gradually lowering the cost of AI training and inference.
• Bill believes continued optimization can eventually meet demand at lower cost and with better results, enabling positive cash flow. There is no need to fear temporarily negative cash flow, provided optimization continues. Making AI more affordable is not the end goal; the point is to accelerate innovation as much as possible.

2. Power is the near-term constraint; hydrogen and nuclear are more realistic, with space data centers a moonshot

• The host cited 40 GW of additional compute capacity next year and asked whether it could come online. Bill said he did not know whether the figure was 40 GW, 30 GW, or 5 GW, but expected many gigawatts of data-center compute capacity to come online next year. Power is the main constraint.
• Power generation currently relies mainly on natural gas, coal, and crude oil; solar and wind are not yet scalable or efficient enough. There are few places around the world that can supply this kind of power. Data centers consume a lot of electricity, generate a lot of heat, and create noise, raising concerns among nearby communities. Some US states will say, “We don’t welcome you building AI data centers in our state.” Bill sees power as a near-term bottleneck at both the technical and political levels, and called on technology leaders and regulators to work together on a solution.
• Google’s position is to invest and work with the US and other governments to solve the power problem. The long-term moonshot is data centers in space; more realistic options are hydrogen and nuclear power, which Bill believes can be cost-efficient, energy-efficient, and green. The logic for choosing a site is like oil extraction: “There’s no point drilling for oil where there’s no oil.” Data centers should be built where energy is available.

3. Inference’s share of compute will rise over time; vertical applications and general-purpose models will coexist

• Asked whether industry compute is already split 50/50 between training and inference, Bill said “it’s hard to say”; inference may already exceed training. Recursive self-improvement, or reinforcement learning, will add to training compute, but inference will take an increasing share over the long term. Training requires a one-time or occasional investment, while inference serves billions of users and many applications. If the split is 50/50 today, it could eventually become 40/60, 30/70, or 20/80.
• Coding is already a successful vertical application in both enterprise and consumer markets. Bill also sees promise in financial analysis, financial investments, legal services, medicine, and biology. AI is strong at analysis, and legal work is highly rigorous. Autonomous driving is already quite mature, as Waymo and Tesla show; Bill believes it is only a matter of time before every car has a full self-driving function. Traditional automakers are competitive at building cars but lack AI competency; third-party software platforms could become a large market.
• Asked whether large models would take over vertical startups, Bill said the field will see many winners. Frontier labs will continue training powerful models that are strong across the board, like martial artists mastering “one move that beats every other move.” At the same time, there will be vertical models that specialize in code, finance, autonomous driving, or legal services—building on general knowledge, then going deeper in a specific field through domain-knowledge fine-tuning.
• It is still early for a unified training-and-inference stack because the infrastructure for training and inference remains different. Pre-training is a separate process; at inference time, models can access online data, update, and use it quickly. But Bill sees no need to combine pre-training, post-training, and inference into one system, and the technology is not mature enough.

4. Startup opportunities: devices and inference-optimization agents; memory will remain scarce in the agentic era

• Asked where he would invest if starting a company in infrastructure, Bill said he would look at devices and agents that can work efficiently at scale. He also cited several startups doing well, including Fireworks, BLM, and Infer Act. The host noted that panelist 邦华 is Infer Act’s founder and also the founder of RedisArc (S line), an inference framework.
• For inference, the goal is to make models run faster, at lower cost, and more efficiently. Bill sees AI agents as a way to improve inference performance, reliability, and scalability, and said he would invest in agents that optimize inference. The combinations to optimize include Nvidia GPUs, AMD GPUs, Google TPUs, AWS training, and different models—a large N×M space. Optimizing each combination with people is “very, very unscalable”; having agents do the work can be “very, very cheap and also scalable.”
• Big tech companies will work on this too, but Bill said that having plenty of money does not mean they will necessarily do it best. Startups made up of brilliant, enthusiastic, passionate, skilled experts can sometimes move faster and do a better job.
• Bill believes memory will remain scarce. Training and raw inference have historically demanded accelerators such as GPUs, TPUs, and ASICs, as well as memory. In the agentic era, demand for CPUs and memory will increase, along with demand for large amounts of flash and hard-drive storage. These components will all remain in relatively short supply.

5. TPU and Nvidia are learning from each other; on-device hybrids are already emerging, and risks remain in the sandbox

• Asked about Anthropic’s 5-year agreement covering 1 million TPUs, Bill said he did not see it as TPUs “competing with Nvidia across the board,” but as a case of mutual learning. Google and Nvidia have different business models: Google chooses to work with selected companies rather than simply selling them TPUs to use as their main source of compute, the way they use Nvidia GPUs. More AI hardware will help lower costs and improve performance.
• Bill was not sure whether all of Muse’s compute runs on servers or some of it runs on devices, but said hybrid on-device and cloud modes are already emerging. TikTok’s image editing and generation run on-device; real-time translation can also let Spanish and English speakers understand each other using on-device models. On-device AI gets less attention, possibly because the business model is still unclear, but on-device chips and models will get more capable. Google’s Gemma 1B and 2B models are doing well, and the company is also exploring internally how to shift more traffic to on-device processing.
• Bill said large language models are neural networks that are well designed and work well, but remain black boxes: “We don’t understand why it works this way.” Because we do not understand why they operate as they do, their behavior is hard to control; risks can be reduced but not eliminated. Once a model has been trained, debugged, and its safety issues addressed, inference is “not too bad” for now. The greater concern is a newly trained model whose behavior is not yet understood, especially unpredictable behavior during testing, evaluation, and in sandbox environments. Possible measures include forming coalitions to control AI, putting safety and security safeguards in place, and making sandboxes more containerized, with a wall around them.
• On education, Bill emphasized that humans can create AI because they have knowledge of science and engineering, as well as an understanding of the arts and literature; people need those foundations to create better AI. The value of knowledge is not just repetition—for example, knowing that “one plus one equals two”—but giving the brain greater capacity for thought and reinvention, which in turn enables the creation of new knowledge.