Ep. 027 - OpenAI Jalapeño: Better Than Nvidia Blackwell (Accelerators)
Summary
OpenAI’s first Jalapeño results beat GB300 decisively and outperform Vera Rubin’s July results on output tokens per utility megawatt. The comparison to Blackwell is imperfect because Jalapeño uses HBM4 versus Blackwell’s HBM3, while Rubin offers a fairer software-maturity comparison. Rubin has likely improved since July, but Jalapeño’s own curve is “improving by the week…even by the day.”
Power—not package count—is becoming the binding inference constraint, making Jalapeño’s performance per megawatt economically meaningful. At fixed interactivity of roughly 50–100 tokens per second, Jalapeño produces about twice GB300’s tokens per megawatt; at low batch size, it reaches roughly 700 tokens per second versus 350. As Jordan puts it, OpenAI is winning at both “fast tokens and cheap tokens.”
The TCO case survives beyond the power chart because OpenAI avoids NVIDIA’s margin while owning and operating the infrastructure itself. SemiAnalysis assumes hourly ownership costs of $2.79 for GB300, $3.61 for Rubin, and $1.56 for Jalapeño, versus roughly $6 an hour for current GB300 NeoCloud pricing. On those inputs, Jalapeño either wins or comes very close on performance per TCO.
The disclosed benchmarks may materially understate Jalapeño’s production potential. Results used single-token prediction without prefill/decode disaggregation across DeepSeek R1, Kimi K2.5, and GPT-OSS 12B; OpenAI reportedly sees a “three to five times” improvement from an undisclosed internal speculative-decoding method on production models. Jordan frames the method as a personal guess because no open-source implementation exists for independent verification.
Jalapeño’s advantage comes from realized bandwidth and workload efficiency, not headline specifications. It has less than half Rubin’s stated FP4 FLOPS, less HBM capacity, and a 700-watt TDP, yet delivers 15.4 TB/s of HBM4 bandwidth and outperforms theoretically larger chips. The lesson is that published FLOPS are “numbers that you are guaranteed not to exceed”; architecture and software determine what workloads actually realize.
AI compressed both chip design and software bring-up, potentially weakening CUDA’s historical switching-cost advantage. Before GPT-5, AI assistance reportedly cut SIMD area by 8% and matrix-engine area by 10%; after silicon arrived, agents generated high-performance kernels that engineers themselves could not meaningfully inspect line by line. Myron’s formulation captures the irony: “NVIDIA’s hardware has enabled moving off of NVIDIA’s hardware.”
This is a real threat to NVIDIA’s accelerator dominance, but not yet to its full deployment ecosystem. Bryan puts the timeline at under two years from concept to a real chip in the lab and under nine months to tape-out, yet OpenAI must still deploy perhaps millions of chips, thousands of racks, and gigawatts rather than demonstrate three test racks. NVIDIA’s moat still includes supply, support, logistics, monitoring, and availability—while Broadcom and Celestica give OpenAI experienced scaling partners.
Deep dive
1. Jalapeño wins where inference economics now bind
Myron’s headline finding: Jalapeño beats GB300 “out of the water” and also exceeds Vera Rubin’s July results in output-token throughput per utility megawatt. Comparing directly with Blackwell is somewhat unfair because Jalapeño has HBM4 while Blackwell uses HBM3; Rubin is the cleaner reference point, though its current software is likely better than the published July snapshot.
Jordan’s denominator matters because data centers are increasingly power-limited. With 100 MW available, buyers can purchase only as much compute as the site can energize; if one system produces more saleable tokens per megawatt, higher chip prices may be justified. OpenAI has no external Jalapeño customers, so tokens per package matter less than tokens produced from its constrained power envelope.
Bryan’s objection to package-level comparisons: “Performance per chip is at the end of the day just an imaginary thing.” A vendor can join two dies and declare double performance per chip, as with Ultra products, or call an entire wafer one chip. Myron adds that performance per watt and performance per cost are closely linked, since higher power generally also implies a more expensive chip.
The TCO model assumes OpenAI owns the systems and data centers: $2.79 per hour for GB300, $3.61 for Rubin, and $1.56 for Jalapeño—not the roughly $6 hourly NeoCloud price cited for GB300. The in-house thesis is straightforward: pay Broadcom’s margin rather than NVIDIA’s, or the Broadcom-plus-Google-TPU margins or an equivalent.
2. The Pareto curve shows both cheap tokens and fast tokens
Jordan’s explanation of the curve: the vertical axis is tokens produced per megawatt, while the horizontal axis is interactivity—the speed experienced by each user. At roughly 50 or 100 tokens per second per user, Jalapeño yields about twice GB300’s total tokens per megawatt, implying potentially twice the revenue-generating output from a power-constrained inference site.
At the low-batch, high-interactivity edge, Jalapeño reaches approximately 700 tokens per second per user while competing systems peak around 350. That is unusual because accelerator challengers typically choose one end: Cerebras-like systems emphasize fast decode, while AMD, TPU, or Trainium-style systems tend to emphasize throughput. Jalapeño presently performs well on “both sides of the curve.”
Jordan notes that Jalapeño became the first non-NVIDIA, non-AMD chip to appear publicly on SemiAnalysis’s inference benchmark, ahead of systems they expected from SambaNova, Cerebras, TPU, or Trainium. Her challenge to every claimant, including Etched: “If your chip is good, just run the benchmark…let the results talk.”
3. Early benchmarks leave several large optimizations unused
The workload set was DeepSeek R1, based on the V3 architecture released in January 2025 and carrying roughly 600 billion total parameters; Kimi K2.5, with 1 trillion parameters; and OpenAI’s smaller GPT-OSS 12B. All delivered strong results, but the random-data 8K/1K test does not exercise agentic workflows, prefix caching, or every load-bearing optimization exposed by SemiAnalysis’s AgentX benchmark.
OpenAI ran single-token prediction rather than multi-token prediction, which guesses future tokens and verifies them together. Jordan’s personal guess is that OpenAI uses an internal speculative-decoding method distinct from MTP, DFlash, and other open-source configurations, but withheld those results because outsiders could not reproduce them.
The reported production uplift from that internal method is “three to five times.” If transferable to the shown workloads, it would move Jalapeño’s curve dramatically, but the hosts retain the verification caveat: disclosed comparisons do not contain that optimization, and the internal technique itself was not available for inspection.
Jalapeño also lacked prefill/decode disaggregation, another material optimization. Bryan says they were told OpenAI had no optimized internal MLA implementation for its own models, although the visiting team later viewed an AI-generated DeepSeek MLA kernel. Bryan says the AgentX timing was poor and he does not think the Jalapeño team actually ran it. The disclosed showing is therefore impressive but not a complete production comparison.
4. Realized HBM bandwidth beats theoretical FLOPS
On paper, Jalapeño looks weaker than Rubin: less than half the stated FP4 FLOPS, less HBM capacity, and much lower TDP. Myron cautions that theoretical FLOPS can be effectively unreachable; raw HBM bandwidth is similarly unhelpful if complicated memory subsystems and excessive data movement prevent workloads from using it.
Jalapeño delivers 15.4 TB/s at 700 watts, using HBM4 at 10 Gb/s pin speed. The discussion compares that with roughly 20 TB/s for Rubin and says NVIDIA is expected to ship Rubin at around 9.6 Gb/s per pin. Even Rubin’s 1,800-watt low-power setting consumes more than twice Jalapeño’s power, so SemiAnalysis’s bandwidth-per-watt comparison puts Jalapeño at about double Rubin.
Samsung’s reversal is central. Its HBM3 and HBM3E had lagged SK hynix, but its HBM4 uses a more advanced 1c DRAM process and an SF4 logic base die; SK hynix was described as using 1b-class DRAM and TSMC 12 nm, while Micron retained a DRAM process for its base die. Broadcom’s historical reliance on Samsung hurt previously but may now look fortunate.
5. The architecture blends TPU regularity with GPU flexibility
Jordan describes Jalapeño as TPU-like but built around smaller systolic arrays. Large arrays can suffer utilization cliffs on skinny GEMMs, odd dimensions, low concurrency, or many-expert workloads: if dimensions are slightly over 128 or 256, an entire kernel launch may be spent on one little skinny GEMM while much of the array remains unused. Smaller tiles trade some theoretical efficiency for better utilization across changing shapes.
Jordan suggests OpenAI addresses the trade-off through careful placement of weights and KV caches, selective synchronization between cores, and sparing use of the on-chip collective network connecting HBM slices to compute. The result may be easier to reason about than a GPU while remaining more flexible than a conventional large-array TPU.
The chip also uses an L1 cache rather than the software-managed scratchpad common in rival accelerators. That can complicate barriers and data movement, but Jordan ties the choice to AI-generated kernels that handle the burden. Its generality is underscored by running unrelated models—and, as the team jokingly emphasized, Doom at 36 frames per second—rather than being narrowly co-designed for one OpenAI model.
6. AI shortened design cycles, but deployment remains the next moat
The RTL freeze occurred in July of the prior year after AI-assisted work from roughly February through July reduced SIMD area by 8% and matrix-engine area by 10%. Jordan stresses this predates GPT-5: even older models accelerated tedious RTL work, suggesting future design cycles could compress below Jalapeño’s already sub-nine-month path to tape-out.
Jordan frames kernel tuning during bring-up as an iterative, verifiable problem suited to reinforcement learning: generate code, test correctness and performance, then repeat. Engineers showed SemiAnalysis a roughly 30,000-line Gluon kernel resembling “AI-generated assembly”; they understood the hardware and system concepts but could not explain the code line by line. “It doesn’t matter,” Jordan argues, because the kernels were correct and fast.
Bryan’s caveat to “CUDA moat’s gone” is that NVIDIA’s moat extends beyond programming. He says OpenAI must figure out how to turn on 100 MW of these chips, not just three test racks, while scaling supply chains toward millions of chips, thousands of racks, and gigawatts—support, logistics, and monitoring work that remains “a monumental challenge.” Broadcom and Celestica bring experience, but working lab racks are not mass production.
The broader competitive signal may be harshest for Meta and Microsoft: Bryan asks whether OpenAI is exceptional or their silicon teams have “skill issues,” concluding it is probably both. Anthropic is now hiring a silicon team, yet the hosts expect frontier labs to integrate further because privileged model access can improve hardware before competitors receive the same models.