Chip design from the bottom up – Reiner Pope

Summary

• The episode’s through-line — and the frame for evaluating any AI chip bet: maximize compute relative to communication, at every level of the stack. Reiner Pope walks from logic gates up to full architectures showing the same trade-off recur — in number precision, register files, systolic arrays, clock speed, and GPU-vs-TPU layout. “This shows up all the way up and down the stack.”
• Multiplier die area scales quadratically with bit width — p×q gates for a p-bit × q-bit multiply — which Pope calls “the single reason low-precision arithmetic has worked so well for neural nets.” Nvidia’s specs are catching up: through B100/B200 halving precision only doubled FLOPs, but B300 lists FP4 at 3x FP8 — “though it should be 4x.”
• The pre-Volta CUDA-core data path’s dirty secret: “seven-eighths of the cost is in reading and writing the register file,” not the math. Tensor Cores/systolic arrays exist to fix exactly this — store weights locally, trickle-feed them in, and get quadratic compute for linear communication. The systolic array (128×128 in older TPUs) is “the most efficient known circuit” for matmul.
• “Most of the decisions in chip design are sizing decisions” — e.g., budgeting 10% of area for data movement vs 90% for the systolic array — and clock speed is itself an area trade: throughput = work per cycle × cycles per second, so cranking frequency by stuffing in pipeline registers can leave you “spending almost all of your area on pipeline registers.”
• FPGA-vs-ASIC economics in one line: an ASIC is ~10x cheaper and more energy-efficient, but “the first FPGA costs you $10,000, whereas the first ASIC costs $30 million” in tape-out — so FPGAs win when workloads change monthly and you need deterministic latency (HFT). CPU non-determinism is a design choice: the cache, 100x faster than DDR, makes latency environment-dependent; TPUs use software-managed scratchpads, while Groq advertises deterministic latency.
• The closest thing to a MatX product tell: a GPU is “a lot of tiny TPUs tiled across the whole chip” (tensor core ≈ MXU), trading the TPU’s better register-file amortization for richer internal bandwidth — Dwarkesh speculates MatX might target both sides of that trade; Pope says MatX has publicly discussed a “splittable systolic array,” “big systolic arrays that can be small systolic arrays too.” Note: Dwarkesh discloses he is an angel investor in MatX.

Deep dive

1. The atom of an AI chip is the multiply-accumulate — and precision is asymmetric

• Pope’s starting point: a matrix multiply is a triple for-loop of output[i,k] += input[i,j] × other input[j,k], so a multiply-accumulate happens at every single step — that’s the primitive AI chips are built around. And precision “will almost always be higher in the accumulation step than in the multiplication step”: the sum repeats j-many times so rounding errors compound, while each chain has only one multiply — hence the demo of a 4-bit multiply into an 8-bit add.
• The worked example builds the circuit by hand: 16 partial products from p×q AND gates, then summed with “full adders” (3→2 compressors) in a Dadda multiplier — the standard area-efficient design. Dwarkesh does the algebra himself: 24 input bits minus 8 output bits = 16 full adders, and p×q in general. Pope: the second reason MAC is the right primitive is that it yields “this very slick, simple p×q” algebra.

2. Quadratic precision scaling — and Nvidia’s specs reflect it

• Dwarkesh’s realization mid-lesson: multiplier area is quadratic in bit length, “so smaller precision is even more favorable than you might think.” Pope’s endorsement is categorical: this quadratic scaling “is the single reason low-precision arithmetic has worked so well for neural nets.”
• The tradeable corollary: historically through B100/B200, halving precision doubled FLOP count — slightly wrong per the math. Nvidia’s B300 specs now list FP4 at 3x FP8 — Dwarkesh: “though it should be 4x.” Pope: “Yeah,” with the caveat that floating-point exponents complicate the pure-integer picture.
• Dwarkesh’s sharp question — can you “funge” FP4 and FP8 circuits? Pope: “as drawn, they’re not particularly fungible” — the FP4/FP8 mix is one of the main design choices, set by customer requirements or by equalizing power budgets. Part of the clean 2x ratio is data movement: two 4-bit numbers pack into one 8-bit storage slot, which “makes the sizing of the buses work out really nicely.”

3. Seven-eighths of the pre-Tensor-Core data-path cost was data movement

• The generic CPU/CUDA-core data path — register file feeding an ALU — hides a brutal tax: selecting a register requires a mux costing n×p AND gates plus (n−1)×p ORs, times three input ports. Plugging in numbers: 24p gates of data movement vs 4p gates of actual multiply-add. “Almost all of the cost, seven-eighths of the cost, is in reading and writing the register file.”
• Dwarkesh’s software-brain moment, worth keeping: “You just think, ‘Oh, I’ll just select element three,’ and something as simple as that is in and of itself quite a complicated circuit.” Pope: “This is the first step of all of the hidden data movement costs” — and this exact problem statement is what motivated Tensor Cores in the Volta generation.

4. Systolic arrays: bake two loop levels into hardware, get quadratic compute for linear wires

• The trick: go two loop levels up and hard-wire the whole matrix-vector multiply. The weight matrix stays fixed and is stored in registers local to the array, reused across many vectors — so you get x·y compute for only x communication with the register file. Dwarkesh’s summary Pope endorses: matmul’s nature lets you “stuff a lot of multiplication in before you get some value out.”
• How do the weights get there? “We just do it very slowly. We very slowly trickle-feed it into the systolic array” — a daisy chain loading one row per clock. Dwarkesh’s gloss: bandwidth equals die area, so load rarely-changing data over small lanes. Older TPUs were described as 128×128; it’s “the most efficient known circuit for implementing a matrix multiply.”
• Asked what keeps him up at night, Pope demurs from drama: “Most of the decisions in chip design are sizing decisions.” How big the systolic array, how big the register file — coupled questions, e.g. budget 10% of area for data movement, 90% for the array. Bigger register files buy application-level flexibility but steal from compute.
• Dwarkesh ties it to their prior episode: inference across chips optimizes compute per memory bandwidth; here it’s multiplies per register-to-logic transport. Pope: same effect “purely in the precision of the ALU, but also in the size of the matrix.”

5. The clock is a chip-wide mutex — and loops in your logic set its speed

• Pope’s framing: chips have 100 billion transistors of parallelism, and instead of software’s expensive mutexes, “every nanosecond or so, all circuitry in the chip will pause for a moment and synchronize.” Timing is margined so signals arrive ~25% of a cycle early — misses are “many standard deviations out,” except at clock-domain crossings.
• Dwarkesh’s Factorio pushback — why synchronize globally at all? “Things are just done when they’re done.” Pope’s answer: manufacturing variance means when computations f and g meet at h, f might arrive against the previous or next value of g. This is also why two chips on the same 3nm node can clock differently — it depends on taming the critical path.
• Pipeline register insertion splits a logic cloud in half for 2x clock at the cost of area — but feedback loops (like a running sum) can’t be split without changing the computation, “and that sets the clock cycle.” Go too far and a register (8 gate-equivalents) dwarfs the AND gate (1) it feeds: “almost all your cost becomes synchronization or communication cost compared to the actual logic.” Dwarkesh’s callback: it’s the batch-size trade-off again — low latency, low throughput.

6. FPGAs are muxes all the way down — and 10x is the price of programmability

• The business case, in Pope’s numbers: anything an FPGA does, an ASIC does ~an order of magnitude cheaper with better energy efficiency — but “the first FPGA costs you $10,000, whereas the first ASIC you make costs $30 million because it requires an entire tape-out.” FPGAs win for deterministic latency plus workloads that change “maybe every month.”
• Mechanically: registers, 4-input lookup tables (16 possible functions, stored as a truth table), and muxes selecting among neighbors — configuration is just setting mux controls. Dwarkesh probes the layering; “It’s muxes all the way down.” The 10x, quantified: a four-way AND is 3 gates in an ASIC, 32 in a LUT — the overhead of listing every truth-table row instead of just placing the gate.
• Why HFT shops don’t just use CPUs: determinism is possible (Groq advertises it; TPU cores have it) but “not very attractive in the market.” The main culprit is the cache — two orders of magnitude faster than DDR and essential (“all programs would run a hundred times slower” without it) — whose hit rate depends on the CPU’s ambient environment. TPUs invert the philosophy: software-visible scratchpad instructions vs separate HBM instructions, no hardware guessing.

7. What CPUs spend their die on — and why the brain’s slow clock isn’t a silicon cheat code

• A CPU is only ~1,000-way parallel (100 cores × 16-wide vectors), and its cores are huge: cache, register files, and — the piece with no GPU equivalent — the branch predictor. Resolving a branch takes ~5ns (a 200MHz clock); to run at 1-2GHz you must predict “five cycles earlier, before you even get to that instruction.” Stripping predictors and tightening register files “drives a lot of the GPU gains over the CPU.”
• On Dwarkesh’s brain comparison: the brain runs batch-size-1 at a slow clock, but under-clocking silicon doesn’t unlock brain-like efficiency. Since dynamic switching power — charging and dumping a capacitor per bit toggle — is most of a chip’s energy, clocking 1,000x slower gives ~1,000x fewer transitions and ~1,000x less energy: proportional, “not a substantial advantage in energy efficiency.”

8. A GPU is a lot of tiny TPUs — and MatX has discussed a splittable systolic array

• Pope’s top-level map: a GPU is a regular grid of near-identical SMs around L2; a TPU is a few huge matrix units around a vector unit. Shrink a TPU down and “that is sort of what an SM is” — “the GPU has a lot of tiny TPUs tiled across the whole chip,” tensor core ≈ MXU.
• The trade: big TPU-style arrays amortize register-file costs better, but data between vector and matrix units must cross “just two lines of perimeter” vs 16 lines of wiring in a GPU — so within-SM movement is cheap and plentiful on GPUs, while cross-unit movement is the TPU’s constraint.
• Dwarkesh floats the MatX thesis directly — GPU-like small systolic arrays with SRAM, minus the CUDA-support baggage in each SM. Pope, carefully: “We’ve talked publicly about something we call a splittable systolic array — big systolic arrays that can be small systolic arrays too.”