How Deep Learning Finally Cracked Messy Tables - Frank Hutter

Summary

• Deep learning has finally cracked tabular data — Tim Scarfe’s framing is that TabPFN now “works dramatically better than CatBoost and XGBoost,” after many failed attempts such as Google’s much-hyped 2019 TabNet. Frank Hutter’s explanation is that in-context learning transfers patterns and feature interactions across datasets: individual rows from a medical table teach you nothing about an insurance table, but a pre-trained transformer can ingest the whole table and emit a bespoke classifier in a single forward pass, with no training loop or hyperparameter search.
• TabPFN is “the first foundation model that’s actually state-of-the-art, yet entirely trained on synthetic data” — hundreds of millions of datasets sampled from priors over structural causal models. There was no useful internet corpus to scrape (“people do not put their tabular datasets online”), which turns the usual data-moat logic inside out: no leakage or memorization issues, and direct control over the generated training distribution — with iterative prior engineering becoming a key advantage.
• Context scale is compounding at “two orders of magnitude a year”: TabPFN1 handled 1K rows, v2 10K, v2.5 100K, v3 1M, with 10M the stated next target. LLMs are poorly suited to this regime — a million rows by a thousand columns is 3–4 billion tokens, and sequence models “don’t understand that if you switch two rows… it’s the same dataset”; with big tables “that’s where the LLMs just really fall flat on their nose.”
• Google’s TabFM (described by Scarfe as roughly 30x larger) validates the field but exposes the trade Prior Labs is exploiting: it’s 15x slower per forward pass, clearly strongest on tiny and small data, and “just breaks, out of memory” beyond roughly 100K rows. Hutter’s post-recording update: TabPFN 3.5 took first place on seven benchmarks, is 20x faster than TabFM at equal quality or over 100 Elo points higher while still faster, and Nick Erickson got the number-one solution on a 3,500-competitor 2015 Kaggle challenge “in one line of code in one minute of compute” on one RTX PRO 6000 C GPU.
• The causal roadmap is the most thesis-relevant moonshot: Do-PFN can use observational data at test time to make interventional predictions, after meta-training on synthetic causal interventions, and papers already show it can reduce the number of RCT data points needed for comparable performance. Hutter says that could “completely revolutionize medicine and A/B tests” if drugs can reach market on shorter studies — hedged with “we need to do this properly and theoretically grounded… this is high stakes.”
• The commercial wedge is the agentic token economy, not a standalone app: “LLMs call calculators because they’re better at doing math… just like that, they should call tabular foundation models.” Revenue stack: free non-commercial open-source use, production license fees, an API, fine-tuning-as-a-service, private VPCs, and distribution via Azure, AWS, and SAP’s Generative AI Hub — Prior Labs was acquired by SAP, with “the leading motive for the acquisition has been DeepMind”: fund an open research arm and hire from 11,000+ applicants (45 hired).
• Positioning takeaway: tabular data is “the most common modality in the enterprise” yet the least glamorous, and Hutter expects tabular foundation models “used left and right, often without you knowing it” within a couple of years — embedded invisibly in agent harnesses (MCP integrations for Claude and Gemini already exist), with relational databases, time series, and text-in-tables as the next priors to be baked in.

Deep dive

1. A decade of failed deep learning on tables — and why in-context learning broke the dam

• Hutter’s diagnosis of why tabular resisted deep learning: “countless attempts” — Google’s TabNet in 2019 was “really hyped, thousands of citations… and it just doesn’t work. It doesn’t generalize to new datasets.” Tables are heterogeneous: outliers, categorical/binary/ordinal values, “missing at random, missing not at random” — and standard deep learning treated everything the same, unlike images with regular spatial relationships.
• The structural problem: you can train one ImageNet covering the visual world, but a medicine table and an insurance table share nothing at the row level. The transfer lives one level up — “detecting the patterns, how the different features interact… detecting causality potentially. And on that level, using in-context learning, you can actually transfer across different datasets.”
• Tim Scarfe’s framing from “bitter experience”: pre-TabPFN data science meant normalization, encoding, imputation, feature engineering — “even though you can make an ML model, you always feel kind of dirty afterwards.” Hutter’s counter-vision: new data scientists “really don’t need to understand everything about XGBoost anymore”; the job shifts to data drift, fairness, causality, privacy — “more human-quality time on a problem rather than fiddling with your hyperparameters.”

2. No internet corpus existed — so the entire foundation model runs on synthetic data

• The core constraint: “humans like to put their text online… but people do not put their tabular datasets online.” Maybe 50,000 usable datasets exist; the millions of web tables are things like “this basketball player has this number on their back” — not datasets from which you can reasonably learn a statistical learning algorithm.
• The solution makes TabPFN “the first foundation model that’s actually state-of-the-art, yet entirely trained on synthetic data” — sampled from priors containing substantial structural-causal structure, approximating “the Bayesian posterior of all kinds of structural causal models that could explain the data.”
• The side benefits are commercial as much as scientific: “we don’t have any leakage, we don’t have any memorization issues… we don’t have to worry about biases, because we have a codified pipeline for generating the synthetic data from scratch.” Prior development is iterative — hold out real datasets, find failure modes such as outliers, uninformative features, and categoricals, add those complexities to the generator, and retrain. “If your prior is limited… the posterior is gonna be limited.”

3. Hutter’s lineage: SAT solvers → Auto-WEKA → neural architecture search

• The through-line of his career is automating design decisions. His PhD, under Holger Hoos, Kevin Leyton-Brown, and Kevin Murphy — Murphy “maybe the least believer in automated algorithm configuration” until he later became excited about neural architecture search — began with SAT solvers requiring hundreds of categorical decisions, then ML tuning ML hyperparameters, then hierarchical model-selection spaces.
• Auto-WEKA — “a little robot riding a Weka bird” — targeted non-experts in the life sciences who would otherwise use “an SVM with default hyperparameters because their friend told them that would be good.” The founding principle: “democratize state-of-the-art machine learning to everyone, also those without a PhD in machine learning.”
• NAS extended the same machinery to architectures — hardware-aware, multi-objective across accuracy, latency, and memory — but Hutter concedes the empirical science was hard: the algorithms were “very complex to run” and costly.

4. TabPFN is AutoML’s convergence point: an entire algorithm learned end-to-end, executed in a forward pass

• The historical beat: the TabPFN v1 paper was published a week before ChatGPT came out in November 2022. Hutter’s framing: “TabPFN is really this natural progression of AutoML where we learn this entire algorithm that is executed in a forward pass, and depending on the dataset that you feed in, it learns a different classifier.” You could export it as ONNX onto a sensor — a chip running not a classifier but a machine learning algorithm.
• Unlike hand-derived XGBoost, the meta-training loss is literally the thing you want: cross-entropy on the unseen test portion of hundreds of millions of synthetic datasets — “that is not peeking at the test set… at meta-training time you have learned to really solve this problem.” Being declarative is the point: specify what datasets you should work well on, and “deep learning just needs to do its magic.”
• The other AutoML machinery survives, but migrates to pre-training time; at test time “data scientists get what they always wanted, namely a just really fast method that works out of the box.” Unlike chain-of-thought’s autoregressive rollout, “it’s literally the one token you need to predict, and that is it” — test predictions are independent by construction.

5. The Bayesian mechanics: skip the posterior over functions, meta-learn the posterior predictive directly

• Hutter’s cleanest technical explanation: MCMC is “just really slow,” while variational inference has “complex math” and approximations that do not always work. TabPFN skips the posterior over functions or parameters entirely and directly approximates the posterior predictive distribution — sample functions from the prior, sample train/test points from each, and learn over millions of samples to predict test Y from training points.
• The correspondence follows the prior: a linear-only prior yields Bayesian linear regression; a GP prior yields the GP posterior, or an approximation thereof; an SCM prior yields “the integral over all possible structural causal models that could cause the data.” The space is effectively infinite, but “our network is so small it can’t possibly memorize that, so it actually learns to generalize across the space.”
• Two ingredients are central: “you need to be able to sample from your prior, and you need to be able to fit strong neural networks in a supervised learning fashion.” Everything else piggybacks on LLM progress — “architectures get better, optimizers get better, and our methods also directly get better.” Hutter notes that AdamW and cosine learning-rate scheduling are being used to train “essentially every transformer in the world.”

6. The scaling ladder — and why LLMs “fall flat on their nose” on big tables

• The row-count progression: TabPFN1 handled roughly 1,000 data points; v2, 10,000; v2.5, 100,000; and v3, 1 million. “We’re getting there like two orders of magnitude a year,” with 10 million the stated next target — “we’re pretty confident.”
• Why not just use an LLM: a million rows × a thousand columns is a billion elements, 3–4 tokens each — “three, four billion tokens in your context, and LLMs wouldn’t be very happy there.” Worse, sequence models lack the relevant invariances: “if you switch two rows or you switch two columns, it’s the same dataset” — and their world-knowledge parameters are not needed for pure statistical reasoning. With five rows and known column names, world knowledge can help; “if you have a million rows, then you actually want to learn about the statistics of these numbers.”
• Kaggle has not been the proving ground mainly because few competitions sat in the size range TabPFN could handle until recently, and the priors lacked Kaggle’s data complexities — non-IID group data, temporal data, and text in tables — which are now represented in the Beyond TabArena benchmark and “that’s a signal we’re optimizing for right now.”

7. Coding agents are the interface; TabPFN is the engine

• Scarfe’s pushback — worth keeping: a bare LLM fed a thousand rows is a strawman, because Claude Code writes Python, understands domain semantics, and can build adaptive pipelines that self-correct on prediction errors. Hutter’s answer is division of labor: agents are “great for feature engineering, great for exploratory data analysis, great as a user interface… but in the end they need to call a model, and they can call XGBoost, but it’s just not gonna be as good as the tabular foundation models.”
• The complementarity has history: when ChatGPT came out, Hutter’s group dropped TabPFN into an agentic data-science workflow and wrote the CAFE paper on automated feature engineering atop TabPFN. The BMI example: if height and weight are columns and obesity matters, let the agent compute it rather than making layer one learn it. Agents can also fetch external features — “this particular day is a holiday in this country, so there’s gonna be more demand.”
• It is already available through MCP integrations with Claude and Gemini; a friend prompted Claude to build a historical-games dataset and had TabPFN predict every World Cup match — “with that they actually won their cake bet in their company.” In the Databricks demo, the agent “won’t even tell the user that it’s using TabPFN. Why should it?” Hutter’s two-year forecast: tabular foundation models “used left and right, often without you knowing it.”

8. Architecture evolution: from a de-positioned transformer to row/column attention, still quadratic

• V1 was a transformer without positional embeddings — attention is already order-invariant — with a naive linear encoder for each row. It only handled continuous features; categorical values encoded as 1, 2, 3, 4 would therefore be treated numerically.
• V2 introduced per-cell embeddings with alternating attention over rows and columns, letting the network infer “aha, in this column the numbers are one, two, three, four — probably this is a categorical.”
• The cost: V2’s complexity is N²·M + N·M², versus V1’s quadratic-in-rows-only design — fine to 10K rows, and extended to 100K for v2.5 using more powerful GPUs. V3 adopted the TabICL architecture, with a more sophisticated column-and-row transformer for embeddings and innovations in output heads and many-class handling. Sub-quadratic attention is on the menu; the tabular analogue of language’s locality prior “could be some sort of locality in the embedding space.”
• On outputs: regression is done as classification over roughly 10,000 adaptive bins because, per PhD student Samuel Müller’s experiments, it beat a mean-variance head — transformers are tuned to classify. The payoff is calibrated multimodal predictions: a bird flying at a pole “gonna turn left or turn right… pretty much zero probability mass in between,” all in one forward pass.

9. Test-time compute for tables: a whole menu, but transduction is off it

• Hutter leaned away from transductive inference despite Scarfe’s ARC-Prize-flavored enthusiasm: he would keep an interface where predictions for one test point do not depend on other test points, “because data scientists just wouldn’t have that… if you predict the same data point in a different batch, you get different results — it would be super strange.” Semi-supervised learning remains an option.
• The menu he does endorse: early exit for easy points; fine-tuning on synthetically generated lookalike datasets, which a master’s student showed can help on very small data; gradient-based prompt tuning — select or even “hallucinate a hundred thousand data points that approximate these billion data points better”; and forward-pass model selection across networks trained with different priors. “There are so many different things you can do, and I think the sky’s the limit.”
• The interpretability gem: former PhD student Robin Schirrmeister condensed a 1,000-point medical dataset into two prompt data points with the same predictive performance — instant prototypes. Gradients also enable feature importance and data-point influence, along with detecting adversarial attacks or influential outliers: “this is one data point that completely dominates my model — how about I train again without it.” These capabilities are not available in this form from ordinary XGBoost.

10. TabArena: Elo for tabular models, with obsessive curation and $2 reproducibility

• TabArena began as a multi-affiliation collaboration led by Nick Erickson, then at AWS and the builder of AutoGluon, “the best AutoML system for tabular data there was.” Erickson and several co-authors have since joined Prior Labs. Unlike LM Arena, no human raters are needed: real train/test splits give an objective criterion, with Elo tournaments layered on top.
• The hygiene is the point: it is a living benchmark with all artifacts and per-split predictions published, independently reproduced, and “running your own benchmark costs something like two dollars on a GPU.” For Beyond TabArena, curation drops sets with fewer than 100 samples and datasets that only look tabular — MNIST-as-columns gets excluded because “you should just use a vision classifier” and tabular data should not carry spatial-neighbor patterns.
• On Scarfe’s leakage worry — does generating synthetic training data risk contaminating benchmarks? Hutter’s flat answer: the benchmarks are real community datasets, curated from thousands with every inclusion decision documented; the synthetic data generated for training is not put onto the benchmarks.

11. Google’s TabFM: “imitation is the sincerest form of flattery” — and the moat is speed and scale range

• Roughly three weeks before recording, Google released TabFM — the same architecture family and types of priors, scaled by roughly 30x with Google compute. Hutter’s honest read: “it’s not surprising that this is better,” but “maybe a little premature to go that big” — the forward pass is roughly 15x slower than TabPFN’s, and even Prior’s thinking-mode test-time compute, 10x slower than its standard model, is faster than one TabFM forward pass while stronger in Elo.
• The scale-range asymmetry defines the competition: TabFM is better on tiny and small datasets, breaks even at medium scale up to 100K data points, and “just breaks, out of memory” on Beyond TabArena datasets beyond roughly 100K. Prior had already shifted focus to the TALENT benchmark, which reaches 1 million points, and to non-IID complexities. Hutter’s concession: “this is a good reminder that we should also focus a bit more” on TabArena — while refusing to “over-index on it.”
• Beyond compute, the claimed moat is prior engineering: the iterative, principled process of encoding how real-world data might have come about and testing failure modes.

12. Causality is the moonshot: interventional predictions from observational data

• The teaching example, as told: patients with a disease get a medicine, with dosage scaling with severity. A predictive model says medicine → disease — “you might be tempted to say, ha, let’s stop giving them that medicine and they won’t have that disease anymore, but that would be foolish.” Pearl’s do-operator versus observation is the whole distinction, but in practice “often you just don’t know the graph.”
• The PFN move: during meta-training, sample a causal graph, observe variables, then intervene on the same sampled system and observe the effects — learning over hundreds of millions of graphs: “if I see these types of observations and I make this intervention, this will be the effect.” Non-identifiability is handled honestly: when A→B and B→A are equally likely and disagree, the output is a bimodal 50/50 with zero mass in between. This is Do-PFN, developed together with advisor Bernhard Schölkopf, whom Hutter describes as the most cited scientist in causal ML; the CausalFM lead author from Layer6 and the University of Toronto joined Prior “just last week.”
• The stakes claim: papers already show causal foundation models can get close to RCT-level performance while reducing the number of RCT data points needed — “that will completely revolutionize medicine and A/B tests… get our medicine to the market much earlier, with the same confidence or even higher.” Hutter immediately hedges: “we need to do this properly and theoretically grounded… this is high stakes.” An ICML follow-up lets domain scientists pin known edges, such as A causes B, and restrict the posterior accordingly; probabilistic structure extraction and root-cause analysis are “something we are working on — our current models do not do that yet.”
• On limits, Hutter doubts LLMs will discover proper causal relationships between observed features from data alone: they will compress knowledge and “form lemmas,” but “I do not think that they will come up with proper causal relationships between features that we see from the data… it’s in the numbers, in the statistics, and that’s what the LLMs are not strong at. And I don’t see them getting stronger at that any time soon.” The challenge is marrying world knowledge with what the numbers say.

13. Enterprise playbook and the business model: licenses, API, and the agentic token economy

• Tabular is “the most common modality in the enterprise,” yet newcomers “try the LLMs on their tabular data and see them fall flat on their nose.” Hutter’s governance advice mirrors LLM practice: centralize — one team that understands fine-tuning, failure modes, and “where do I need to check whether maybe XGBoost is still better if you have a billion data points.”
• Revenue stack: free non-commercial open-source use; production license fees; thinking and scaling modes; small-data specialist models; fine-tuning as a service, either customer-run or Prior-run; a managed API where batching keeps GPUs busy (“that GPU might idle all the time” if a client reserves one); private VPCs; availability on Azure, AWS, and SAP’s Generative AI Hub.
• The forward bet is agent-mediated consumption: “LLMs call calculators because they’re better at doing math. Just like that, they should call tabular foundation models because they’re better and cheaper.” Scarfe’s read: any Anthropic listener’s first move is a system-prompt update routing tabular work to TabPFN. Hutter: “Yeah, absolutely do that. We have skills files and everything.”
• Relational data came “for free” from the open-source community — an embedding that flattens relational-database queries into one table; TabPFN 3 out of the box beat KumoRFM on the original RelBench, with an improved benchmark slated for release. Hutter confirms multi-table, graph, and tree priors are possible in principle: “this is clearly the name of the game of building priors… the sky’s the limit there. I think this is totally going to happen.”

14. SAP acquisition, an open research arm modeled on DeepMind — and a 3.5 victory lap

• The disclosure: “we just got acquired by SAP, and we have a lot of funds available… The leading motive for the acquisition has been DeepMind” — build the best place in the world to work, “publish up a storm,” and run a fully open research arm collaborating with universities and institutes including Freiburg, Toronto/Layer6, Singapore, and the ELLIS Institute in Tübingen. Hiring reality check: 11,000+ applications, 45 hires, priorlabs.ai/careers — European frontier AI from Freiburg.
• The post-recording update lands the episode’s thesis: TabPFN 3.5 “took first place on seven different tabular-related benchmarks, including tables with text, multimodal data, relational data” — Pareto-dominating other tabular foundation models, “twenty times faster in a forward pass than TabFM for the same quality, or over a hundred Elo points higher while still being faster,” with the biggest gains on Beyond TabArena’s realistic complexities.
• The kicker, as Hutter tells it: Nick Erickson took the 2015 historical AutoML Challenge on Kaggle — 3,500 competitors, a $10,000 prize, won by a 36-model stacked ensemble with handcrafted features; AutoGluon previously reached a top-1% solution in 24 hours on 96 CPUs — and with TabPFN 3.5 “got the number one ranked solution in one line of code in one minute of compute” on a single RTX PRO 6000 C GPU. Scarfe’s coda: he’s already running it as his Discord spam filter — “you need a fairly beefy machine,” but it is free for non-commercial use.