AMA: career advice given AGI, how I research ft. Sholto & Trenton

Summary

• Trenton and Sholto’s answer to why LLMs with all human knowledge memorized have no known examples of cross-domain discoveries: pre-training doesn’t teach the skill of making novel connections — “at a minimum you need significant RL in at least similar things,” and the field hasn’t done that “in a meaningful or scaled way.” Sholto adds that models are “not good at knowing what memories they should be storing,” with primitive memory scaffolding; Dwarkesh wonders whether models are “idiot savants” like Kim Peek.
• The career call for an AGI world: treat the next few years as leverage compounding — engineers already report 2-5x speedups, and the trajectory runs from pairing session → managing a small team → “managing a division or a company” of AIs. Deep technical knowledge “absolutely will” still matter in four years because human management bandwidth becomes the constraint.
• Trenton stresses this “is not just cope” — models lack long-term coherence; “getting a fucking office is kinda complicated,” and the economy is “really big… and really complex.” Dwarkesh speculates that data-sparse sectors where context matters might leave people well-positioned.
• Concrete timeline behavior: Trenton canceled his 401k contribution (“things look so different then”), Sholto just works all the time, and Dwarkesh reinvested the ad money from his Zuckerberg interview straight into Nvidia after a negative-23-cent bank balance.
• Media economics per Dwarkesh: slow compounding audience growth “is kind of fake” — Leopold’s situational-awareness essay proved that if content is good enough, “literally everybody who matters… will read it” maybe within a week. The “Matt Levine of AI” niche remains “totally open.”
• Talent arbitrage is real in editing (Argentine farmer, Sri Lankan math freshman, ex-Mr Beast editor, all found via Twitter contest) but not in executive hiring: ~1,000 applications led to a hire through a mutual-friend referral — “the best people in the world don’t want to apply.”

Deep dive

1. Models memorized everything and have no known discoveries — the RL gap

• Dwarkesh restates his standing puzzle: LLMs hold all human knowledge, yet unlike humans — his example, a guy who noticed magnesium-deficient brains look “exactly the structure you see during a migraine,” said magnesium supplements would cure migraines, and it worked — “I don’t know of a single example of LLMs ever having done it.” Scott Alexander’s defense (humans lack logical omniscience too) doesn’t satisfy him, because humans demonstrably do make these leaps.
• Trenton’s diagnosis: pre-training “imbues this nice flexible general knowledge about the world, but doesn’t necessarily imbue the skill of making novel connections or research” — the thing PhD programs train. His bar: “at a minimum you need significant RL” in similar tasks, and “I don’t actually think we’ve done that in a meaningful or scaled way as a field.”
• Sholto’s memory diagnosis: models “aren’t good at knowing what memories they should be storing” — a human learning something new constructs a summary that sticks; models don’t currently get the opportunity to do that — and memory scaffolding is “very primitive” (Claude Plays Pokemon improved fast once someone iterated the scaffold). Dwarkesh wonders whether models are “idiot savants,” using Kim Peek as an analogy; he says Kim Peek was born without a corpus callosum “if I recall correctly,” with perfect encyclopedic recall but social debilitations — “it’s just kind of amazing how good LLMs are at very niche topics, but can totally fail at other ones.”
• The memorization-generalization gradient, via Terrence Deacon and Gwern’s optimizer theory: children learn best but forget almost everything from childhood, adults sit in between, and LLMs occupy the wrong extreme — “they’ll get the exact phrasing of Wiki text down, but they won’t be able to generalize in these very obvious ways.”

2. Career advice given AGI: buy leverage, stay at the frontier

• Dwarkesh’s frame for the 17-year-old: think of the next few years as “increasing your individual leverage by a huge factor every year.” Engineers already claim 2x, or 5x in new languages; the progression is pairing session → managing a small team → managing a division. Even short of “a true singularity world where you have AIs managing AIs,” you’ll command “vastly more resources than an individual could command today” — so deep technical knowledge in four years “absolutely will” matter.
• Trenton’s emphasis — “this is not just cope”: models genuinely lack the long-term coherence needed to run a company. “Just getting a fucking office is kinda complicated.” Dwarkesh speculates that data-sparse sectors where context matters might leave people in a good position.
• The one piece of career advice Sholto isn’t skeptical of: “put yourself close to the frontier” — “it’s actually remarkably obvious at the frontier what the problems are.” Dwarkesh confirms the GitHub-open-issue path isn’t dead: “That’s still what we look for in hiring.” Dwarkesh adds: learn everything AI-native, top-down, and “be worried and skeptical about any subject which prioritizes rote memorization… instead of ways of thinking.”
• Dwarkesh’s hedge, worth keeping: he’s “quite skeptical of career advice in general” — 80,000 Hours-style guidance was “mostly useless” in retrospect; the podcast itself began as a way to figure out what he wanted to do. “Try things, do things.”

3. New-media growth: compounding is fake, the blogosphere is efficient

• Dwarkesh’s core take: “slow compounding growth in media is kind of fake.” Leopold’s situational-awareness essay had no built-up audience — “if it’s good enough, literally everybody who matters — and I mean that literally — will read it.” Sholto: after it posted, “every single person in the entire city was talking about that essay. It was like Renaissance Florence.” The real compounding was skill: “it took a while to get better.”
• Evidence for market efficiency: a famous blogger told Dwarkesh he discovers a genuinely new blogger “maybe once a year” — and the rest of the world finds them “maybe a week” later. Trenton: anything high-quality on AI “is almost invariably going to be shared around Twitter and read.”
• The playbook: the “Matt Levine of AI” niche is “totally open”; successful new media is “propelled by a single person’s vision,” not a collective; you’ll “feel like shit in the beginning”; and “what is the three months of blogging on the side really gonna cost you?” Sholto recalls an Annus Mirabilis post that Jeff Bezos retweeted, he thinks; Dwarkesh says that was probably his first big success. Trenton adds that writing doesn’t need a new insight, just to “express cleanly a set of ideas that they are already aware of.”
• Cold-start hacks for Substack newbies: podcasting (leverage guests’ takes and platforms) and book reviews — “a totally under-supplied thing.” He’s probably visited economist Jason Furman’s ~1,000-review GoodReads “on a hundred independent visits.”

4. Guest selection: big names don’t matter — Sarah Payne does

• The filter is the two weeks, not the hour: “The research is my life, and I wanna have fun while doing it.” He turns down influential people often. By far his most popular guest is Sarah Payne — a then-unknown scholar — “and then it’s Sarah Payne, Sarah Payne, Sarah Payne… I host the Sarah Payne Podcast where I occasionally talk about AI.” David Reich next; Satya and Zuckerberg trail. “Big names just don’t matter that much.”
• The real flywheel isn’t audience or reps (“if there’s no reward signal you’ll keep doing whatever you were doing before”) — it’s that good episodes earn access to people who teach him things: one China blog post “netted me an amazing China network.” Hence: “move to San Francisco? Yes. If you’re trying to do AI.”

5. Fast timelines: cancel the 401k, keep the podcast running

• Dwarkesh’s plan for fast timelines: if there are fast timelines, “there will be this six-month period in which the most important decisions in human history are being made.” He feels having an AI podcast during that time might be useful. His goal for the show is deliberately modest — “an epistemic tool,” because “it’s just very easy to be wrong”; talking to the Epoch folks changed his mind on the intelligence explosion within the past week.
• Revealed preferences: Trenton canceled his 401k contribution — “it’s hard for me to imagine a world in which I have all this money… waiting until I’m 60 and things look so different then” (with the hedge that he might restart it). Sholto changed nothing: “I just work all the time.” Dwarkesh put the ad money from his Zuckerberg interview into Nvidia.
• On who should run a frontier lab: Dwarkesh picks LBJ or Robert Moses for “making shit happen,” but concedes “great people are very rarely good people.” Sholto’s counterpoint: today’s actual lab leaders care about the moral side — which is why Dwarkesh is “skeptical of big grand schemes like nationalization”: “we live in a pretty good counterfactual universe, all things considered.”
• On funding new talent: grant applications for bloggers probably don’t work (the good ones get noticed), but paying people to move to SF for two months might — the model being MATS and the Anthropic Fellows Program, where Trenton says ROI “has been massive” via a rising-quality flywheel of mentors and hires.

6. Distribution and hiring: arbitrage exists for editors, not executives

• Dwarkesh’s rarely-asked-about take: people rightly focus on content but “consistently underrate… getting distribution right.” YouTube Shorts drove “at least half the growth of the podcast” — unpredictable in advance. Tweet-writing intuition: “write like you’re writing to a group chat of your friends rather than this formal whatever.” TikTok remains uncracked.
• His editor bench proves a global talent arbitrage: a farmer in Argentina, a freshman math student in Sri Lanka, a former Mr Beast-channel editor, and a Czech director. He says he rounded up a couple through a public clip-making competition; he also says he has roughly 10x’d their salaries. “It’s not even about the wages… it’s getting somebody who is really data-oriented.”
• The arbitrage fails for generalist roles: a year-long chief-of-staff search drew ~1,000 applications, and the hire came through a mutual-friend referral (Max, Trenton’s childhood best friend). Lesson: “the best people in the world don’t want to apply… you just gotta seek them out.” Trenton confirms hiring is “relentlessly the number one” issue for startup CEOs.
• His favorite history book, and its operating principle: the Caro LBJ biographies, and LBJ’s line to his debate students — “If you do everything, you’ll win” — going past the 20/80 point into “an unreasonable use of time.”