Zuckerberg’s Anti-Doom Fantasy + Finally an A.I. Detector That Works + A.I. Math
Zuckerberg’s Anti-Doom Fantasy + Finally an A.I. Detector That Works + A.I. Math
Summary
- Casey Newton reads Zuckerberg’s 6,500-word “The Future Is For Everyone” manifesto as a policy wish list dressed as optimism: accelerated data-center construction, maintained chip export controls (which advantage Meta’s open-weight models over Chinese rivals), reduced “training data restrictions” to fight creator lawsuits, and legal protection for distillation. The timing matters — Meta is “getting its ass handed to it in court case after court case,” including a New Mexico ruling ordering an extra $567M into a teen mental-health abatement fund on top of $374M in civil penalties, a “public nuisance” finding comparing Meta to a polluting factory, and a 90-hour-per-month cap for under-18 users.
- Kevin Roose raised his probability that Meta actually builds superintelligence from ~1% six months ago to ~10% now, citing improving training runs since the Meta Superintelligence Labs reorg — and that improvement worries him. He resurfaced a former DeepMind executive’s taxonomy: tool companies and AGI companies can both be fine, but “the real danger is if you have a company that thinks it’s designing tools but is actually designing superintelligence” — once said of Google, now fitting Meta.
- Casey’s central metaphor, via the House of the Dragon finale: Zuckerberg’s “personal superintelligence for everyone” is “a little bit like giving a dragon to everyone.” Defender-catches-up equilibrium may hold in cybersecurity but not bioweapons, where defenders may take longer to catch up to a novel pathogen — a gap Zuckerberg acknowledges in the essay and then punts on.
- Both hosts agree AI positivity is not a messaging problem: “It is delivered via real experiences that human beings are having,” Casey argued — cures, higher pay, kids excelling — a chasm no leading lab has crossed. Kevin’s compression: “ship it or zip it,” and Zuckerberg is “possibly the worst messenger” — “from the people who brought you Cambridge Analytica comes superintelligence.”
- Kevin openly reversed himself on AI text detection — he’d argued on the show it was “basically worthless,” but independent studies now suggest Pangram is genuinely accurate. CEO Max Spero explained the shift: ditching the flawed perplexity metric for a trained classifier that pairs real pre-2022 human writing against LLM imitations and aggregates “a bunch of really weak signals” — good enough to flag a “random” number string as AI because LLM sampling isn’t actually random.
- Spero’s investment-relevant thesis: bot traffic has just surpassed human traffic at roughly 50/50 and may reach “99% bot traffic” within a decade, so platforms must “discriminate in favor of humans” — and detection demand is being validated by regulation, with Anthropic agreeing to watermark all its text under the EU AI Act. He disputes Ben Thompson’s take that watermarking degrades outputs: entropy-constrained watermarking (per Google’s SynthID sampling-perturbation approach) shouldn’t.
- The new “Running the Numbers” segment carried three data points: an Anthropic employee’s Claude session made real progress on a side problem involving the Riemann zeta function via pep-talk prompting (“You got this”) — new-knowledge production, though explicitly not a proof; Airtable was acquired by Bending Spoons at $1.29B enterprise value versus an $11.7B 2021 mark, which Kevin reads as AI eating SaaS margins broadly; and AI wealth is repricing dating markets, from some Korean chip workers’ $400–500K bonuses to “Anthropic goggles” in San Francisco.
Deep dive
1. Zuckerberg’s manifesto: personal superintelligence for everyone is a dragon for everyone
- The document: 6,500 words titled “The Future Is For Everyone,” promising that “in the next few years, people will be able to use superintelligence beyond human capacity” — every Meta user getting “an exceptionally capable personal agent,” accessible through glasses, illustrated by Zuckerberg’s own use cases: prototyping ideas, sleep monitoring, and personalized weekend baking recipes with his daughter.
- Casey’s framing, fresh off the House of the Dragon season-three finale: the show opens with one family holding the super-weapon, then a schism, then bloody civil war. Zuckerberg’s answer to AI risk — maximal proliferation — “is a little bit like giving a dragon to everyone… some people are gonna be launching cyberattacks and engineering novel bioweapons.”
- Kevin’s dissection of the rhetorical move: Zuckerberg argues no universally aligned AI can exist because values differ, so everyone should get a superintelligence tailored to their own. “Sounds great, but in practice, giving the Chinese Communist Party an AI superintelligence that obeys their values would allow them to commit atrocious acts against their own people.”
- On offense/defense equilibrium, Casey’s key carve-out: it may hold in cybersecurity, but “if I release a novel pathogen… it is just going to take the defenders a little bit longer.” Kevin notes Zuckerberg actually acknowledges bio may break the equal-tools logic — “he kind of punts on it… I don’t think he’s fully naive, but he just doesn’t have a good answer for that part yet.”
2. The real purpose: a policy wish list timed against courtroom losses
- Casey’s inventory of what the essay actually asks for: accelerated data-center buildouts (feeding Meta’s potential neocloud business), maintained export controls on advanced chips — notable from “Mr. Open Source,” since it advantages Meta’s open-weight models over Chinese alternatives — reduced “training data restrictions” amid creator lawsuits, and legal protections for distillation. “Buried inside this very positive, happy vision of AI is just a series of policy requests to help Meta as a business.”
- The timing: state after state is coming after Meta on teen safety. Last week a New Mexico judge ordered an extra $567 million into a teen mental-health abatement fund atop $374 million in civil penalties, ruled Meta’s platforms a “public nuisance” — comparing them to factories with “the psychological harm and exploitation of children as the pollution it emits” — and imposed a 90-hour-per-month cap for under-18 users plus new chatbot restrictions.
- The historical rhyme Casey won’t let go: Zuckerberg once wrote happy manifestos promising that connecting everyone on Facebook would yield “more democracy than you’ve ever seen before.” “Suffice to say, that didn’t really work out. Now we get the exact same maximalist, universalist framing — which of course also maps 100% to Meta’s business interest — but this time in the context of AI.” His self-aware confession: “I wish that I was born yesterday and could just believe everything that I read, but Kevin, I know too much.”
3. Can Meta actually build it? Kevin moves from 1% to 10%
- Kevin’s updated odds: he’d have given Meta “a 1% chance of creating superintelligence six months ago, and now I’m up to maybe a 10% chance.” Training runs launched after the Meta Superintelligence Labs hiring spree are coming online with “pretty impressive results” — not a frontier lab, but showing rapid improvement.
- Casey concurs: “The first rule of Mark Zuckerberg is never count out Mark Zuckerberg.” His AI contacts rate Meta’s past-year moves highly, particularly reassigning engineers to create reinforcement-learning training data — “the Meta engineers didn’t really love that,” but AI people say it will give Meta something valuable.
- The load-bearing anecdote, from a former DeepMind executive Kevin spoke with years ago: there are tool companies and AGI companies, and either can be run responsibly — “the real danger is if you have a company that thinks it’s designing tools but is actually designing superintelligence,” because it will never treat the technology with “the correct sense of reverence and suspicion.” Said then about Google; Kevin now thinks Meta may build “something extremely powerful with this very naive attitude” that these are just recipe tools.
- Casey’s coda on culture: a company that treats everything as existential competition and gives societal externalities “short shrift” — “I do not trust these people to rank a list of viral dances for me without it corrupting my mental health. Once you give these things access to novel biotechnologies, it starts to get pretty worrisome.”
4. AI positivity is earned by shipping, not by essay
- Kevin’s verdict on the messenger: “Mark Zuckerberg is possibly the worst messenger for the AI industry on all of this” — if he thinks a sunny vision will stop data-center protests, “he is badly mistaken.” Casey’s tagline: “From the people who brought you Cambridge Analytica comes superintelligence.”
- Casey’s positive case, worth quoting in full: real AI positivity is “new medical advances powered by AI, diseases cured, new jobs created, old jobs paying more money, kids excelling in education… crucially, it is not delivered via manifesto. It is delivered via real experiences that human beings are having, and so far, that has just seemed to be a chasm too great for any of our leading AI labs to cross.” Kevin’s compression: “Ship it or zip it.”
- One grudging credit: the manifesto passed AI detectors and reads like Zuckerberg actually wrote it — “I never once thought I was reading slop,” Casey said, though “to what extent he believes everything in it… I guess I don’t know that.”
5. Pangram’s technical break: classifiers over perplexity, and Kevin’s reversal
- Kevin’s on-air change of mind: “I have argued before on this show that AI text detection is basically worthless… but in just the last few months, there’s been more and more evidence that at least Pangram, and probably some of these other tools, have gotten quite good.” Casey now takes a “100% AI-generated” Pangram screenshot seriously.
- Max Spero’s explanation of what changed: legacy detectors used perplexity — AI text is “suspiciously smooth,” but so is any memorized document like the Declaration of Independence, and English-language learners write low-perplexity text too, generating false accusations. Pangram instead trains a classifier on pairs — a real seventh-grader’s Moby-Dick essay against an LLM’s imitation of one — on a clean, all-pre-2022 human corpus.
- What it detects isn’t em dashes or “not just X but Y”: it’s “combining a bunch of really weak signals” across a document — consistent word-choice decisions where AI is mode-collapsed and humans have “a wider decision tree.” The party trick illustrating it: ask ChatGPT or Claude for random numbers, and Pangram flags the string as AI — “they’re not actually random,” the model’s sampling biases leak through even with no such training data.
6. The cat-and-mouse: humanizers, new model releases, and the false-positive trade-off
- On blind spots, Spero conceded the asymmetry is deliberate: Pangram is “very much tuned to minimize false positives, so we’re not making false accusations” — which means false negatives exist, like tech writer Alex Heath’s AI-assisted newsletter scoring as human. The payoff: when Pangram says 100% AI, “you just know, okay, this is probably slop.”
- Humanizers — typo injectors, zero-width Unicode characters invisible to readers but seen by models, wholesale paraphrasers — are a real cat-and-mouse: “we’re always picking up the latest humanizers and training on them for our next model.”
- New frontier releases cost “slightly lower recall” for a few weeks until retraining, but generalization within families holds — “if Pangram has seen GPT-5.4 and GPT-5.5, then 5.6 Sol coming out is not a huge surprise,” and it caught writing samples from Mythos in a system card despite not having seen Mythos before. His durability argument: labs apply preferences — good writing, correct code, correct math — instead of average next-token prediction, “and these preferences are largely what Pangram is able to pick up on,” even if surface style adapts.
7. Why narc at all: a 99%-bot internet needs pro-human discrimination
- Spero’s answer to the “you wouldn’t detect spell check” objection: “AI as a tool is probably the wrong abstraction. It’s a little bit closer today to AI being an employee or another individual that you collaborate with.” Bot traffic has just surpassed human traffic — “about 50/50” — possibly heading toward “99% bot traffic” within a decade, so “as humanity, [we need to] discriminate in favor of humans,” including algorithmically, because AI short-form video “will activate neurons and engagement in a way that a real human video cannot.”
- The Substack integration has been “very controversial”: AI-assisted writers fear “witch hunts” now that audiences can see. Casey’s dry gloss: “Now that they know what it is, they don’t like it. Let’s go back to the time when they didn’t know that.”
- On whether normal people care: “I would totally care if a note from my friend was AI-generated. That would seem like such a big violation of trust.” But Spero also flagged where norm-setting overshoots — the Hank Green pile-on “felt like it just went way too far,” since Green was honest about AI-assisted research; the hosts noted social media rewards the effortless “LOL, AI-generated” dunk.
8. Watermarking arrives via the EU — and Pangram goes multimodal
- The news peg: Anthropic has agreed to watermark all its text to comply with the EU AI Act. Spero calls it “pretty huge” validation — an additional verification layer that can eliminate the “one in 10,000 false positives” doubt — but says watermarks have limits Pangram will fill.
- Mechanics, per the current state of the art (Google’s SynthID): the watermark perturbs the token-sampling algorithm in a reverse-engineerable way. Against Ben Thompson’s take that this makes model outputs worse, Spero’s rebuttal: work only “within the entropy available” — skip tokens where the watermark cannot be applied, as can happen in code when there is effectively only one correct token — and quality shouldn’t degrade; the trade-off is that some tokens may go unwatermarked.
- Roadmap: image detection in research preview, which Spero claims “wins on basically all of the public academic benchmarks” against frontier generators like GPT Image by reading pixel-level generation patterns now that “you can no longer just count the fingers”; video detection reportedly in progress; and the Chrome extension already pulls Google Doc revision history so users can replay writing and check big pastes for educators. On Zoom-scam identity verification: the models come first, then “bringing it to where people work.”
9. Running the Numbers: Claude’s Riemann progress, Airtable’s 89% haircut, chip-nerd dating
- The math beat: Anthropic employee Jared Sumner — a non-mathematician — set Claude on the Riemann hypothesis while jogging, and a day and a half later it had made progress on a side problem involving the zeta function, coaxed by pure pep talk: “You need to take a big leap of faith in your capabilities… You got this.” Casey’s read: “This feels like the production of new knowledge to me,” and the unreleased model behind it “can do a lot of other things — some good, probably some scary.” Kevin’s caveat, per Anthropic’s own careful framing: “We did not solve the Riemann hypothesis.”
- SaaS math: Airtable, valued at $11.7 billion in the 2021 peak-COVID SaaS mania, was acquired by Bending Spoons at a $1.29 billion enterprise value. Kevin’s mechanism: expensive enterprise subscriptions plus customers thinking “maybe I can make a free version of this” with AI — a pattern he expects across early-2020s enterprise software. Casey’s rules: “If your business is a fancy spreadsheet, you are in for a rough time,” and if Bending Spoons — the Italian, newly public acquirer of “zombie tech brands” — buys your product, “you in danger, girl.”
- Dating math, via a Wall Street Journal A-hed: Samsung and SK Hynix “chip nerds” now dominate South Korea’s dating scene, with some workers receiving $400–500K annual bonuses — one Samsung engineer, Annie Kwon, 26, coupled up with a fellow Samsung semiconductor-division employee for “double income”; another woman saved up to buy her Samsung boyfriend a mini PC in return for his roughly $450 Nintendo Switch 2 gift. Kevin reports the same repricing in San Francisco — “Anthropic goggles”: “Is that boy really cute, or are you just wearing Anthropic goggles?”