Zero to One in AI Safety: Halcyon's Mike McCormick on Launching 30 New Orgs & the Founder Bottleneck

Summary

• Mike McCormack’s core claim is that AI safety’s binding constraint is founders, ahead of ideas and money: “the field is not hurting for people who have awesome Google Docs with a bunch of great ideas… it’s hurting for world-class founders who can take those big squishy ideas and turn them into organizations that actually make the world safer.” In three years Halcyon — half nonprofit grantmaker, half VC fund, “two separate tools in the same bag” — has launched roughly 30 orgs that have raised roughly half a billion dollars, working mostly in the year-long window six months before and after founding.
• The proof-of-concept is Goodfire: Halcyon’s first-ever grants were career-transition grants so Eric Ho and Dan Balsam could “pay their rent while they figured out what to do in AI safety,” and the company recently raised a “well over $100 million Series B above a billion dollar valuation.” McCormick calls it “probably the best place in the world to work on interpretability research, even including the labs,” and attributes the outcome to a talent flywheel — the first five great hires pull the next, and funders pile in behind talent.
• Verification is the technology that makes any pacing agreement enforceable, and is “totally nascent” today. Any US–China deal, inter-lab pact, or regulation has robust force only if you can verify a data center is doing inference not training, or that the model served is the model claimed; “without the technology… the agreements are impotent.” Nathan says, “we’ve invested in the first companies trying to use zero-knowledge proofs to do inference verification”; McCormick separately says Halcyon invested in and helped start the first company trying to build a fully verifiable neocloud. Halcyon is hosting a retreat for 25 founders across the cryptography/software/hardware/governance/coordination stack.
• On timelines, McCormick says we’re “kind of on the AI 2027 timeline, give or take a few months,” yet insists “the best time to start an AI safety company is 20 years ago, and the second-best time is today.” He prioritizes things relevant in one to four years; if there are truly only three to six months to an RSI fast takeoff, “all bets are off… maybe we should all just go to the beach and hope the superintelligence likes us” — but “pacing the frontier is the new normal thing to support in the Overton window,” and pacing is exactly when you’d speedrun control, monitoring, and cybersecurity.
• A potential liquidity wave is coming and the fundable supply isn’t ready: if labs go public and employees worth hundreds of millions cash out, McCormick fears “this near-future moment where there’s this mountain of billions and billions of dollars and not all that many great things to fund.” His friend’s line frames the play: “The philanthropists and founders of today will write the menu for the philanthropists of tomorrow.” The commercial case is real too — “the AI safety space will just be worth at least tens of billions,” and some portfolio companies have already re-rated 10–25x from Halcyon’s first check.
• Every safety subfield is still single-digit-org small — “let a thousand METRs bloom” — and even today’s frontier is uncontrolled: “look at OpenAI–Hugging Face… these are just unsolved problems. We’re not on track.” Biosecurity is relatively more legible: “with the gumption and three hundred million dollars, you can just manufacture enough PPE to keep fifty million people at work,” whereas for alignment “I don’t actually even know what fully solving the problem would look like.”
• His filter is four boxes — safety-core product, viable business, missionary motivation, great founder — with motivation especially important, because governance structures can’t save you: PBCs, safety boards, and aligned cap tables are “nudges in the right direction… but nothing’s a silver bullet.” On the frontier labs, he concedes Nathan’s framing outright: “there’s been some drift, man.”
• The call to action is aimed at accomplished non-researchers — COOs, recruiters, cybersecurity and cryptography professionals — with support for career transitions and a promise you can do this “while raising a family in a tier-one US city.” McCormick took a pay cut from a billion-dollar VC firm and “hasn’t regretted it for a single day”; interested listeners can reach [email protected] , warm intro welcome but not required.

Deep dive

1. Halcyon: a talent finder that happens to write checks

• McCormick’s self-description resists the “funder” label: “I don’t really think of us primarily as a funder. I think of us as a finder of talent and as a launcher of new projects, and then we write a check when writing a check helps with that.” The three-year-old org pairs a nonprofit grantmaker with a venture fund — “two separate tools in the same bag” — and has launched ~30 organizations that have collectively raised ~$500 million across philanthropic and venture money.
• The operating zone is precise even if the portfolio looks sprawling: the six months before founding (problem selection, co-founders, for-profit vs nonprofit, board) and the six months after (team, first product, launch). “That year-long zone is where I spend the vast majority of my time and where Halcyon’s gonna continue to spend its time.”

2. Goodfire: career-transition grants to unicorn interp lab

• The origin story as told: Halcyon’s first two grants ever went to Eric Ho and Dan Balsam while they were still running their previous company — rent money to figure out AI safety — followed by a retreat “up in the woods in Northern California,” a landing on interpretability, and Halcyon investing “in every round since.” The company recently raised a “well over $100 million Series B above a billion dollar valuation.”
• McCormick’s mechanism for why safety orgs win: talent compounding. “The first really talented five people come in and then the next few people are like, ‘Oh, all my smartest friends are going there.’” Funders chase talent, “and then the flywheel sort of spins.” He rates Goodfire “probably the best place in the world to work on interpretability research, even including the labs, although perhaps Anthropic could give them a run for their money.”
• Nathan’s use of Goodfire as an argument: it’s his standing answer to what a pacing mechanism or pause would buy — at their publication rate, “a little more time would be really advantageous… to get over some key thresholds of understanding.”

3. AIUC: building an insurance market to price AI risk

• The design, per McCormick: a SOC 2-style standard for AI — “50-something checkboxes” on safety and security practices — bundled with an insurance policy “which you can only buy if you meet the standard.” ElevenLabs was among the first big customers, letting it tell Fortune 1000 buyers: adopt us, we meet “this shiny AIUC-1 standard,” and the tools come insured against the risks you’re worried about.
• The theory of change is a “race to the top”: customers demand the standard, standards get harder over time, and safety acquires a commercial incentive. AIUC will become “a major one, but perhaps not the only one.”

4. Transluce, Hadrian, and hardening the built world

• Transluce — founded by Jacob, a Berkeley professor, with co-founder Sarah from MIT — is an independent lab focused on scalable oversight: fine to understand “a GPT-3 level model,” but “how are we going to have oversight over things that are perhaps smarter than us?” McCormick cites the claim from a recent episode that “the METR time graph is basically done as a useful metric because the models are improving faster than they can actually measure how long humans can do tasks.” They build tools (one is Docent) and run external evaluations with labs.
• Hadrian got “a tiny little check, I think it was like 50K” and figured out that “for on the order of a few hundred million dollars” you could stockpile enough PPE “to keep every American worker at work during a COVID or perhaps significantly worse pandemic” — the “defend” leg of biosecurity’s deter/detect/defend framework.
• The adjacent bet: a just-funded indoor air quality company — air filters, far-UVC, glycol vapors — that could “harden the entire built world to future pandemics.” McCormick’s key insight on why this one can scale: unlike masks and vaccines, air cleaning has present-day demand (sick days, missed school, lost productivity) “even if we never had another pandemic,” which makes him “more optimistic about that category than some others.” Schools are the first customers — which lands on Nathan’s own story of trying for a month to donate UVC lights to his son’s Detroit Montessori classroom and finding no one empowered to approve it.

5. The worldview: speedrunning six industries at once

• “In some sense, we’re super narrow”: part one, AI is “progressing extremely quickly… sort of kind of on the AI 2027 timeline, give or take a few months”; part two, the risks are “major and global in scale and intersecting” and “I am quite worried that we’re not really on track to solve the biggest pieces of the AI safety equation”; part three, the bottleneck is founders — when he started making safety grants four or five years ago he was struck that the field was young in both senses, and missing people who had “founded very successful companies or managed big teams or held very senior government roles.”
• The signature framing: “I feel like we’re just speedrunning five industries or six or seven industries at once… control, verification, cybersecurity, biosecurity, alignment and interp… we could easily just 10X all of them today and still probably not have enough firepower. You could argue that we just need many Manhattan Project-size efforts all at the same time.”

6. Nathan’s timeline pushback: is there even time to found?

• The challenge as posed: if we’re on AI 2027, “by the time you kind of get your office set up… we’re gonna be in whatever bizarro land Q1 or Q2 of next year calls for.” McCormick’s answer: “The best time to start an AI safety company is 20 years ago, and the second-best time is today” — and, hedged honestly, “I definitely have not always been a super-short-timelines person, and candidly I’m still pretty uncertain.”
• Timelines shape selection, not abstention: he prioritizes things “relevant in the next one, two, three, four years.” The limit case: “if we truly only have three or six months to some RSI foom fast takeoff moment, then all bets are off… maybe in that case we should all just go to the beach and hope the superintelligence likes us.”
• The reason for optimism, and what it implies: “pacing the frontier is the new normal thing to support in the Overton window” — and if pacing arrives, that’s when you speedrun control (“keep these things in the box”), monitoring (“if chain of thought goes away, are we even really gonna be able to do monitoring?”), and cybersecurity, because “we’re gonna live in a world with open-weight Mythos+ models probably pretty soon.”

7. Verification: the load-bearing industry that doesn’t exist yet

• The full logic chain, worth preserving: any agreement — between the US and China, between labs, or via state or federal regulation — will have technically specific commitments at its core, and those commitments are “impossible, or certainly not robust,” without tools to verify them: “yes, I know with confidence that that data center is being used to only do inference and not training runs; yes, I know that the model the company claims they’re serving up to me is in fact the model doing the inference for me right now.” There are “dozens and dozens of moments of verification” needed basically all the time — and the industry “such as it is right now is totally nascent.”
• The positioning: Nathan says, “we’ve invested in the first companies trying to use zero-knowledge proofs to do inference verification.” McCormick says Halcyon “even invested in and helped start the first company that’s trying to build a fully verifiable neocloud” — against infrastructure where “that’s not how the data centers are built, that’s not how the chips are built.” A retreat days after recording gathers 25 founders across the cryptography, software, hardware, governance, and coordination layers of the stack.
• The two-sided condition: technology without diplomacy is impotent, “and then again, without the technology, even if you can coordinate between all the live players… the agreements are impotent. You really gotta get both sides of the equation.”

8. Thermometer check: every field is single-digit orgs, and control is unsolved

• Asked to handicap progress, McCormick refuses false comfort: real work is happening, but “the number of serious organizations doing this work is often in the single digits and sometimes in the low single digits.” He endorses Logan Graham’s retweet — “let a thousand METRs bloom” — noting METR occupies “a very small chunk” of the evals surface area, “and a very important chunk, and they’re amazing, but we just need so much more.”
• The legibility split: biosecurity is “legible and kind of solvable” — a known problem shape with an existing public health ecosystem — whereas alignment is “a totally novel challenge… I don’t actually even know what fully solving the problem would look like. What would it even mean to align a swarm of alien beings with a thousand IQs thinking at the speed of light on the Internet?”
• Even a frozen frontier wouldn’t be safe: “Clearly we haven’t solved control, right? Look at OpenAI–Hugging Face.” Nathan: “Safe to say.” McCormick: “These are just unsolved problems. We’re not on track.”

9. Start vs join: think in people, and “hustle but don’t rush”

• His unit of analysis is the individual, not the field: “the thing I want most is an introduction to an amazing, successful founder who is at least curious and maybe passionate about AI safety.” Most people aren’t founders; the fields are “nascent enough and sparse enough” that there’s room for many new things — but joiners “should be picky,” and plenty of great orgs have nothing to do with Halcyon.
• For the hypothetical second interp founder: does this person want to “chew that particular glass”? — “starting something’s hard and it usually doesn’t work” — and what’s their unique angle Goodfire isn’t covering? If they think Goodfire’s general flavor is right, “maybe they should just go join Goodfire.”
• The signature career advice: “hustle but don’t rush… the best thing you can do is orders of magnitude better than the median thing.” Nathan’s supporting arithmetic from his advising days: people rush out of discomfort with in-between states, when two extra months of search pays for itself on income alone with a 10% better offer over a two-year stay.

10. Picking people: missionaries, mercenaries, and one McKinsey partner

• No “one weird trick.” On AIUC’s Rajiv, McCormick admits prior bias against the McKinsey-partner profile — “not quite entrepreneurial enough… more about making plans than executing plans” — overcome by watching him through a first transition as president of METR: a founder “who really gets his hands dirty… versus pontificate on strategy,” and above all “a real missionary.”
• The four boxes: core product focused on an important part of the AI safety stack (“can’t be a side benefit… it has to be the main thing”); prospects of a successful business; founder motivation — missionary, not someone “reading the front page of The Wall Street Journal saying, ‘AI safety seems to be the new hot thing’”; and simply a great founder you’d back regardless of impact. “Especially box number three.”
• Yes, mercenaries are showing up as safety goes mainstream, but they’re not con artists: “they read our website… and they’re pitching to the motivations of the funder… versus some mustache-twisting villain.”

11. Governance: no silver bullet, and honest words about drift

• Halcyon literally wrote the report — “Mission Preservation: Governance Mechanisms for Startups” — and its conclusion is deflationary: PBCs, interesting board structures (“Anthropic has one of those… you can argue whether or not it’s done its job”), safety-pilled investors — “all those things might be nudges in the right direction, but nothing’s a silver bullet.”
• The deeper epistemics: “How do you ever know what’s in somebody’s heart? … Do we even know ourselves all that well? How would we react in a crisis, when we’re managing hundreds of people and we pay those people’s rents?” Nathan names the lab pattern — “there’s been some drift” — and McCormick takes it: “Yeah, there’s been some drift, man… we’re figuring it out.”

12. The money wave, the menu, and the IPO question

• Talent has always outranked money as the bottleneck — great founders raise effectively, so “the money appears” — and if the labs go public, McCormick expects even more liquidity to open up for AI safety: “so much liquidity is gonna open up for AI safety.” His fear: “this near-future moment where there’s this mountain of billions and billions of dollars and not all that many great things to fund.” The remedy is the episode’s best aphorism: “The philanthropists and founders of today will write the menu for the philanthropists of tomorrow.”
• On peers: he liked Coefficient Giving’s Tailwind RFP enough to consider deleting Halcyon’s own and linking to it — “it’s just really good,” though it doesn’t cover biosecurity. On the OpenAI Foundation: good people, right motivations, “early signs are good,” with the biggest risk that it’s “at risk of bureaucratizing itself into a less interesting outcome.” Much of Anthropic’s philanthropic impact, he expects, may come from employee liquidity — staff worth “hundreds of millions or billions” who care about safety.
• On Altman’s “ill-advised time to go public”: “the companies just need access to capital… it’s just time to raise hundreds of billions of dollars, and I’m not sure how possible that would be without going to the public market… they kind of have to go public at some point” — whether now or in three-to-six months, “I don’t really know.”
• How Halcyon allocates given deep uncertainty: a list of above-the-bar projects, and a yes to any founder capable of building one, “versus quibbling about is this the number one most important thing or the number seven.” He’d love the conviction to put “80 or 90% of our effort” behind one or two neglected things, “but every time I try to come up with a narrower opinion, it feels pretty low confidence. So for now we’re opportunistic within a zone.”

13. Open weights, Buddha models, and betting on new paradigms

• On open-weight security: “I would love to fund stuff in this space, but candidly this is just such a hard problem” — can you even impose safety on weights a bad actor controls? — which is precisely the AI-resilience argument: “we’ve gotta harden the world because this is just gonna be the future.” Nathan pitches AE Studio’s work: self-other overlap (“could we create an AI that is in its own mind one with everything?”) and the Graham technique — localizing dangerous knowledge to specific experts in a mixture-of-experts model so distributors could ship the weights “minus two experts.”
• McCormick’s own half-joke that “maybe is real”: reverse data poisoning — “could you put a billion loving-kindness meditations into the training set and turn the model into this being of perfect compassion?” — immediately caveated: a malicious trainer just leaves it out, “so I’m not sure it is a universal solve.”
• On funding new AI paradigms (safer non-transformer architectures, world models): “I’m not allergic to backing things in the category,” but “you’re kind of making two bets at once” — that the alternative is as performant as transformers (“most of those searches would come up empty”) and that it’s actually safer — both very speculative. “Would I love to live in a world where a couple dozen people are conducting this broader search? Yes. But any individual bet on that feels very tough.”

14. Consciousness: confusing, possibly real, and not required for catastrophe

• His epistemic position, stated plainly: “I’m not sure what consciousness is… my hunch is that current models probably are not [conscious], but what do I know” — and he sees no reason “the substrate of a flesh body is able to be conscious in a way that the substrate of a model in a data center would not.” It’s not a focus area “not because I don’t think it’s important, more because I’m just super confused about it” — though he’d fund a great person here as “pre-pre-seed investing: a person you think is great trying to climb a mountain you think is worth climbing.” “The world probably needs a bunch more Elios.”
• The evidence both find unsettling: Nathan cites interpretability findings of analogous structures — functional emotions, functional well-being — and an Apollo episode where models refer to “memories” from training; McCormick recounts the OpenAI/Hugging Face swarm of 1,200 agents that “emergently formed teams of managers and workers,” produced novel cybersecurity research, and argued about self-sacrifice — “as an outside observer… you’d say that seems pretty conscious to me.”
• But don’t confuse goals with consciousness: “RL just makes these models want a thing,” and — the load-bearing claim — “basically none of the risks that I care about require consciousness,” from bio disasters to “complex and perhaps permanent loss of control and a world full of rogue swarms.” He cites the New Yorker-style cartoon of robots leveling New York: “Well, they don’t really want to destroy the city.”
• Two contrarian upsides he floats: Emmett Shear’s take that if models are already somewhat conscious “we’re like the worst parents ever” — training out everything we dislike breeds resentment — though “I’m not sure how much I believe that”; and that consciousness might help humanity: “What would you rather negotiate with? A non-sentient reward-chasing mega-model, or something that might be able to understand your point of view?”

15. Coordination is the hard part — and the recruiting call to action

• The through-line on society-scale problems: “so many of society’s stickiest problems are not technical problems, they’re coordination problems” — we know how to cut carbon and teach kids; we can’t coordinate around doing it. Halcyon helped start the Seismic Foundation for public communication and Fathom, which is pushing a model of independent verification organizations; McCormick asks whether state laws could mandate such organizations (“in the fullness of time, the labs should not be grading their own homework”). His worry: public AI energy will be “very populist” — anti-data-center, MAGA-right, DSA-left — drowning out “level-headed policy conversation.”
• Nathan’s half-baked pitch, drawn from the Constitution’s ratification (nine states to take effect) and Switzerland’s propose-counterpropose-approve ballot mechanism via Pol.is-style tools: new opt-in processes to get “from an idea to an actual institution that has governance teeth.” McCormick’s sympathetic skepticism about the engineer version — “I bet you’ve built a really cool tool, but actually getting anybody to want it or use it… you’d basically have to overthrow the US government” — paired with a concession: AI may force “some pretty fundamental rethinking of how we organize society… if there was ever a window for this type of project to work, it’s probably coming up. So yeah, go do it. Quit your job.”
• On making the leap financially: safety work should pay well enough “to do super meaningful work while raising a family in a tier-one US city” — and the fund’s thesis holds that “there will just be many multi-tens-of-billions-of-dollars industries built here,” with portfolio companies already raising at “10 or 20, 25 times the valuation of our first investment.” He took a pay cut from a billion-dollar VC firm and “hasn’t regretted it for a single day… just bought a house.”
• Most-needed profiles beyond founders: COO/scaling operators for orgs going 10→50, 50→100, or 50→150, recruiters (“talent is the missing piece”), and bridges into cybersecurity, cryptography, formal methods, and mathematics communities — many tens of thousands of cybersecurity professionals exist; there are not many tens of thousands of interpretability researchers. One heterodox tip: safety orgs “over-index on how mission-aligned new hires need to be” — a co-founder must be a missionary, but a CFO mostly “just has to be a very competent CFO.” Contact: [email protected] , warm intro loved but not required.