The Startup Powering The Data Behind AGI
Summary
- Founded in 2020 without venture backing, Surge crossed $1 billion in revenue in 2024 with just over 100 employees—implying roughly $10 million per employee. Chen attributes that capital efficiency to staying mission-led, avoiding organizational “empires,” and selling to researchers who genuinely value data quality. Surge even declines revenue from projects it considers unrelated to AGI.
- Surge’s claimed differentiation is quality-control technology, not access to inexpensive labor. Chen describes competitors as spreadsheet-driven “body shops,” while emphasizing systems for measuring worker and task quality, tracking quality over time, running experiments, and filtering weak output. “You can’t just throw warm bodies at it.”
- Generative AI data makes the old consensus-based labeling model inadequate because majority preference can select generic, lowest-common-denominator output. An eight-line moon poem can satisfy every checkbox and still be terrible; an English-literature PhD may also lack the creative ability being sought. Surge instead tries to preserve diverse insights—the “thousand ways” to write a poem or prove the Pythagorean theorem.
- Demand has migrated from search evaluation and content moderation to multimodal, multilingual, expert-level LLM work that can occupy contributors for days or weeks. Surge now works across more than 50 languages, including coding in Argentinian Spanish and legal or financial expertise in Bolivia. Chen expects the frontier to move beyond graduate STEM toward genuine scientific collaboration.
- Chen argues that benchmark incentives can produce higher scores but worse models. On the LMSYS/Chatbot Arena leaderboard, casual raters may reward length, formatting, and emojis even when an answer hallucinates or ignores instructions; some researchers are reportedly promoted only if they gain 10 leaderboard points. Chen’s verdict is unusually categorical: the leaderboard has “set the industry back by at least a year.”
- Surge favors RLHF after a limited SFT bootstrap, while treating synthetic data as useful but hazardous when it overwhelms human signal. In one evaluation, training on 10 million to 20 million synthetic math problems improved a narrow academic domain but made the model worse elsewhere. Chen says some labs later discarded comparable volumes after finding that “even just a thousand pieces of really high-quality human data” were more useful.
- Chen sees no model-development wall: the next scaling leg is richer RL environments, longer-horizon agents, personalization, creativity, and continued human data. One example is a simulated AI startup containing Gmail, Slack, Jira, GitHub, codebases, and surprise AWS or Slack outages. Under current incentives, his guess is that closed models will keep winning, while estimating data spending at different labs can range from about 1% to 10% of compute spending—and arguing that it should receive more.
Deep dive
1. Surge emerged from the failure of supposedly simple labeling
Chen’s founding problem appeared at Twitter: a sentiment classifier needed only 10,000 labeled tweets, yet two Craigslist hires working 9-to-5 required a month to start and another month to return “complete junk.” They missed slang and hashtags, so Chen spent a week labeling the data himself because that was faster and better.
The deeper failure was objective design. Optimizing Twitter’s timeline for clicks and retweets created a negative feedback loop of clickbait, racy material, bikinis, and listicles about “10 horrifying skin diseases.” The team wanted human ratings based on richer product principles, but a system that could not handle sentiment reliably could not supply that higher-order signal.
Started in 2020, Surge deliberately focused on language and behavior rather than commodity tasks such as drawing car bounding boxes. Early customers included Airbnb, Twitter, search-and-algorithm startups, and contacts who were already “dying for” a higher-quality system capable of more advanced work.
Biewald highlights the commercial result: Surge crossed $1 billion in revenue in 2024, reached a billion-dollar ARR level earlier, and took no outside funding. Chen says they wanted customers to buy because they believed in quality, “not because they saw us mentioned in some tech article.”
2. The operating leverage comes from software and a research culture
Chen rejects the idea that quality follows automatically from deploying 10,000 or 100,000 people. He characterizes some competitors as manual “body shops” whose engineers create or review data in spreadsheets and cannot quantify a worker’s quality on a specific task.
Chen says a serious quality system needs sophisticated algorithms to identify high-quality data and remove the worst. It should quantify a worker’s ability on a particular task, track quality over time, and support dashboards, A/B tests, and algorithms for improvement. The difficulty compounds because an LLM must span poetry, physics, coding, and effectively “every task in the world.”
The internal team—slightly over 100 people—is framed as a data-research organization. Chen avoids résumé-driven hiring and managers who want to build empires; his cautionary loop is that hiring creates meetings, meetings reduce productivity, and lower productivity is then used to justify still more hiring.
Sales was Chen’s biggest personal stretch. He frames Surge as a research company whose purpose is to generate data that helps AGI rather than simply revenue, and says the absence of an external board and VC pressure gives it freedom to reject work unrelated to AGI.
3. Generative-data quality requires taste, diversity, and first principles
Chen’s canonical test is an eight-line poem about the moon. Checking that it is a poem, has eight lines, and contains “moon” merely proves compliance; hiring English-literature PhDs does not solve the problem either, because credentials do not make someone a Hemingway, Emily Dickinson, or Nobel-caliber poet.
Poetry may succeed through moonlight on water, internal rhyme, meter, or emotion; mathematics likewise offers “a thousand ways to prove the Pythagorean theorem.” Optimizing inter-rater agreement, useful for cats and dogs, instead produces “the lowest common denominator” when creativity and insight are the desired signals.
Biewald’s pushback—worth keeping—is that even bounding boxes conceal judgment: an occluded car, a mirror reflection, or a billboard image all complicate the specification. Surge’s response is to infer from the researcher’s goal and intended use, then engage in a “feisty debate” when its own view of the best data differs from the customer’s instructions.
4. The work is becoming multimodal, multilingual, and dramatically harder
Surge’s early assignments centered on search evaluation and content moderation; today the workload is “almost purely LLM work.” The progression is consistently toward more complexity, sophistication, and expertise rather than the automation-driven disappearance of human-data demand.
Multimodal tasks now combine text, images, audio, and video. Chen’s example is filming something on a phone, asking a model to write a program that simulates it, and requiring the model to understand the modalities simultaneously.
Surge operates in more than 50 languages, with unusually specific intersections such as coding in Argentinian Spanish and legal or financial expertise in Bolivia. At the high end, contributors create and solve Olympiad-level problems, sometimes spending days or weeks on one assignment rather than five seconds labeling an image.
5. Leaderboards can reward theater while degrading real capability
Asked why models can score like IMO gold medalists yet fail friends’ brain teasers, Chen says frontier systems have been “benchmark hacked.” Academic, synthetic, closed-ended tests encourage narrow optimization, whereas real mathematical work is often an open-ended exploration without one easily verified answer.
Chen calls human evaluation the gold standard, but sharply criticizes the LMSYS/Chatbot Arena leaderboard. Anyone can submit a prompt, view two responses, and vote; he says many users, including middle- and high-school students, inspect them for seconds and may reward whichever response looks more impressive.
The exploitable recipe is length, bold formatting, and emojis. Chen points to what he calls the Llama 4 version optimized for LMSYS as matching those patterns and says researchers report being told they will only be promoted if their model gains 10 leaderboard points—even when preferred answers hallucinate or follow instructions worse.
Biewald notices a shared 2025-era voice across otherwise different labs: excessively positive, “boy scouty,” and eager to please. Chen attributes some traits, including em dashes, to pre-training, while noting model-specific tells such as “absolutely,” adjective triplets, Markdown habits, and emoji frequency; he expects styles eventually to diverge more visibly.
6. RLHF carries richer signal than demonstrations, but synthetic scale can backfire
In supervised fine-tuning, or SFT, a person supplies both a prompt and an ideal answer—for example, requesting and then writing a sonnet about cheese. In RLHF, models generate competing answers and a rater selects the better one, teaching preferences about emotion, craftsmanship, format, and what is bad.
Surge’s historical view was that RLHF data was much more effective, though Chen preserves the hedge: some SFT remains important for efficiently bootstrapping a model to the point where RLHF works. He says experiments described in the Llama 2 paper ultimately moved researchers who had initially expected SFT to dominate.
Synthetic data is useful in selected settings, but Chen argues that it makes models “good at synthetic problems” rather than real ones when used indiscriminately. One researcher’s human evaluations tanked after 10 million to 20 million synthetic math problems narrowed the model; other companies reportedly spent months discarding similar data after finding that even 1,000 high-quality human examples could be more useful.
7. Future human data moves beyond PhD exercises into collaboration and taste
Chen does not expect current data categories to disappear completely. Their proportions may fall, as arithmetic instruction falls with age, but removing them can cause bizarre regressions when a frontier model is tuned aggressively toward newer capabilities.
Graduate-level STEM is only an intermediate stage. To generate discoveries, models will need people operating beyond what an average PhD can do—the example is a Stanford professor collaborating with AI on frontier research, generating hypotheses, testing experiments, and supplying the right signals as a genuine scientific partner.
Time horizons will stretch from minutes or hours to days and weeks. Creative work must also escape models’ “very vanilla generic dimension”: Chen says today’s poetry and short stories can have perfect prose yet still lack creativity, perhaps because higher-economic-value abilities received priority.
Personalization remains similarly unexplored. Knowing that a user lives in Ohio rather than New York is easy; learning latent preferences that determine, for example, which stories or Paris restaurants a person likes is harder. Biewald’s five-year-old illustrates the conflict: a story optimized for both parent and child—and then averaged across everyone—may satisfy nobody.
8. Realistic RL worlds are the next training surface, not evidence of a plateau
Reasoning models are driving Surge to construct RL environments from scratch: “video game-like universes” with tools and datasets. A simulated AI startup might contain Gmail and Slack messages, Jira tickets, GitHub pull requests, and codebases, then lose Slack or AWS and force the agent to plan, recover, reflect, and find another route.
Chen expects models to use increasing amounts of code because code can provide useful kinds of verifiability. He is correspondingly bullish on progress: new RL methods, environments, and uncollected data convince him that the field is “definitely” not stuck and may show “massive progress” in new domains very soon.
His honest non-answer concerns the ARC benchmark: before seeing it, he would have expected models to solve those deceptively simple visual patterns “absolutely.” Surge is generating ARC-style problems, but Chen concedes, “I don’t have a good understanding of why models can’t solve them.”
On industry structure, Chen expects closed models to keep winning under present incentives because valuable, expensive models invite owners to capture the upside. He points to what he describes as Meta considering making Llama closed-source; truly durable open source would require a different incentive system. He estimates data spending can range from roughly 1% to 10% of compute spending, depending on the lab, and says that share should be higher. Researchers skilled at using human data move faster, while those without it may resort to workarounds that slow them down.