Summary

LLMs can turn algorithm design from artisanal trial-and-error into a scalable search process, acting as a “very strong and intelligent mutation operator.” DiscoPOP has models propose short preference-optimization objectives, test them, and feed results back into context, replacing random perturbations across millions of parameters with structured exploration informed by code, mathematics, physics, and other fields. The result is a potential automated R&D layer that turns money and compute into useful algorithmic insights.
The economic case does not require machine researchers to outperform the best humans one-for-one. Robert Tjarko Lange argues that even a system “50% worse than what a human would do” may win through vastly greater throughput, while Chris Lu currently places the models around undergraduate or first-year-PhD level. Humans remain supervisors and taste-makers, choosing important problems and filtering a torrent of technically valid but uninteresting output.
DiscoPOP’s most intriguing result is a non-convex preference loss whose discontinuity may make it robust to noisy data. Chris hypothesizes that a local optimum can absorb bad observations while the global optimum handles the rest, although he stresses that this is difficult to prove rigorously. The output is still only roughly 5–8 lines of readable code, including an unusual exponential term that appears weak alone but useful in combination.
Model collapse is a real concern, but the speakers do not think recursive AI research must exhaust its creativity. Their proposed defenses are human-generated inputs, randomly pairing unrelated fields, evolutionary crossover, quality-diversity methods, and ever-growing archives of stepping stones—deliberately injecting “external entropy.” The bottleneck shifts toward diversity mechanisms, selection, evaluation, and maintaining productive search distributions.
Specialized smaller models may capture meaningful portions of the optimization market without frontier-scale inference costs. In Lange’s black-box experiments, smaller Llama 2 and PaLM 2 models sometimes performed surprisingly well, while the method could handle gradient-free problems up to roughly 50 dimensions. Tokenization and representation mattered enormously: integer discretization initially beat floating-point inputs, although newer frontier models were already eroding that constraint.
Automated Design of Agentic Systems extends the search from individual losses to entire software organizations. Cong Lu’s system can discover long, non-intuitive workflows involving initial solvers, refiners, specialized critics, debate, tools, and code; an agent optimized cheaply with GPT-3 on math transferred to GPT-4, other math domains, and even literature tasks. The upside is general orchestration, but runtime, API cost, degenerative loops, and benchmark overfitting must become explicit objectives.
The AI Scientist is a proof of concept for chaining ideation, coding, experimentation, and writing—not yet a substitute for mature scientific judgment. Its current process is too linear: it may run one ablation and write the paper even when the result is negative, whereas Lange says human science is closer to 80% hypothesis rejection and 20% consolidation. The next capability jump depends on retaining failed experiments, revising causal hypotheses, and learning from the much larger “tip of the iceberg” that publications omit.
If “money compute API calls equals paper,” the scarce asset becomes trustworthy filtering rather than paper production. The guests envision AI-only conferences, model reviewers, debate, and human validation at the top of the funnel, while questioning whether conference acceptance remains the right unit of contribution. Their preferred metric is closer to diffusion over time—whether an idea changes subsequent work—alongside verification, reproducibility, evaluation, and human-readable interfaces.

Deep dive

1. Preference-optimization research becomes an executable search loop

Chris’s elevator pitch starts from an uncomfortable observation: preference algorithms are supposedly designed through expert mathematical judgment, but much of the actual process is “a lot of trial and error.” Because LLMs have absorbed mathematical intuition and can write code, they can perform far more trials across a broader search space.
Lange describes the loop as an evolutionary black-box optimizer. The model names and explains a candidate PyTorch-style objective, the system trains with it and evaluates downstream preference performance, and that result returns to the context so the next proposal can build on what succeeded or failed.
Earlier meta-evolution work encoded a loss as a neural network and randomly perturbed millions of parameters across populations numbering in the thousands. Most perturbations produced nothing useful; LLM proposals instead offer “much more structured exploration” because pretraining supplies a prior over plausible programs.
The resulting model is not merely a code generator but, in Lange’s phrase, “a type of very strong and intelligent mutation operator.” It can combine smoothing, regularization, and other concepts drawn from machine learning, physics, chemistry, or economics rather than searching locally around one researcher’s reading history.

2. Creativity may be interpolation across a space too large for humans to search

The host’s challenge is whether a model bounded by its training distribution can produce paradigmatically new ideas. Lange answers with Picasso: he was “a child of his generation,” finding Cubism within a “convex hull” that also contained artists such as Salvador Dalí and Joan Miró.
Lange’s point is not that LLMs perform unlimited extrapolation. It is that interpolation across an enormous, high-dimensional corpus may already connect concepts no individual researcher has encountered together: “Interpolation at this point…is already quite a lot in terms of value that we can bring to society.”
Chris adds that creativity could be encouraged more deliberately through temperature, beam search, or other sampling choices, though DiscoPOP did not explore these thoroughly. The system already reasons sequentially—trying an idea, inspecting failure, forming an explanation, and mixing the surviving components into its next candidate.

3. DiscoPOP finds a strange loss that conventional taste might reject

Correcting the host’s description, Chris emphasizes that DiscoPOP is not convex: it contains a discontinuity, local optima, and a global optimum. Most hand-designed losses favor one clean optimum, making the discovered structure precisely the kind of result an automated search can surface against prevailing intuition.
Their hypothesis is that the local optimum may capture noisy or bad preference data while the rest is optimized toward the global solution. Chris preserves the uncertainty: the paper offers some evidence, but “it’s really hard to, like, rigorously prove this.”
A similar non-convex feature appeared in their earlier Discovered Policy Optimization work, where evolution also produced behavior the authors initially did not understand. Its recurrence is evidence worth investigating, not proof that non-convexity is universally responsible for performance.
The final objective is only about 5–8 lines of readable code. One unusual exponential loss does not appear especially effective alone, Chris says, but becomes useful when combined with other terms—an example of search finding complementarities that component-wise evaluation could miss.

4. Machine throughput changes the human role from inventor to supervisor

Lange rejects the premise that automation must match the best researcher’s creativity before it matters. Even if an automated discovery system were “50% worse than what a human would do,” its scalable throughput could let researchers turn “money and compute into, like, really useful insights for the next generation of AI.”
Chris currently compares the systems to an undergraduate, first-year PhD student, or very young researcher. A human supervisor can still identify important problems, rule out poor strategies, set priorities, and decide which outputs deserve deeper work—much as a professor directs a lab.
That selection function is not incidental. Chris argues that after a point, producing papers is “more like an art than a science”: novelty, usefulness, and community interest resist a single objective, just as image and video generation ultimately require someone with taste to choose among abundant outputs.

5. Recursive discovery needs renewable sources of entropy

The host asks whether repeatedly using an LLM to discover a better loss, retraining the next model with it, and repeating the cycle would improve indefinitely or collapse. Lange distinguishes an objective’s indirect effect on training from directly teaching a model to reproduce its own prior outputs, so feedback severity depends on what enters the loop.
Lange concedes uncertainty about internet-scale synthetic-data collapse, but notes that today’s systems already generate substantial value. Even if deterioration arrived within five years, he argues, society would reach that point with far more capable tools than it has now.
Chris treats entropy as essential to new ideas but sees external supplies: humans continue to ask unexpected questions, or a system can randomly sample two fields and demand a connection. That deliberately introduces “external entropy” rather than repeatedly sampling only from one self-generated distribution.
The host cites research where roughly four synthetic generations drain distributional entropy; Chris’s pushback is that such experiments may omit live human inputs. Lange extends the answer to evolutionary crossover, quality-diversity algorithms, and cultural recombination: complexity may reside as much in how models are used as inside their weights.

6. LLMs can optimize abstract numerical problems without gradients

Lange’s Large Language Models as Evolution Strategies project grew from work showing LLMs improving a CartPole policy represented only as integer sequences. If a model could detect progress in such abstract strings, he reasoned, it might function as a general black-box optimizer.
The prompt includes past candidate parameters and their fitness, sorted from worse to better using a least-to-most strategy. From that improving sequence, the model infers productive moves and proposes the next evaluation, balancing exploration and exploitation without access to gradients.
The method can outperform traditional black-box algorithms in some tested settings and presently scales to medium-sized problems of roughly 50 dimensions. Lange is explicit that evolutionary search is not suitable for optimizing transformer-scale parameter spaces; its niche is where gradients are unavailable.
For Lange, this weakens the “stochastic parrot” objection: detecting improvement across abstract numerical sequences requires pattern inference beyond replaying familiar prose. His qualified claim is that whenever a problem can be represented as a structured string, an LLM may extract useful in-context patterns—though manual algorithms will remain better for some domains.

7. Representation choices can outweigh parameter count

Comparing GPT-4, Llama 2, and PaLM 2, the team found smaller models surprisingly competitive on many black-box tasks, with GPT-4 the notable larger-model exception. Lange speculates about overtraining, undertraining, and possibly smaller experts inside GPT-4, but labels these explanations as speculation.
Fine-tuning on trajectories from a teacher optimizer improved certain tasks but not all. The result captures a recurring tension: explicitly baking an algorithmic bias into training can strengthen targeted behavior while removing some of the creativity the general model would otherwise discover.
Tokenization created concrete numerical artifacts. In the cited Llama 2 tokenizer, numbers 1–50 and many years from 1950–2020 receive individual tokens, while other floating-point values fragment unevenly; standardized integer discretization therefore made patterns easier for the model to infer.
Newer frontier models were already becoming capable of consuming floating-point numbers directly, so Lange expects some conclusions to date quickly. Smaller specialized models could reduce compute, but removing apparently irrelevant pretraining knowledge might also remove material later reused for an unexpected “perturbation” or analogy—hence Sakana AI’s belief that “learning sort of always wins.”

8. Evolution can produce hyperscale methods without running at hyperscale

Lange reframes mainstream model training itself as collective evolution: generations of PhD students discovered default hyperparameters, optimizers, and engineering practices that seed every new run. If Adam had never been published, he argues, even a hyperscaler starting from scratch would occupy a different position.
LLM-guided search need not maintain nature-sized populations. DiscoPOP can evaluate one intelligent candidate, return its fitness to context, and repeat; earlier work also evolved algorithms on small, low-dimensional tasks and transferred them to harder, higher-dimensional settings.
Promising targets include data mixtures and synthetic curricula that induce chain-of-thought reasoning. One could train small models on tasks generated by an outer algorithm, optimize that generator for downstream in-context reasoning, then test whether the discovered recipe transfers to larger systems.
Lange says Sakana AI deliberately avoids the language-model pretraining race, seeking ideas others will not fund heavily. His diagnosis is a “meta-scientific local optimum”: scaling is brutally hard engineering, yet conceptually linear—raise capital, train a state-of-the-art model, and watch another replace it weeks later.

9. Collective intelligence may unlock abilities individual scaling cannot

Natural evolution remains the only process the speakers know has produced general intelligence, but Lange cautions against copying biology wholesale. Machines have different affordances: Sakana’s Sensory Neuron work, for example, examined policies over pixel-mutated representations that humans would struggle to adapt to but transformers could learn.
Drawing on Alison Gopnik’s discussion of child development, Lange argues that intelligence depends partly on social structure. Parents protect children from overwhelming parts of reality while exposing them to manageable causal experiments; this safety and information asymmetry permit focused learning that an isolated agent could not reproduce.
The implication is lifelong adaptation through collective systems, not merely a larger solitary model. Lange imagines personalized assistants and synthetic forms of evolution, while acknowledging that machines may settle into cognitive structures unlike the local optimum imposed on humans by biological evolution.
The host’s enthusiasm comes with an engineering objection: a “living, breathing system” performing several nested levels of meta-optimization sounds like a software nightmare. Lange offers no guarantee that this threshold will be crossed, only that rising compute and new architectures make the direction plausible enough to test.

10. Agent design turns whole software workflows into searchable artifacts

Cong’s Automated Design of Agentic Systems starts from the observation that the AI Scientist was painfully handcrafted over months, while Cursor-like tools and literature assistants are also LLM agent systems; crucially, these systems are pieces of Python code. With Claude 3.5 Sonnet and GPT-4o, such systems can now discover code that is hundreds of lines long.
The search space includes prompts, chain-of-thought structures, debate, retrieval, tools, code execution, and even programs that load other neural models. Because generated code is Turing-complete, the search can include workflows beyond pure LLM calls, including code that writes or trains another model.
Cong resists choosing categorically between code generation and direct transduction. Arithmetic, web search, reusable multi-step operations, and compositional tools remain naturally suited to code; neural components can still handle the perceptual or intuitive steps within the same evolved system.
Discovered workflows often reuse archived modules: one agent makes an initial prediction, another refines it, and both may descend from earlier candidates. The archive preserves serendipitous stepping stones rather than forcing every discovery to improve one monolithic lineage.

11. Evolved organizations can look stranger—and flatter—than companies

Exploration comes from extensive sampling, mutation, prompts to use adjacent fields, and effectively unbounded archives such as OMNI. Cong reports runs lasting more than 5,000 generations in which an LLM judge still found later artifacts noticeably different, suggesting contextual entropy can sustain prolonged search.
The resulting systems transfer better than benchmark-overfitting fears would predict. A strong outer coding agent optimized a cheap GPT-3 inner agent on math, yet the discovered reasoning loop transferred to GPT-4, other mathematical domains, and “shockingly” to literature tasks.
Debate is a recurring pattern, but evolved variants become elaborate: separate experts may criticize efficiency, readability, and accuracy before additional rounds reconcile them. Some workflows contain roughly 20 chained calls, prompting Cong to ask, “How on earth would I even…begin to come up with these structures?”
Runtime limits and cost objectives can constrain infinite loops, while evolution determines how to organize perhaps 1,000 actors. Human hierarchies reflect working-memory and communication bottlenecks; a Gemini-scale context with millions of tokens could hypothetically integrate 1,000 reports, collapsing layers that human companies require.

12. Open-ended search must eventually abandon a fixed benchmark

The host invokes Kenneth Stanley’s warning that high-bandwidth coordination could create monoliths, while genuine novelty may require agents to follow private gradients of interest. Lange says the present system remains benchmark-guided despite prompts to “think out of the box.”
Cong’s preferred next step co-evolves agents with task proposers that invent progressively harder challenges, switching goals rather than optimizing one score forever. Such a loop could accumulate a “tech tree or a skill tree” and pursue the open-ended cultural trajectory associated with full AGI-like capability.
A meta-agent can also improve the search procedure that improves the task solver, echoing STOP and Gödel-machine-style recursion. Lange tempers the recursive-self-improvement story: eventually the system may be “squeezing blood out of a stone,” bounded by the underlying model, though gradient updates could further optimize what the search finds.
Cong cannot predict the endpoint; ten years into the future, accumulated innovations may look like “magic and sorcery.” The defining property is not mastery of a frozen benchmark but continuous generation of new objectives, artifacts, and stepping stones that make later capabilities possible.

13. A model’s “nose for interestingness” can replace brittle exploration rules

Intelligent Go-Explore builds on the 2017 Go-Explore algorithm: archive promising states, revisit them, take exploratory actions, and retain useful discoveries before robustifying successful trajectories. Its weakness is the need for a handcrafted function deciding which states are interesting.
Montezuma’s Revenge historically required intuitions such as descending levels, collecting keys, and preserving agency; Cong clarifies that his team had not yet tested that game. Their contribution replaces comparable heuristics in other hard-exploration environments with a foundation model’s “nose for interestingness.”
That prior works when environments reflect familiar human concepts, but a scientific field invented ten years from now may require new abstractions and retraining. Cong says generic structures such as debate may still transfer once the right abstractions for the new problem are found.
The method now accepts images in visual grid worlds as well as text, with a higher-level loop caching interesting states atop a VLM agent. Cong also names the danger plainly: if a model were biased toward green objects, the search could amplify green trajectories, requiring reward filters, corrective data, human supervision, and continuous monitoring.

14. The AI Scientist shifts the bottleneck from writing to epistemic judgment

Robert summarizes the AI Scientist in one sentence: “We try to use LLMs to write new papers that are hopefully helpful to the community eventually.” DiscoPOP suggested the route—the discovered loss and experiments already resembled a contribution, leaving ideation, implementation, execution, interpretation, and write-up to be chained together.
Chris says Cong describes this as a “GPT-1 moment” for scientific generative AI and calls the current period a “Cambrian explosion,” not a finished scientist. Separate advances in ideation, coding, review, and frontier models can be recombined, while Lange says the ultimate impact remains to be seen despite moments when the authors were “mesmerized” by generated results.
The host’s Google Maps analogy captures why the papers can look convincing: a zero-shot whole paper is banal, whereas repeated zooming allocates computation to experiments, tables, and local revisions. The system automates that refinement, though current papers still contain flaws beneath their polished surface.
Its deepest limitation is linear experimentation. An idea becomes an ablation and then a paper even when results disappoint; Lange’s own workflow spends roughly 80% of its time rejecting hypotheses and only 20% consolidating results. Iterative causal modeling—not smoother prose—is the next essential step.

15. Scientific logs and AI conferences could matter more than papers

Addressing the objection that researchers learn by building, failing, and arguing—not merely reading papers—Cong agrees completely. AI scientists could generate the missing experience: broad explorations, failed experiments, and intuitions about what might work, exposing knowledge far below the published “tip of the iceberg.”
Cong suggests that these logs could eventually feed systems optimized against grounded coding, math, diffusion, or NLP benchmarks, much as systems such as R1 learn where correctness is measurable. The proposal is speculative: the goal would be to incorporate failed experimentation and hypothesis-testing knowledge into future AI scientists.
Papers nevertheless remain a human-readable interface. Alongside them, the AI Scientist produces code and reproducible results; yet Lange argues that once “money compute API calls equals paper,” conference acceptance stops being an adequate definition of contribution. A two-line idea such as a residual connection matters because it diffuses through later work.
Lange imagines AI-only conferences with generated submissions, reviewers validated against ICLR 2022 decisions, debate, area-chair judgments, and best-paper awards, followed by human validation. Alien machine subcultures remain possible, but the telescope analogy offers a compromise: humans need not match an instrument’s resolution if it exposes the result through an intelligible abstraction.

Grok 3, AI Memory & Voice, China, DOGE, Public Market Pull Back | BG2 w/ Bill Gurley & Brad Gerstner US-China Relations: History, Culture, and AI Competition, with Noah Smith, from Econ 102