An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio

Summary

• Leash Bio makes scientific-agent decisions auditable by premeasuring the full landscape. Agents get a target, a catalog, and a budget of 100 molecules to find the best binder; metabolism and other drug-like properties are outside scope. Across about 90,000 rollouts, Leash can compare results with the measured landscape.
• Better models’ clearest behavioral edge is leaving unproductive chemistry sooner. Agents may repeat tests, choose molecules at odds with their hypotheses, or keep modifying poor binders; stronger models abandon “crappy neighborhoods” faster. Why they do so remains unknown: “We don’t know that at all yet.”
• Astra 6 leads, while expensive models can still underperform. Quigley also rates Sonnet 5.5 and Sol 6.1 well, and says DeepSeek V4 Flash was “pretty good.” Sonnet 5 is “both very expensive and not very performant”; Opus 5.5 had not yet been tested.
• Improved chemistry fluency changed what Leash could attempt with general models. After poor SMILES handling last summer, models began working meaningfully with chemistry around February. Removing molecular information hurts their decisions. Quigley says they may understand some of it, but qualifies that they are at least using the information—not changing weights during these tasks.
• Leash’s scaling plan reuses measured experiments across vast numbers of agent rollouts. Quigley calls it “a lab in the loop with amortized latency,” with choices restricted to measured molecules. Leash has screened about 1,000 proteins against a couple of chemical libraries and is doing two environments a day; it plans to try for maybe 20 a day, with that goal for the beginning of 2026 or 2027.
• Transfer to new biology remains unproven, and a proposed baseline is untested. Quigley does not know whether reinforcement-learning progress will transfer. Asked about a Bayesian hyperparameter algorithm as a deterministic baseline, he says they have not run one and think it probably will not work because the landscape is “too spiky” and “very high-dimensional.”

Deep dive

1. A measured chemical world makes scientific decisions auditable

• Quigley’s structure–activity relationship task gives agents a target, a catalog, and a budget of 100 molecules: “go find the best binder you can.” It measures binding improvement; metabolism and other drug-like properties are outside scope.
• Agents use different search policies, state readable-language SAR hypotheses about why molecules do or don’t work, and test those hypotheses. Across about 90,000 rollouts, Leash knows the entire measured landscape. An agent might find something only “half as good as the best thing in the entire environment,” making missed opportunities visible alongside final scores.

2. Better models abandon unproductive chemistry sooner

• Quigley sees agents that give up early, retest molecules, or spend budget on choices inconsistent with their hypotheses. Better models leave poor neighborhoods faster; weaker ones keep making “little subtle changes on molecules that aren’t ever gonna go anywhere.”
• A host asks whether this means recognizing bad neighborhoods, beyond simply jumping more. Quigley agrees that they abandon them faster, but says the reason is unknown: “We don’t know that at all yet.”
• Medicinal chemists have judged sampled rationales plausible. A host’s proposed human game faces a catalog constraint: “you can’t do whatever you want, you can just do a lot of stuff.”

3. Astra 6 leads an already shifting model field

• Quigley names Astra 6 the champion, with Sonnet 5.5 and Sol 6.1 also good. Sonnet 5 is “both very expensive and not very performant”; Opus 5.5 has not been tested. DeepSeek V4 Flash was “pretty good.”
• Frontier releases roughly every 11 days keep disrupting publication plans. Quigley expects the preprint around the episode’s release, with a blog, data, environment packages, and many thousands of transcripts.

4. Chemistry fluency improves, but a proposed optimization baseline remains untested

• Last summer, Leash began supervised fine-tuning because frontier models struggled to convert SMILES into noncanonical strings representing the same molecule. A SMILES string traverses a molecule’s graph; starting at a different point can produce a different string for that molecule.
• Around February, models began working meaningfully with chemistry. Removing molecular information now hurts decisions. Quigley allows that models may understand some of it, but clarifies that the evidence shows they are using the information: “Or at least they’re using it”—“We’re not changing weights here.”
• A host asks about using a Bayesian hyperparameter algorithm as a deterministic baseline. Quigley says they have not run one and think it probably will not work because the landscape is “too spiky” and “very high-dimensional”; he calls that the party line.

5. Reusable experiments scale feedback before proving biological transfer

• Quigley calls the design “a lab in the loop with amortized latency”: measure once, then run effectively unlimited rollouts. Live experiments allow more freedom; premeasured worlds restrict choices to what Leash already measured.
• Leash has screened about 1,000 proteins against a couple of chemical libraries. It is doing two environments daily and plans to try for maybe 20 a day, with that goal for the beginning of 2026 or 2027.
• Whether reinforcement learning in these worlds transfers to new biology remains unknown. A host suggests a covering set might help; Quigley also sees broader questions in experimental budget allocation: “when do I stay? When do I go?”