New in Nature: Google Agents Beat Human Doctors, Make Scientific Discoveries – With Vivek and Anil
Summary
AMIE outperformed primary-care physicians in simulated consultations testing diagnosis, analysis, and longer-horizon treatment planning. It grounded recommendations in clinical guidelines and medication labels, but Vivek Natarajan stressed that actors following scripts are not real patients: “all kinds of things can happen” outside simulation. A supervised study with real patients at Beth Israel Deaconess is the critical next gate.
In cardiology and oncology, AMIE surpassed fellows and approached—but did not consistently match—the most experienced attending physicians. The more investable near-term result was augmentation: cardiologists using AMIE were judged superior to unassisted cardiologists across almost every dimension, a “landslide” driven by complementary error patterns. That matters when Anil Palepu said a neurology consultation can involve a stated “12 to 18 month wait time.”
Google’s co-scientist recovered the same drug-resistance mechanism that Imperial College collaborators had discovered experimentally but not yet published. It progressed from drug repurposing to therapeutic-target identification and then an open-ended bacterial-resistance problem, running autonomously for days before ranking the correct mechanism first. The startled collaborator asked whether Google was “reading my email”—strong evidence, in Nathan Labenz’s framing, of synthesis rather than simple retrieval.
The architecture shifts differentiation away from bespoke model training and toward inference-time agent design. Current AMIE and co-scientist systems rely largely on commercially available Gemini models, specialized prompts, tools, structured reasoning, and orchestration rather than custom fine-tuning. For investors, the grounded takeaway is that workflow design, validation, and orchestration become increasingly important sources of practical differentiation.
Long context, fresh external information, and test-time compute let these systems improve for days without collapsing into repetitive answers. Co-scientist can place ideas, reviews, debates, and search results into contexts approaching 2 million tokens; AMIE performed best with roughly 256k of guidelines. Most co-scientist queries were estimated to cost only “a few dollars,” generally below $10 at then-current GCP prices, with further declines expected over 6–18 months.
Two techniques are immediately portable: generate multiple plans before synthesis, and rank candidates through pairwise tournaments. AMIE improved sharply when it produced four concurrent management plans and combined their strongest elements; structured outputs forced the sequence from analysis to goals to treatment. Co-scientist’s AlphaStar-inspired tournament is computationally inefficient, but diverse candidates can be compared and stronger ideas surfaced.
The next frontier is asking worthwhile questions and connecting agents to experimental data, simulations, and specialist models. Natarajan called question selection the upstream “root node problem” whose solution would justify saying there are “geniuses in data centers”; meanwhile, co-scientist is already working with close to 100 trusted scientists, with an optimistic goal of millions by year-end. The capability upside is paired with explicit deployment risk: the team is staging access because a highly capable scientific system has “many ways in which it can possibly not do so well.”
Deep dive
1. AMIE moves beyond medical question answering into the clinical encounter
Natarajan’s starting distinction: medical question-answering shows that language models encode clinical knowledge, but “that’s not clinical practice.” Doctors must interact with patients, discover relevant information rather than receive it upfront, and then reach a defensible diagnostic endpoint.
The original AMIE study therefore used objective structured clinical examinations: patient actors conducted text conversations with either physicians or the AI. AMIE’s diagnostic accuracy exceeded that of human primary-care physicians, establishing the baseline that the new management work extends.
Nathan Labenz’s framing was deliberately stark: an AI can now perform work that would require years of specialized human training, yet the result barely enters public consciousness. His own update was that “the AIs might in fact now be clearly smarter than me,” albeit within carefully scaffolded tasks.
2. Longer-horizon management is the new benchmark—and still a simulation
The new challenge is not merely naming a disease after one encounter. Clinical care unfolds across visits, and the correct first endpoint may be “order the right test” or place the patient on the right path—not pretend that diagnosis and treatment are already certain.
AMIE was asked to reason through this longer horizon, produce precise management recommendations, and ground them in accepted clinical-practice guidelines and medication labels. Within that setup, it outperformed general practitioners on analysis and treatment planning as well as diagnosis.
The team deliberately kept the study text-only to isolate the reasoning component; adding images would introduce more confounders. Multimodal pieces already existed in earlier work, and the guests said a multimodal version should follow “very soon.”
Natarajan’s essential caveat: these remain “simulated consultations” with actors, not uncontrolled encounters with real patients. The team is optimistic that performance will translate “at least to a certain degree,” but it does not present simulation as proof of clinical safety.
3. Specialist AI looks strongest when it augments human doctors
In cardiology and oncology, AMIE exceeded medical-fellow performance and closed in on attending physicians, but the guests conceded that it was not yet as consistently strong across every domain as “the most experienced attendings.”
Head-to-head comparisons with general cardiologists were mixed: AMIE did better on some dimensions and doctors on others. The error patterns differed enough, however, that a cardiologist given AMIE’s assessment became superior to an unassisted cardiologist on almost every measured aspect—a “landslide.”
Palepu tied the result to access: a neurology consultation can reportedly involve a 12–18-month wait, while specialist care is costly and organized into silos partly because “there’s only so much expertise that we can cram into our given brain.” AI need not inherit that cognitive constraint.
The result was already aging quickly. The cardiology work used Flash 1.5 and was described as three to five months old; Gemini 2 and 2.5 had arrived since, leaving open whether the attending-level gap would persist under a straightforward model upgrade.
4. Co-scientist independently recovered unpublished biology
Nathan organized the evaluation as three ascending challenges: repurpose an existing drug; identify a therapeutic target inside a diseased cell, where causal biology is only partly mapped; and explain how bacteria developed drug resistance from a sparse observational clue.
For the hardest problem, Imperial College collaborators José and Thiago supplied a question based on experimental results they had not published. Co-scientist ran for roughly two days and returned the same mechanism as its leading hypothesis.
The collaborator responded within about 10 minutes: “I need to talk to you right now,” followed by, “Vivek, are you reading my email?” Natarajan answered, “We do many things at Google, but reading your email is not one of them.”
The system also produced four additional hypotheses that the scientist said “make a lot of sense”; postdoctoral researchers began validating them. The distinction matters: co-scientist proposed computational hypotheses, while human scientists supplied the experimental work and validation.
5. Thousands of bad hypotheses taught the team to engineer reliability
Natarajan traced co-scientist to 2023, when Stanford professor Gary Peltz approached the team after its Med-PaLM work. His proposition was direct: models trained on scientific literature might uncover causal facts that could help “millions of people with rare diseases.”
On a mouse model of hearing loss, an early model proposed a digenic mechanism Peltz had not considered. According to Natarajan, subsequent CRISPR knock-in experiments allowed the lab to reverse the course of the disease; later Med-Gemini work extended the approach retrospectively to human variants of unknown significance.
The success was radically unreliable: to generate one valuable hypothesis, Med-PaLM produced “thousands of things that were utter garbage.” A less expert or less patient collaborator might have rejected the project after the first five answers.
That experience changed the objective from occasional brilliance to calibration: every generation should be useful, and the best ideas should rank near the top. Repeatedly calling one model failed because the process drifted into mode collapse and degenerate solutions.
6. Prompted agents are replacing bespoke medical model training
Natarajan described a major change from earlier Med-PaLM and Med-Gemini work. Those projects specialized general models through medical fine-tuning; current AMIE and co-scientist largely skip that step and instead assemble prompted agents around general-purpose Gemini models.
Some relevant medical data has since moved upstream into Gemini, so this is not proof that domain data ceased to matter. But stronger instruction-following now lets one base model adopt precise specialist roles without maintaining a separate trained model for every task.
AMIE’s management agent improved markedly when it stopped drafting one plan and instead generated four concurrent plans, then synthesized them. The model could preserve overlapping strengths, import a good detail from one candidate, and “leave out the bad.”
Co-scientist required roughly 18 months of iteration: establish evaluations, identify weaknesses, add or revise specialized agents, and finally simplify the architecture. The team called the result a functional prototype with “so much room for improvement,” not a hyperparameter-optimized endpoint.
7. Long context became working memory for days-long reasoning
Co-scientist has no explicit memory store. Gemini instances can accept up to 2 million tokens, so generated ideas, reviews, debates, and other “walls of text” can be returned as feedback in later rounds without a separately engineered retrieval architecture.
AMIE similarly loads many clinical guidelines into context. Internal evaluations suggested performance improved when the team reduced a planned 1-million-token input to roughly 256k—still enough breadth to include relevant guidance even when retrieval was imperfect.
Natarajan conceded that co-scientist’s validation of long-context use was “a little bit more unscientific.” The safeguard was redundancy: many new generations receive the accumulated feedback, increasing the chance that at least one correctly incorporates its decisive elements.
The team also resisted spending another year hill-climbing on “nice cozy benchmarks.” Its preferred end-to-end test was to take a reasonable system directly to domain experts, show them the ideas, and seek experimental validation that could become a real discovery.
8. Fresh entropy lets self-improvement continue without collapse
Nathan noted that earlier GPT-4 systems often peaked after three to six self-critique rounds and then deteriorated. Co-scientist can run for days because each cycle can import new information through web search, tools, specialist knowledge bases, or human suggestions.
Natarajan’s mechanism is increased entropy: new evidence prevents a closed loop from repeatedly polishing the same idea into a degenerate answer. Future feedback could come from AlphaFold, simulations, experimental databases, or other agents—not merely text retrieved after a model’s training cutoff.
More compute is not universally useful. Trivial problems resolve quickly, while questions missing indispensable information may remain impossible regardless of inference budget; the opportunity lies in the “sweet spot” of difficult but bounded problems, which Natarajan thinks includes much of humanity’s current scientific agenda.
Vivek offered a rough, non-exact estimate of under 10 million total inference tokens for the whole resistance run, then separately estimated that most queries would cost “a few dollars,” below $10 at then-current GCP prices.
9. Scientific tools and hidden datasets are the next capability frontier
Published papers are only the visible layer of science. Negative results are rarely published because of academic incentives, creating “some dark matter that’s hidden away” and leaving potentially decisive evidence outside the system’s reachable corpus.
Supplementary files contain large experimental datasets with useful information. Natarajan envisioned co-scientist generating hypotheses while a data-science agent analyzes the underlying files and feeds empirical results back into the next reasoning cycle.
His flagship example was the Arc Institute’s Virtual Cell Atlas, described as spanning roughly 300 million gene perturbations for 300 million cells. Human teams could spend years exploring that space; persistent agents could propose questions, interrogate the data, and iterate across target discovery and basic biology.
Nathan asked whether biology, materials, or other specialist representations would eventually merge directly into a general reasoning model. Natarajan was unsure; he said the architectures and compute likely exist, but adding medical images or genomic data through continued pre-training or SFT has produced regressions on mainstream language and vision benchmarks.
10. Structured reasoning and tournaments are today’s practical playbook
Co-scientist’s ranking system drew inspiration from AlphaStar: hypotheses compete pairwise, and repeated comparisons surface stronger candidates. Natarajan stopped short of calling it an ideal standard because naïve comparison scales as n²; the team clusters and groups ideas to avoid thousand-by-thousand tournaments.
His expected destination is latent search: models should eventually generate, review, and rank alternatives “in your head” without explicitly emitting every branch and comparison token. Today’s tournament buys performance and interpretability at the price of obvious computational inefficiency.
AMIE’s management-reasoning agent uses structured outputs to force an ordered path: perform detailed analysis, identify higher-level management goals, then construct the final plan. Nathan compared it with an airline checklist that prevents a model from wandering around and declaring, “Yeah, it all looks good to me.”
More structure was not always better. Highly granular templates reduced flexibility, while broad fields for analysis and management goals performed better in internal automated evaluations and “vibe checks.”
11. Question selection and real-world deployment are now the gating tests
Nathan proposed prepending an agent that searches the internet, identifies worthwhile unanswered questions, and gives them to co-scientist. Natarajan called question selection an even more upstream “root node problem”: solving it would mark the day one could credibly claim “geniuses in data centers.”
The underlying papers used Gemini 2.0, not Gemini 2.5. Natarajan believed all the necessary building blocks were visible and that, for “the next couple of years at least,” progress looked mainly like an engineering challenge; the first workable solution need not be elegant.
Co-scientist was already being used with close to 100 leading scientists through a trusted-tester program. Google planned staged waves so each cohort could expose weaknesses before broader release; Natarajan’s optimistic goal was access for millions of scientists by year-end.
AMIE’s path is slower and more regulated. A forthcoming Beth Israel Deaconess study will place the LLM system with real patients in a clinic where clinical experts can oversee it and take over if needed; if few things go wrong, the team hopes to expand to more centers, reduce required supervision, and add patient-facing capabilities.