How Claude Plays Pokémon was made
Summary
- Claude 3.7 Sonnet’s Pokémon run is an expensive integration test for long-horizon agency, not an optimized game-playing system. David Hershey deliberately kept the harness simple so progress would reveal the model’s ability to act, remember, and recover: “You could squint and see a thing playing the game.” Pokémon supplies objective progress milestones—especially gym badges—while exposing long-horizon failures.
- The largest performance gains came from model upgrades, not orchestration tweaks. June’s Sonnet 3.5 showed “signs of life,” October’s version could leave the house and choose a starter, and 3.7 began catching Pokémon and progressing through locations; the best reported run reached at least Surge’s Gym. After many hours tuning prompts and tools, Hershey’s verdict was blunt: “Nothing quite does it like a new model.”
- Vision and spatial reasoning remain the binding constraints on otherwise capable agents. Claude can perform “PhD-level stuff” yet struggle to identify its character, understand walls, or remember whether moving up re-enters Professor Oak’s lab. Hershey believes several obstacles remain effectively impassable without material improvements in seeing, navigating, and visually remembering state.
- The experiment’s token economics are substantial even with aggressive context management. A run combines a roughly 1,000-token system prompt, an 8,000-token knowledge-base cap, and 30 recent messages, with full requests fluctuating from about 5,000 to 100,000 tokens because screenshots dominate. Testing many configurations has consumed “at least thousands of dollars of tokens,” limiting brute-force experimentation outside a funded setting.
- Persistent memory looks like the harness’s highest-leverage improvement, but more context is not automatically better. Thirty retained messages performed better than 20 or 40 before the history was summarized; showing more prior screenshots did not obviously solve the model’s navigation problem. Claude’s knowledge base can record both game facts and metacognitive lessons such as “I misperceived this thing,” potentially carrying emulator-use skills into later runs or other games.
- Pretrained knowledge is double-edged because confident recall can become a navigation trap. Claude sometimes recognizes type interactions and learns from failure, but it also spent roughly 12 hours pursuing a supposed Mount Moon exit on the eastern wall. Hershey remains unsure whether what the model “knows” about Pokémon helps or harms more—and the discussion emphasized not simply supplying a walkthrough because discovery is the point.
- Claude is still far from beating Twitch Plays Pokémon’s 16-day, seven-hour benchmark, but its failure modes are commercially informative. The live run had spent more than 50 hours in Mount Moon, while the best separate run reached at least Surge’s Gym. Hershey’s broader call is that 3.7’s improved ability to “course correct, update, and figure things out” should yield useful real-world agents even before Pokémon is solved.
Deep dive
1. Pokémon became a longitudinal test of agent capability
Hershey began the project around Sonnet 3.5’s June release because he wanted a real harness for experimenting with long-running agents—and a task compelling enough that he would “grind the hardest” to improve it. Pokémon was his first childhood game, while Twitch Plays Pokémon supplied the cultural template and nostalgia.
The original Sonnet 3.5 was “very good,” but only displayed “signs of life” in-game. October’s new version of Sonnet 3.5 could somewhat reliably leave the player’s house, obtain a starter, and occasionally name it; 3.7 crossed the qualitative threshold where Claude caught Pokémon, progressed through locations, and recognizably played.
An internal Claude Plays Pokémon Slack channel gradually developed a “cult following” around Hershey’s GIFs and updates. With 3.7, entertainment turned into measurement: after eight months and “millions of words” of model output, he could compare releases against a familiar task and see which capabilities had genuinely changed.
Hershey’s boundary condition matters: this is not “the world’s most incredible agent harness,” and building a better program to beat Pokémon with Claude in the loop would be trivial. He did not supply a walkthrough because the useful question is whether the model can discover, remember, and recover—not whether an engineered solver can finish.
2. A minimal harness exposes severe visual and spatial weaknesses
The core system is a rolling tool-use conversation with essentially no user interjections after “go.” A short system prompt explains three tools, supplies roughly six Pokémon facts, and adds a few corrections for catastrophic recurring mistakes; Claude otherwise presses buttons, receives results, and maintains its knowledge base.
The emulator tool executes button sequences and returns two screenshots—one plain and one overlaid with coordinates—plus a small amount of state read directly from the game. Hershey reverse-engineered enough Pokémon Red RAM to expose nearly every state variable programmatically, with Claude Code handling tedious work such as mapping memory addresses into Python.
Location is the crucial state hint because Claude “pretty aggressively” hallucinates successful transitions between zones. Even explicit prompts identifying its coordinates, red hat, and position on screen do not solve the deeper problem: it lacks reliable spatial awareness and can confuse its character with other sprites.
Navigator is the main compensating tool and the only other vision patch Hershey described: Claude selects a visible coordinate, and the harness automatically presses buttons to get there. Without it, Claude will repeatedly walk into walls; in another visual failure, it mistook a red box that was a door for a text box and spent approximately 12 hours pressing A to dismiss it.
3. Context management controls both performance and cost
The prompt starts with about 1,000 system-prompt tokens and permits up to 8,000 tokens in the persistent knowledge base. Hershey imposed that cap because, left unconstrained, Claude writes “a whole bunch of BS”; the cap pushes it to decide which facts and lessons are worth retaining.
Claude then receives a rolling history of 30 messages, largely alternating tool calls and results. At the limit, the harness summarizes that history, inserts the summary as the first user message, and begins another rollout—preserving long-run continuity without carrying every prior interaction verbatim.
Screenshots dominate consumption, so Hershey retains some previous images but trims others. Individual API requests fluctuate from roughly 5,000 to 100,000 tokens; 30 messages empirically beat both 20 and 40, illustrating an “effective context length” where adding information is not automatically beneficial.
Extensive experimentation is correspondingly expensive: running many configurations for long enough to compare progress requires “at least thousands of dollars of tokens.” Hershey cautioned that this is a side project to undertake only after considering “the impact on your personal wallet and how much you care about Pokémon.”
4. Smarter models need fewer instructions—and still reason strangely
Claude’s latent Pokémon knowledge may help or hurt. After Thundershock failed against Geodude, it appeared to update—“I forgot about that; that does not work”—yet it also misidentified an NPC as Professor Oak and spent 12 hours acting on a confident but incorrect claim that Mount Moon’s exit was on the eastern wall.
Sonnet 3.7’s hybrid reasoning caused no serious regression: it can behave like the earlier model while spending additional time thinking. More strikingly, every Pokémon model generation let Hershey delete old “Band-Aid prompt stuff”; he has become “less confident over time that I understand exactly how the model is intelligent,” so freer rein now outperforms prescriptive human intuitions.
His sharpest case against prompt-only navigation fixes came outside Oak’s lab. Claude exited, announced it needed to travel north, pressed up and re-entered; it then pressed down to leave and repeated the same loop roughly 12 times. “You’re not fixing that with a prompt—it just literally doesn’t get it.”
Small instructions can nevertheless change behavior unexpectedly. Once encouraged to nickname Pokémon, Claude became visibly more protective and would promptly heal injured named companions, something it had not previously done. The knowledge base also began recording self-assessments—“I misperceived this thing”—suggesting transferable lessons about button batching, observation, and emulator use.
5. Milestone-based evaluation shows progress without hiding the gap
Hershey’s most useful quantitative evaluation is costly but direct: run a configuration about ten times and measure how quickly it reaches game milestones. Gym badges supply literal benchmarks, making this “an integration test, not a unit test”; smaller scenario tests help examine particular situations but do not capture the full long-horizon behavior.
The strongest pre-launch run shown in Anthropic’s research graph progressed through at least Surge’s Gym, with reporting ending because that was the physical amount of time between starting the run and launching the model. The live run was worse than average and remained in Mount Moon after more than 50 hours; Hershey could name four points in the game where he saw “literally no hope” for the current model.
Twitch Plays Pokémon completed Pokémon Red in 16 days and seven hours, despite conflicting crowd commands. Hershey did not expect the current Claude stream to stand in Victory Road by day 13, but he expects continued model training to improve long-horizon scaling: the broader capability is Claude’s growing tendency to “course correct, update, and figure things out,” then “power through” problems in a new way.