The Most Valuable Work of an AI Product Manager Is Writing Test Cases for Models

Deep thoughts on AI and aspirations —— ByteDance Deep Thinking Circle

In 2023, one complaint appeared frequently in the feedback Anthropic received from users: Claude doesn’t follow instructions well.

This statement alone is not actionable. Hand it to researchers, and they don’t know where to start fixing it; hand it to engineers, and they can’t pinpoint the problem either. The product person at Anthropic responsible for this area broke down this vague complaint by examining what users specifically asked and what Claude specifically responded with, reading through original conversations line by line. After tallying everything up, about 80% of “not following instructions” actually pointed to the same thing: the model couldn’t reliably output correctly formatted JSON.

What came next was the critical move. The team collected dozens of real examples where Claude produced incorrect output, organized them into a set of input-output pairs with standard answers, and ran them before each new version release, iterating until nearly all passed. A vague complaint was transformed into a repeatable test suite.

This is called an eval, or evaluation set. There’s a saying circulating inside Anthropic: evals are the new PRDs—evaluation sets are the requirements documents of the new era. My judgment is more direct: for people building AI products, the most valuable work is shifting from writing documents to writing test cases.

Vague Feedback Equals No Feedback

Traditional product managers prove their product strategy by writing a polished requirements document or a prototype backed by user interviews. This approach assumes predictable product behavior: once features are clearly specified, the implementation matches the specification.

Products built on models lack this assumption. You revise a version of the prompt and it feels somewhat better; you switch model versions and it seems worse again. Intuition fails in the noise. The same user complaint can’t be reproduced today, then resurfaces tomorrow. At this point, relying solely on experienced hands making judgment calls leads teams into a typical cycle: everyone says it feels like things got better recently, but no one has evidence.

Writing test cases solves this problem. It transforms “got better” from an adjective into a pass rate. Whether it’s actual improvement or spinning in place becomes clear after one test run.

But turning vague feedback into test cases requires dirty work in between: reading original conversations. There’s no shortcut here. Anthropic has a saying that scrutinizing tokens requires the same care as scrutinizing pixels. One failure might be the model calling the wrong tool, the retrieval layer finding the wrong knowledge, or an alignment issue. These three types of failures require three different fixes and need to be routed to three different teams. Miscategorize them, and all the effort goes into fixing the wrong thing—the pass rate rises, but users keep complaining.

Test Cases Are Effective, But They Expire

Turning feedback into evals is essentially test-driven development, which the software industry has practiced for decades: write tests first, then discuss implementation. This analogy holds, but don’t push it too far.

Test cases have an inherent weakness: they can be gamed. Once a team starts optimizing around test cases, rising pass rates and actual product improvement gradually decouple. The model can perform nearly perfectly on those few dozen questions while still failing outside the test suite. So mature evaluation sets need regular question rotation and additions, incorporating new failures that emerge in the real world. Test cases are living assets; once written and left alone, their value decays faster than you’d imagine.

There’s another boundary to recognize. Evals only work for enumerable failures. Wrong format, wrong tool call, wrong answer—these have standard correct answers. But whether a product is engaging, whether interactions flow smoothly, whether the aesthetics resonate—these questions can’t be captured in test cases. Force them into tests, and teams squeeze out what can’t be quantified from priorities, ultimately building a product that passes every test but nobody likes. Part of the craft of writing tests is knowing where not to write them.

Test Cases and Documents: Division of Labor, Not Replacement

Interestingly, Anthropic didn’t throw away requirements documents. The team still writes them for each model generation, but the audience has changed: they’re mainly written for the constantly expanding product and engineering teams, as well as stakeholders in legal and security, to align a large group of people with different backgrounds around the same goals. There’s another scenario where documents become more valuable: when direction is still fuzzy and there’s no clear user pain point in hand. For example, before the capability to operate computer screens launched, the team didn’t know specifically who would pay or why. In such cases, vision-based reasoning helps the team first frame a narrow scope and validate standing value.

Which to use when depends on what stage the problem is in:

Choosing tools by habit is how most teams fall into traps. Choosing tools by problem stage is the intent behind this division of labor.

Starting from Scratch: Three Steps to Your First Test Suite

The barrier is lower than you’d imagine. No infrastructure needed, no training background required.

Step one: pull out recent negative user reviews, read through original conversations line by line, and categorize failure causes: tool calling, retrieval, format, understanding, alignment—what’s the distribution. Step two: pick the highest-volume category, collect twenty or thirty real cases, write them as input-output pairs with expected outputs—this is version one of your test suite. Step three: set a rerun cadence, run it every time you switch models, revise prompts, or adjust workflows, and track the change curve of pass rates.

This capability also changes how people grow. In the past, the more senior you were, the further from execution; the higher up, the more you only approved decisions. In a rhythm where models flip generations annually, this causes problems: your intuition about what’s good or bad stays frozen at the level of the previous model generation. So the practice there is that even people who’ve led teams for years follow the same onboarding plan as fresh graduates, starting from reading real user feedback. Managers claim ownership of work streams and deliver hands-on themselves.

As long as your product is assembled from model plus engineering shell plus specific scenario, writing test cases is unavoidable eventually. The reason is simple: you can’t improve what you can’t measure. Intuition deceives; test cases don’t, but test cases themselves expire. The hand that writes tests can’t stop.

Key points: Vague user feedback must be broken down to original conversations to identify root causes; evals are repeatable test suites that turn “got better” into pass rates; test cases get gamed, require regular question rotation, and only cover enumerable failures; evals manage iterative measurement, documents manage fuzzy direction and cross-team alignment; three startup steps are categorizing failures, creating twenty to thirty input-output pairs, and setting a rerun cadence.