The Most Valuable Work for AI Product Managers Is Writing Exams for Models

Deep thoughts on AI and aspirations —— ByteDance Deep Thinking Circle

In 2023, among the user complaints Anthropic received, one appeared frequently: Claude doesn’t follow instructions well.

This statement alone doesn’t tell you what to fix. Hand it to researchers, they don’t know where to start. Give it to engineers, they can’t pinpoint the problem. The Anthropic product team responsible for this area broke down the vague complaint: what exactly did users ask, what exactly did Claude respond, reading through original conversations one by one. After going through them all, they found that roughly 80% of “not following instructions” actually pointed to the same thing: the model couldn’t reliably output correctly formatted JSON.

What happened next was the key move. The team collected dozens of real examples where Claude produced incorrect output, organized them into a set of input-output pairs, and ran them before every new release until they achieved near-perfect results. A vague complaint had been transformed into a repeatable exam.

This approach is called eval, or evaluation set. A saying circulates inside Anthropic: evals are the new PRDs. My assessment is more direct: for people building AI products, the most valuable work is shifting from writing documents to writing exams.

Vague Feedback Equals No Feedback

Traditional product managers prove their product strategy by writing polished requirement documents or creating prototypes backed by user interviews. This approach rests on a premise: product behavior is predictable. Once you specify the features clearly, what you build is what you get.

Products built on models don’t have this premise. You revise a prompt and it feels somewhat better; you switch model versions and it seems worse again. Intuition fails in the noise. The same user complaint can’t be reproduced today, then resurfaces tomorrow. At this point, relying solely on experienced judgment leads teams into a typical loop: everyone says it feels better lately, but no one has evidence.

Writing exams solves this problem. It transforms “got better” from an adjective into a pass rate. Whether you’ve made progress or just spun in place becomes clear after one run.

But turning vague feedback into exams involves dirty work in between: reading original conversations. There are no shortcuts here. Anthropic has a saying: scrutinize tokens as carefully as you scrutinize pixels. A single failure could mean the model called the wrong tool, the retrieval step found the wrong knowledge, or alignment went off track. These three types of failures require three different fixes and need to be routed to three different teams. Misclassify the failure, and all your fixing effort goes in the wrong direction. The pass rate goes up, but users keep complaining.

Exams Work, But They Expire

Turning feedback into evals is essentially test-driven development, something the software industry has practiced for decades: write tests first, then talk implementation. This analogy holds, but don’t push it too far.

Exams have a natural weakness: they leak. Once teams start optimizing around the exam, rising pass rates and actual product improvement slowly decouple. The model can perform nearly perfectly on those dozen questions while still failing outside the exam. So mature evaluation sets need regular rotation and expansion, incorporating new failures that emerge in the real world. Exams are living assets. Write them and leave them there, and their value decays faster than you’d imagine.

There’s another boundary to recognize. Evals only work for enumerable failures. Wrong format, wrong tool call, wrong answer—these can have standard answers. But whether a product moves people, whether interactions flow, whether the aesthetic resonates—these types of questions can’t be turned into exam problems. Force them into exams, and teams will squeeze unquantifiable things out of priorities, eventually building a product that gets every question right but that nobody likes. Part of the craft of writing exams is knowing where not to write them.

Exams and Documents: Division of Labor, Not Replacement

Interestingly, Anthropic didn’t throw away requirement documents. For each model generation, teams still write them, but the audience has changed: mainly for the constantly expanding product and engineering teams, plus related parties like legal and security, used to align a large group of people with different backgrounds around the same goals. There’s another scenario where documents are more valuable: when direction is still fuzzy and there’s no clear user pain point. For example, before launching the ability for models to operate computer screens, teams didn’t know specifically who would pay and why. In such cases, vision-level reasoning helps teams first frame a narrow scope and validate value that holds up.

Which to use when depends on what stage the problem is in:

Choosing tools by habit is how most teams fall into traps. Choosing tools by problem stage is the intent behind this division of labor.

Starting from Zero: Three Steps to Your First Exam Set

The barrier is lower than you’d imagine. No infrastructure needed, no training background required.

First step: pull recent negative user feedback, read original conversations line by line, categorize failure causes: tool calls, retrieval, formatting, comprehension, alignment—what’s the distribution. Second step: pick the highest-proportion category, collect twenty to thirty real cases, write them as input-output pairs. That’s your first exam version. Third step: set a rerun cadence. After every model switch, prompt revision, or workflow adjustment, run it once and track the pass rate trend.

This capability also changes how people grow. Previously, the more senior you became, the further from execution, the more you only signed off. At the pace of models turning over multiple generations per year, this approach causes problems: your intuitions about what’s good will freeze at the previous model generation’s level. So their practice is that even for people who’ve led teams for years, the onboarding plan is the same as for fresh graduates, starting with reading real user feedback. Managers claim work streams themselves and deliver hands-on.

As long as your product is assembled from model plus engineering shell plus specific scenario, writing exams is unavoidable eventually. The reason is straightforward: you can’t improve what you can’t measure. Intuition deceives, exams don’t—but exams themselves also expire. The hand that writes exams can’t stop.

Key points: Vague user feedback must be broken down to original conversations to locate root causes; evals are repeatable exams that transform “got better” into pass rates; exams leak and need regular rotation, covering only enumerable failures; evals handle iterative measurement, documents handle fuzzy direction and cross-team alignment; the three startup steps are categorizing failures, creating twenty to thirty paired questions, and setting a rerun cadence.