I went looking for good writing on evaluating AI features from a product perspective and found almost none.
How do you evaluate an AI feature as a product manager?
AI evaluation is two jobs. The engineering half — harnesses, scoring, regression gates — is well covered. The product half is not: deciding what a correct answer is for your task, who adjudicates disagreements, and who owns the test set as the world changes. The second half determines whether the first measures anything useful.
There’s a great deal of material on evaluation. Nearly all of it is published by companies selling evaluation tooling, and nearly all of it is addressed to engineers. It’s often technically excellent. It’s also silent on the questions that decide whether any of it works, which aren’t engineering questions.
Who decides what a correct answer is? Who adjudicates when two of your experts disagree? Who owns the test set when your business changes and it slowly stops representing reality? Those are your decisions, and they’re the hard half.
The two-layer picture, briefly
The practitioner consensus on the mechanics is fairly settled, and useful to know even if you never touch it yourself.
Offline evaluation runs a curated set of examples against every change, before release, and blocks anything that makes quality worse. Online evaluation samples real traffic in production and watches for quality moving over time. The distinction people repeat is that offline catches the changes you make, and online catches the changes that happen to you.
That second category is the one product managers underestimate. A model provider can update the model behind an unchanged API identifier — this happened with GPT-4 Turbo more than once during 2023 and 2024, with documented user complaints about behaviour shifting. Your code did not change, your prompt did not change, and your product did.
Your code didn’t change. Your prompt didn’t change. Your product did. If nobody is watching production, you find out from a customer.
The part nobody assigns
At the centre of all of this sits the golden dataset: a curated set of examples with known-good answers. Guidance converges on something like a hundred to a few hundred cases, weighted towards real failures and edge cases rather than volume, with human review on a sample to keep any automated scoring honest.
Fine. So what actually decides whether it works?
Who says what the right answer is? For a summarisation feature — three domain experts will produce three different good summaries. Somebody has to decide what the bar is. That’s not a data science decision and it can’t be delegated to whoever is building the harness.
What happens when experts disagree? They will, and the disagreement is information. If two experienced people read the same case differently, the task is ambiguous, and shipping a model into a ambiguous task with a single expected answer produces a metric that measures conformity to one person’s opinion.
Who owns the set over time? This is the one I’ve seen fail most. You build it during the project. Everyone is proud of it. Then your product changes, your customers shift, new categories appear, and eighteen months later it’s scoring your system against a world that no longer exists. A stale test set is worse than none, because it hands you a confident green number.
What is the failure taxonomy? An aggregate score of 0.87 tells you almost nothing, which is the coverage mistake wearing a decimal point. Which cases fail, and do the failures share a shape? A model that’s wrong uniformly across everything is a different product problem from one wrong on a specific customer segment, and the second is usually more serious and less visible.
On using a model to grade a model
The standard approach when human review doesn’t scale is to have an LLM score the outputs, calibrated against human judgements on a sample.
It works well enough to be a good idea, and it has one failure mode nobody names: the judge is a model with its own biases, and when you update the judge your metric moves without your product changing. If you’re reporting that number upwards, you’re reporting something that can shift for reasons unrelated to quality.
I treat the judged score as an indicator rather than a measurement. Anything I am going to defend in a steering committee gets a human sample behind it.
The question that outranks all of the others
None of the above matters if you skipped this one.
What does a wrong answer cost, and would anyone notice?
An evaluation programme is an investment in detecting errors. How much you should spend depends entirely on what one of your errors does. A wrong suggestion in a drafting tool that a person reviews is cheap, whereas a wrong answer in a customer-facing statement about someone’s money is expensive in ways that reach beyond the ticket. Pricing that difference is the sizing work.
If a wrong answer would go unnoticed, no evaluation setup saves you, because your problem is the workflow rather than the measurement. That’s the same argument as meaningful human oversight, reached from the measurement side.
What I ask for before a launch
Four things, and none of them need me to open a harness.
What does correct mean for this task, in a sentence a domain expert would sign. Who decided that, and what happened where they disagreed. What are the failure modes we know about and which are we accepting. And who is looking at production quality in three months, by name.
The fourth is the one that gets a vague answer, and a vague answer there means you’ve built a launch rather than a product.
The uncomfortable summary
Evaluation is usually described as a technical maturity problem, and framing it that way puts it in someone else’s backlog.
Most of what makes evaluation work is deciding things: what good means, who arbitrates, what you’re prepared to be wrong about, and who is accountable for noticing when it drifts. Those decisions belong to whoever owns the product. If they aren’t made, the harness will run beautifully and measure the wrong thing, and you’ll find out from a customer.
Common questions
How do you evaluate an AI feature as a product manager?
Define what a correct answer means for the specific task, decide who adjudicates when experts disagree, assign ownership of the test set over time, and name who will be reviewing production quality months after launch. The engineering layer — harnesses, scoring, regression gates — is well covered elsewhere; these four decisions are the ones that determine whether it measures anything useful.
What is a golden dataset?
A curated set of examples with agreed correct answers, used to score every change before release. Guidance generally suggests a hundred to a few hundred cases, weighted towards real failures and edge cases rather than sheer volume, with human review on a sample. The harder question is who decides what the correct answer is, and who keeps the set current as the product and customers change.
What is the difference between offline and online evaluation?
Offline evaluation scores a curated set before release and catches regressions you introduce. Online evaluation samples live traffic and catches changes that happen to you, including model providers updating a model behind an unchanged API identifier, which occurred with GPT-4 Turbo more than once in 2023 and 2024. Most mature teams run both.
Can you use an LLM to evaluate another LLM?
Yes, calibrated against human judgements on a sample, and it's often the only approach that scales. The caveat is that the judge is itself a model with biases, so updating the judge can move your metric without your product changing. Treat a judged score as an indicator, and put a human sample behind any number you intend to defend to executives.
What happens when domain experts disagree about the right answer?
The disagreement is information rather than a nuisance. If two experienced people read the same case differently, the task is ambiguous, and scoring against a single expected answer will measure conformity to one person's opinion rather than quality.
Why do AI evaluation setups stop working?
Most often the test set goes stale. It is built during the project, then the product changes, customers shift and new categories appear, and it eventually scores the system against a world that no longer exists. A stale set is worse than none, because it produces a confident green number.
What should you ask before launching an AI feature?
What correct means for this task in a sentence a domain expert would sign; who decided that and what happened where they disagreed; which failure modes are known and which are being accepted; and who is looking at production quality in three months, by name. The last question is the one that usually gets a vague answer.
I’m an AI product manager working across fintech, SaaS, and regulated enterprise — currently leading AI and workflow product at T-Systems International. If you’re building AI governance into a product right now and want to compare notes, I’m at csincsakf@gmail.com or on LinkedIn.