Proving It Works

The bar here is far lower than people think, and almost nobody clears it. Most systems go into production on the strength of a demo and a good feeling. That is how a company ends up unable to answer the simplest question anyone will ask.

That question is: how do you know it works. Not how does it work. How do you know.

The answer should be a number, produced the same way each time, that you can watch move. Getting there takes an afternoon for most systems, and it is the single clearest signal that separates a serious practice from an enthusiastic one.

Benchmarks are not your evidence

A model's score on a public benchmark tells you something about the model in general and nothing about whether it does your job. Your job has your documents, your edge cases, your definition of a good answer. You have to build your own measurement, and it is smaller than you fear.

Build the golden set

Collect twenty to fifty real examples of the task, with known correct answers. Real ones, from your actual history, not invented ones, because invented examples are always cleaner than reality and they will flatter the system.

Composition matters more than size:

• Ordinary cases, maybe half. The boring middle.
• Known hard cases. The ones a new employee gets wrong in their first month. Ask whoever does this work today. They can list them from memory.
• Cases that should be refused or escalated. Requests outside scope, missing information, ambiguity a person should resolve. Systems that never decline are a specific and common failure.
• Two or three genuinely nasty ones you would not blame a person for missing.

Thirty well-chosen examples beat three hundred scraped ones. The work is in the choosing and it is done by the person who knows the job, not by whoever is building.

Decide what counts as right

Before running anything, write down what right means, because "it looked good" is not repeatable and a disagreement about it at month four is painful.

• Exact match for extraction and classification. Which number, which category. Easy to score, and this covers more real tasks than people expect.
• Contains the required facts for summaries and drafts. List the three things that must appear. Ignore style.
• A rubric scored by a second model for open-ended output. Workable, cheap, and you must check the grader against human judgment on a sample first, or you have automated an opinion rather than a measurement.
• Human review for the genuinely subjective. Smaller sample, fixed rubric, more than one reviewer on the first pass so you can see whether your own people agree with each other. Often they do not, which is a finding worth having.

Run it and write it down

Run the set, record the score, and keep the failures. The score alone is nearly useless. The failures are the entire value, because they cluster, and each cluster is a fix.

Then set a threshold before you look at the result. What score would make you willing to ship this. Deciding after you see the number is how everything ships.

This is also what makes the model choice in chapter nine decidable. Run the same set against the cheap model and the expensive one. The question stops being an argument about capability and becomes a table with two columns.

Watching it in production

The golden set tells you it worked on Tuesday. Production tells you whether it still does. Four things are worth logging from day one, and all of them are cheap:

• Volume. Runs per day. A sharp drop usually means people quietly stopped using it, which is information you want early.
• Failure and retry rate. Rising retries mean something upstream changed.
• Human override rate. The most useful number in the whole system. If people edit the output ninety percent of the time, it is not working regardless of your eval score, and if they edit it two percent of the time, they may have stopped reading it.
• Cost per task, tracked over time rather than at launch.

Sample real outputs on a schedule, monthly is fine, and add anything interesting to the golden set. The set should grow over the life of the system. That growing file is the best documentation you will have of what the system is actually for.

Why this is the afternoon that pays

Beyond the obvious, the eval set changes the conversation you are able to have. When someone asks whether the system is trustworthy, you have a number and a method rather than a reassurance. When someone wants to swap models to save money, you can answer in twenty minutes. When something breaks in production, you have a before and after.

And it is a governance artifact, not just an engineering one. An auditor asking how you validate an automated decision is asking for exactly this, and the honest answer either exists or it does not.