Evals before features

When a client asks for an AI feature, our first deliverable is not a prototype. It is a spreadsheet of examples and a definition of what “good” means for each one. The prototype comes second.

This feels backwards to teams used to building first and polishing later. With AI features it is the only order that works, and this post explains why and how we do it.

If you cannot say what a good answer looks like, the model cannot either.

Why the test comes first

A language model feature is easy to demo and hard to trust. Three good outputs in a meeting say almost nothing about the next thousand. Without a fixed set of cases, every change is judged by feel: someone tries two prompts, likes one, and ships it.

Writing the evals first gives us:

• A spec. Writing ten examples forces the team to disagree early about what the feature should do. That argument is cheaper in a spreadsheet than in production.
• A baseline. We can score the dumbest possible approach, such as a single prompt with no tools, and know what every later addition has to beat.
• A way to say no. Some features should not ship yet. A score that sits below the bar is a better argument than anyone’s opinion.
• Freedom to change things. Swapping models, rewriting prompts or adding retrieval is safe when a check tells you what broke.

The set does not need to be large or clever to start paying for itself.

Build a golden set of 30

Thirty examples is enough to start. It is small enough to read every one, and large enough to expose obvious failures. We build it in one sitting with the people who know the domain.

Pull from real material wherever possible: actual support tickets, real product listings, documents the system will see. Remove personal data first. If no real data exists yet, write realistic inputs by hand, and replace them as production data arrives.

Aim for a spread:

• Typical cases (about half). The everyday inputs. These protect the core behavior.
• Hard cases (a quarter). Ambiguous wording, messy formatting, mixed languages, long inputs. Indian businesses often get Hindi, Gujarati and English in a single message, so we include that when it applies.
• Refusal cases (a few). Inputs where the right behavior is to decline, ask a question or hand off to a person.
• Past failures. Anything that has already gone wrong goes in permanently.

For each example, write down the input, the ideal output or the properties a good output must have, and a short note on why. The note matters most. Six months later, it is the reason a reviewer understands a failing case.

Do not generate the whole set with a model and call it done. Generated examples can seed ideas, but a human has to read, fix and approve each one, or the set will quietly encode the model’s own habits.

Grade with rubrics

Some outputs have one correct answer, and exact match or a schema check is enough. Extracted fields, classifications and structured JSON usually fall here. Use code to grade them. It is fast, free and never moody.

Free-text outputs need rubrics. A rubric is a short list of yes/no questions about the output, each tied to something the user cares about:

• Does it mention the return window stated in the source text?
• Does it avoid promising anything the policy does not cover?
• Is it under 80 words?
• Does it use the customer’s name correctly?

Binary criteria beat a 1 to 10 score. People and models both grade more consistently on “yes or no” than on “how good, out of ten.” A case passes when every required criterion passes; softer criteria add to a score we track over time.

You can use a model as the grader for rubric items code cannot check. Two precautions keep that honest. First, calibrate it: hand-grade a dozen outputs, compare with the model grader, and read every disagreement. Second, keep the grader’s prompt fixed and versioned, so a score change reflects the system and not the judge.

Make CI the gatekeeper

An eval that only runs when someone remembers is a document. An eval that blocks a merge is a control. We run the suite on every pull request that touches prompts, tools, retrieval, model settings or the code around them.

A config like this keeps the gates explicit and reviewable:

# evals/gates.yaml
suite: product-copy
cases: evals/golden.jsonl
runs_per_case: 3            # sample several times; models vary

gates:
  schema_valid:    { min: 1.0 }    # structured output must always parse
  hard_rules:      { min: 1.0 }    # banned claims, required fields
  rubric_pass:     { min: 0.90 }   # share of cases passing every required criterion
  no_regression:   { against: main, tolerance: 0.0 }

on_fail:
  comment_on_pr: true
  attach: [failed_cases, diffs_vs_main]

Several decisions are packed into those few lines.

Run each case more than once. Model outputs vary. Three runs per case show whether a pass is stable or lucky, and a case that flips is worth investigating in its own right.

Separate hard rules from quality. Some failures are never acceptable, such as a made-up price or a missing disclaimer. Those gates sit at 1.0. Quality scores get a threshold you can negotiate.

Compare against main. An absolute threshold lets a slow slide go unnoticed as long as the number stays above the bar. A comparison with the main branch catches any case that used to pass and now fails.

Show the failures. A red check with no detail teaches people to ignore it. The pull request comment lists the failing cases with the input, the output and the criterion that failed.

Grow the set as you learn

The golden set is a living file. Every time a user reports a bad output, we reproduce it, add it as a case, and only then fix it. Over a few months the set becomes a precise map of where the feature is fragile.

Keep it honest as it grows. Review it on a schedule, remove duplicates, and watch for criteria that never fail, since those add no information. If a gate is always green, either the feature is excellent or the test is too easy, and the second is more common.

Also keep a small held-out slice that nobody tunes against. When prompts are edited until the main set passes, scores rise without the feature improving. The held-out cases tell you whether the improvement is real.

Where this pays off

Evals do not make a model smarter. They make the team faster and calmer, because every change comes with evidence. Clients can read a score report instead of taking our word for it, and a model upgrade becomes an afternoon of running the suite rather than a week of nervous manual checks.

Start with thirty examples and a script that fails when quality drops. You will learn more from the first run than from the next ten demos.

#evals#testing#llm#ci