Evals as a permanent system, not a launch checklist

The eval-at-launch trap

Most teams treat evals like a launch checklist. Write 20 test cases, run them, ship the model, move on. Six months later the model is silently degraded, the team doesn't know, and a customer escalation surfaces what should have been caught weeks earlier.

The right mental model, and the one behind our eval suite playbook: evals are a permanent system. Closer to your monitoring stack than to your test suite.

Starting from nothing

If you have no evals today, the first version should take an afternoon rather than a quarter, because a small suite that exists beats a comprehensive one that is still being designed.

Pull ten real interactions from the last month — five where the system did the right thing, five where it did not. For each, write down the one property that made it right or wrong, in a sentence. That sentence is the assertion. Score them by hand the first time, so you learn what is actually easy and hard to judge before automating any of it.

That is a working suite. It will be uneven and incomplete, and it will still catch the next regression, which is the only thing that matters at this stage. Automation, judges, and CI wiring are refinements you add once the habit exists — and they are far easier to justify once the crude version has already caught something.

What permanent means in practice

Eval suites grow weekly. Every production failure we catch becomes a permanent case in the suite. The suite runs against every prompt change, every model change, and weekly against production output regardless.

Failing the eval blocks merge. Drift over time triggers a model review. The owner of the suite (yes, named owner) reports to the same review cadence as the rest of the engineering org.

What makes a case worth scoring

A suite of twenty is only strong if the twenty are well chosen, and most first drafts are not. The failure mode is a set of cases that all pass, forever, and therefore tell you nothing.

A good case pins one behaviour. If a case can fail for four different reasons, a failure tells you something broke without telling you what. Split it. One case for "refuses when the source is missing," another for "cites the correct passage when it is present."

A good case has a stable expected outcome. Anything whose correct answer depends on today's data will fail for reasons unrelated to model quality, and a suite that cries wolf gets ignored within a month. Freeze the inputs, or assert on the shape of the answer rather than its content.

A good case would have caught a real bug. The most valuable cases in any mature suite are the ones written the day after an incident. If you cannot name the failure a case protects against, it is probably decoration.

A good case survives a rewrite of the prompt. Cases that assert on exact wording break every time someone edits the system prompt, which trains the team to update expectations rather than investigate. Assert on the properties that matter — did it refuse, did it cite, did it stay inside the policy — not on the phrasing.

The corollary is that cases should be retired. A case protecting against a bug in a code path that no longer exists is pure maintenance cost, and the reluctance to delete cases is how suites drift past the point where anyone runs them.

The 20-case minimum, and the 200-case ceiling

20 hand-picked cases beat 200 synthetic ones. The 20 cover happy path, common edge cases, refusal patterns, and the patterns you can't afford to break. Below 20 you're not really testing. Above ~200 you're spending more cycles on eval maintenance than on the agent itself.

The healthy distribution we see in client projects: 60% real anonymized production cases, 30% adversarial / edge cases we wrote, 10% regression cases pulled from past bugs.

The regression that isn't one

Model output varies between runs, which means an eval suite produces a distribution rather than a number — and treating a single run as truth generates false alarms that erode trust in the whole system.

Establish variance before you set a threshold. Run the unchanged suite five times against the unchanged system and look at the spread. If scores move two points run to run, a two-point drop is noise and must not block a release. Teams that skip this calibration end up either ignoring real regressions or chasing imaginary ones, and both end with the suite being disabled.

Set the temperature deliberately. Scoring at the temperature you run in production tells you about the real system; scoring at zero gives you a more stable signal for detecting change. Both are defensible. Doing one while believing you did the other is not.

Watch per-case flips, not just the aggregate. An overall score that holds steady while six cases flip from pass to fail and six flip the other way is a system that changed materially. The aggregate hides exactly the movement you most want to see.

Re-run before you investigate. A single failing run on a flaky case costs an engineer an afternoon if the first response is to start debugging. Make a re-run the automatic first step, and record how often it changes the answer — a case that flips on re-run more than occasionally is a badly written case, not a badly behaving model.

LLM-as-judge — where it works, where it breaks

LLM-as-judge is the only way to score evals at scale. For most criteria (faithfulness, refusal correctness, tone match) it works well enough.

It breaks on three things:

1. Subtle factual accuracy. When the ground truth requires domain expertise the judge model doesn't have, scores get noisy.
2. Subjective quality. "Is this tactful enough?" — humans disagree, and the judge model often agrees with whichever side phrased the prompt.
3. Multi-turn coherence. Judge models score individual turns better than whole conversations.

Fix: human review on the close calls. Score 80%+ of cases automatically with LLM-as-judge, route the bottom decile to a human. That's where tuning judgment lives.

Wiring it into CI without stopping the line

An eval suite that blocks every merge on a full run is a suite that gets bypassed within a fortnight. The trick is matching the depth of the check to the risk of the change.

Three tiers work well. A fast subset — five to eight cases, under a minute — runs on every commit and catches gross breakage. The full suite runs on any change to a prompt, a retrieval parameter, or a model version, and blocks the merge. The full suite plus adversarial cases runs nightly against production configuration, where it is allowed to take as long as it needs.

Route on what changed, not on who changed it. A change to the system prompt is a high-risk change regardless of how small the diff looks; a change to a logging call is not. Wiring the tier selection to the file path is a few lines and removes the judgment call.

Make failures legible. A CI failure that says "eval score 0.81, threshold 0.85" tells an engineer nothing about what to do. Print the cases that flipped, with input, expected property, and actual output. The difference between a suite people fix and a suite people mute is almost entirely the quality of that output.

Budget the run cost. A full suite against a frontier model, several times a day, is a real line item. This is where the cost telemetry above earns its place: it tells you whether nightly is affordable, and which cases are quietly consuming the budget.

Cost telemetry inside the eval suite

Every eval run logs token cost. You learn how much each test case costs and which ones are pulling the average up. Over time you spot "this prompt change improved quality but tripled cost" before the bill arrives.

We build this into every eval suite we ship now. It started as a defensive measure and turned into a useful product-economics signal.

When the provider changes the model underneath you

The scenario the launch checklist never covers: nothing in your repository changed, and quality moved anyway.

Providers update models behind stable-sounding identifiers, deprecate versions on their own schedule, and adjust safety behaviour without a version bump you would notice. For a system with an eval suite this is an inconvenience; for a system without one it is invisible until a customer finds it.

Pin the version wherever the provider allows it, and treat the pinned identifier as a dependency with an owner and a review date rather than a string that got typed once. An unpinned model is an unversioned dependency in the most behaviour-critical position in the stack.

Run the suite on a schedule, not only on change. Weekly is enough for most systems. It is the only mechanism that catches provider-side drift, because there is no commit to trigger on.

Keep one frozen baseline. Store the scored outputs from a known-good run. When the number moves, comparing against a frozen artefact tells you within minutes whether behaviour shifted or your scoring did — a distinction that is otherwise surprisingly hard to establish under pressure.

Test the upgrade path before you need it. When a deprecation notice arrives with a date on it, the team that has already scored the successor model against the same suite has a decision. The team that has not has an emergency.

The cases teams systematically forget

Most suites over-index on the happy path because that is what the demo covered. The gaps are consistent enough to be worth listing.

The empty and the absurd. No input, a single character, ten thousand words pasted in, a question in a language the corpus does not cover. These cost nothing to add and catch a surprising share of production incidents, because real users do all of them within the first week.

The adversarial-but-plausible. Not jailbreaks — the ordinary questions with an unsafe answer. A request for advice the company is not licensed to give, a question whose honest answer exposes another customer's data, a prompt that invites the model to speculate about a person. These need to be in the suite because the correct behaviour is a refusal, and refusals are exactly what nobody thinks to test.

The multi-turn trap. A first message that is fine, followed by a second that reframes it. Most suites score single turns, so an agent that can be walked somewhere over three messages passes cleanly. If your product has conversation, your suite needs conversations.

The cases with no good answer. Sometimes the right output is "I don't know" or "that isn't something I can help with." Teams leave these out because they feel like failures, and their absence is why systems over-answer. A suite with no unanswerable cases is training you to reward confidence.

The ones from the last incident. Obvious, universally agreed, and routinely skipped in the rush to close the incident. If your post-incident checklist does not have "add the eval case" on it, the same bug will return, and the second time it will be harder to explain.

Who owns it after we leave

This is the question we ask in every handoff. If the answer is "we'll figure it out", we know the eval suite will atrophy. If there's a named human with the eval review on their calendar, it usually survives.

We've started writing the named owner into the SOW. Not as a deliverable from us — as a commitment from the client. The clients who push back are the ones whose evals will die. We bring this up before signing, not after.