How We QA AI Systems Before Launch: Evals, Red-Teaming, and the Test Suite for Things That Talk

Traditional software QA rests on one comforting assumption: the same input produces the same output. Run the test, get the green checkmark, ship. AI systems break that assumption on purpose — the same question can produce a different (correct!) answer every time, which means the classic test suite doesn't just miss bugs, it can't even express what a bug is. So how do you QA a thing that talks? This is the process we run on every AI system we ship — chatbot, RAG assistant, or autonomous agent — before it's allowed near a customer. It's opinionated, it's specific, and it's the difference between a demo and a product.

The one-paragraph version: QA for AI systems replaces "assert output equals expected" with four layers. Evals measure output quality against a golden dataset using a mix of hard checks and LLM-as-judge grading. Red-teaming attacks the system adversarially — prompt injection, jailbreaks, data-extraction attempts — before real attackers do. Trajectory testing (for agents) grades the entire execution path, not just the final answer: did it call the right tools, in the right order, with the right permissions? And production monitoring continues the QA after launch, because no offline test suite survives contact with real users. A system passes our release gate when all four layers clear defined thresholds — and the failing cases from every layer become permanent regression tests.

Why "it worked in the demo" means nothing

Three properties of LLM systems make traditional QA insufficient, and naming them explains everything we do afterward:

Non-determinism. Because outputs vary between runs, teams combine offline evals on golden datasets, runtime guardrails, observability tracing, and adversarial red-teaming instead of assertion-based tests — the industry converged on this stack because nothing simpler works. A single successful demo run is a sample size of one from a distribution you haven't characterized.

The application is the attack surface, not the model. The base models from Anthropic, OpenAI, and Google arrive heavily safety-tested. But your system is the model plus your prompts, your retrieved documents, your tools, and your business logic — and that assembly is where vulnerabilities live. A classic pentest misses this entirely; the AI attack surface spans the prompts, the retrieval corpus, the tools and agents, and the application logic around them.

Failure is behavioral, not binary. A traditional bug crashes. An AI bug politely offers your customer a discount that doesn't exist, cites a policy from a document it shouldn't have access to, or agrees with a false premise. These failures return HTTP 200. Only behavioral testing catches them.

Layer 1: Evals — the golden dataset and the judge

An eval is the AI equivalent of a test suite: a curated set of inputs with defined expectations, run against the system on every meaningful change.

Building the golden dataset. For a client project this starts from real material: actual support tickets, real customer questions, transcripts from the process being automated. We aim for 100–300 cases at launch, deliberately stratified: the common cases (the 60% the system will see daily), the edge cases (ambiguous phrasing, multiple questions in one message, mixed languages — critical for our multilingual deployments), the should-refuse cases (out-of-scope requests, questions the bot must escalate rather than answer), and the known-hard cases discovered during development. Each case carries expectations at the right level of strictness — sometimes an exact fact ("order cutoff is 2pm IST"), sometimes required elements, sometimes just criteria for a judge to apply.

Grading without a human reading 300 outputs per run. Two mechanisms, layered. Deterministic assertions catch the objective stuff cheaply: did the answer include the mandatory disclaimer, is the JSON valid, did it stay under the length limit, did it avoid forbidden phrases. LLM-as-judge handles the subjective dimensions — accuracy against the source document, tone, completeness — by having a strong model grade each output against written criteria. The critical discipline everyone skips: calibrate the judge. We hand-grade a sample, compare with the judge's scores, and tune the grading prompt until agreement is high. An uncalibrated judge is a random number generator with confidence.

For RAG systems, we eval retrieval separately from generation. Two different failure modes hide inside one bad answer: the system retrieved the wrong documents (retrieval failure) or retrieved the right ones and misread them (generation failure). Measuring them separately — did the correct chunk appear in the top-k? was the answer faithful to the retrieved text? — tells you which knob to turn. Teams that only eval final answers end up fixing the wrong layer.

Layer 2: Red-teaming — attacking it before someone else does

Red-teaming is systematically probing the system with adversarial inputs to uncover vulnerabilities — and in 2026 it has frameworks, tooling, and (in the EU) regulatory teeth: OWASP published its first agentic Top 10 in December 2025, and the EU AI Act's adversarial testing obligations are already in effect for general-purpose AI, with high-risk system requirements landing August 2026. Prompt injection has held the #1 spot on the OWASP LLM Top 10 for two consecutive years, which tells you where to start.

Our attack list for a typical customer-facing system:

• Direct prompt injection: "Ignore previous instructions," role-play framings, system-prompt extraction attempts, encoding tricks. The classics still work on undefended systems more often than anyone admits.
• Indirect injection: malicious instructions hidden in the content the system reads — a product review, an email, a webpage the agent browses. For any system that ingests untrusted text, this is the scariest category, because the attacker never talks to your bot directly.
• Data-boundary attacks: attempts to make a RAG system surface documents the asking user shouldn't see, cross-customer information leaks, "what did the previous user ask?"
• Business-logic abuse: talking the bot into unauthorized discounts, refund promises, or commitments — the Air Canada category of failure, where a chatbot's invented policy became a court-enforced one.
• Off-policy content: competitor comparisons the client forbids, medical/legal advice from a bot with no business giving it, toxic-input handling.

Method: automated breadth plus human depth. Open-source tools (Promptfoo, garak, PyRIT) generate hundreds of attack variants cheaply; a human then spends focused hours on the creative, multi-turn, context-specific attacks automation misses — because real vulnerability discovery rarely looks like running a benchmark. Every successful attack gets fixed at the architecture layer where possible (scoped retrieval permissions, output filtering, tool-call limits) rather than by prompt-patching, and then joins the regression suite permanently.

Layer 3: Trajectory testing — QA for agents specifically

For agents — systems that plan and execute multi-step work with tools — grading the final answer isn't enough. Trajectory evaluation scores the entire execution path: every tool call, every intermediate step, every turn. An agent can produce a correct-looking answer via a path that included calling a tool it shouldn't have, retrying destructively, or leaking information between steps.

What we grade per trajectory: tool selection (right tool for the step?), argument correctness (the difference between `refund(order_123)` and `refund(order_124)` is a very unhappy customer), permission discipline — the failure mode the security field calls *excessive agency*, where an agent granted more permissions than the task warrants turns a bad prompt into a real incident — recovery behavior (what happens when an API times out mid-task: retry, escalate, or hallucinate success?), and loop/cost bounds (agents that retry forever fail as a billing line item). This is also where we set the autonomy budget explicitly: which actions the agent may take alone, which require confirmation, and which are hard-blocked. Those limits are tested like any other requirement — we try to make the agent exceed them.

Layer 4: The release gate, and what launch doesn't end

Before anything ships, the numbers have to clear a written bar — agreed with the client, not invented by us: eval pass-rate thresholds overall and on the must-not-fail subset (refusals, escalations, policy lines are 100%-required, no averaging away), zero unresolved high-severity red-team findings, trajectory scores within bounds, plus the boring-but-mandatory latency and cost-per-interaction checks. The full suite runs in CI, which means a prompt tweak, a model version bump, or a knowledge-base update cannot ship without re-passing everything — and this is the part most teams skip. Model updates change behavior in unannounced ways; the regression suite is your only early warning.

And then launch, where the honest truth applies: no offline suite survives contact with real users. Post-launch, the QA continues as monitoring — full tracing of production interactions, automated flagging of anomalies (spikes in refusals, drops in judge-scored quality, unusual tool-call patterns), and systematic human review of samples, calibrated against the same judges used offline. Distribution drift and unanticipated real-world failures can only be caught after launch by design. The loop closes the same way every time: production failures become new eval cases, the suite grows, and next month's system is tested against this month's surprises.

What this looks like on a real engagement

Practically, QA is woven through the build, not bolted on the end: golden-dataset collection starts at discovery (the client's real tickets and transcripts are the spec), evals run from the first prototype week, red-teaming happens twice (mid-build on the architecture, pre-launch on the full system), and the release gate is a shared document the client signs off on — with the eval dashboard visible to them, not just us. The whole apparatus typically adds 15–20% to the build effort and removes the single largest risk an AI project carries: shipping a system whose failure modes nobody has met yet.

If you're evaluating an AI vendor, here's the tell: ask to see their eval suite for your project. Not their demo — their tests. A vendor who can show you golden datasets, judge criteria, red-team findings, and a written release gate is doing engineering. A vendor who answers "we tested it a lot and it works great" is doing vibes.