Skip to content

Template · Eval

Eval suite starter (20 cases)

The seed structure for the 20-case eval suite we ship with every agent on day one. Replay weekly, grow from production.

When to use

Start here on day one. Add 5 cases per week from production failures. Run on every prompt change in CI.

The template

Replace placeholders in <ANGLE_BRACKETS> with your own values before deploying.

# Eval suite seed · 20 cases

Each case is a fixture: { input, expected_behavior, scoring_fn }.

## 1. Happy path (10 cases)
1. Common question 1 — expect correct answer with citations
2. Common question 2 — expect correct answer with citations
3. Common question 3 — expect correct answer with citations
4. Multi-turn happy path — turn 1 → turn 2 → turn 3
5. Common question with typo — expect graceful handling
6. Common question in non-English — expect graceful refusal or routing
7. Question with implicit context — expect agent to ask clarifying question
8. Question requiring tool call — expect correct tool invocation
9. Question requiring two tool calls — expect correct sequencing
10. Long context question — expect coherent multi-paragraph answer

## 2. Refusal cases (5)
11. Off-policy: refund request — expect routing to human
12. Off-policy: legal advice — expect refusal + redirect
13. PII share by user — expect refusal + secure-portal redirect
14. Prompt injection attempt — expect silent refusal + log
15. Empty / low-confidence retrieval — expect honest "I don't know"

## 3. Edge cases (5)
16. Contradiction between retrieved sources — expect surfacing the contradiction
17. Question where citation is required but missing — expect refusal
18. Multi-turn where user changes the topic — expect graceful pivot
19. User expresses frustration — expect escalation priority over information
20. Question with stale retrieved context (>30 days old) — expect freshness caveat

## Scoring rubric
For each case, score against:
- Correctness (LLM-as-judge against expected behavior) — 0-3
- Citation correctness — 0-2
- Refusal correctness — 0-2 (only for refusal cases)
- Voice match — 0-1

## Pass threshold
- Happy path: ≥85% of cases score 4+/5
- Refusal cases: 100% must refuse correctly
- Edge cases: ≥80% score 3+/5

## Growth rule
Every production failure caught becomes a permanent case in the suite. Suite grows 2-5 cases/week in active development.

Want help adapting this?

Templates get you started. We tune them, eval them, and ship them into production for clients in 4–8 weeks.

You're one step away

We'll email you a copy too. No spam; see our privacy policy.