Playbooks
The patterns we ship — written down.
What we'd brief a new engineer on before they touch a real engagement. Read here or download the PDFs to circulate inside your team.
All three exist because of the same failure mode: AI systems demo beautifully and then fall over on the cases nobody designed for. Agents built for the happy path meet an exception. Retrieval that looked fine on curated documents meets a real knowledge base. Prompts get tuned by feel until a regression reaches a customer. Each playbook is the countermeasure we now apply by default rather than after the incident.
- Playbook8 min6 pages
Production agent design
How to design agents that ship — guardrails, escalation, observability.
Most teams stall agents at the demo stage because they design for the happy path and figure out exceptions later. We design exception-first. This is the playbook we use on every agent engagement.
Inside
- 1. Define the action surface first
- 2. Wrap every write action in approval
- 3. Build the escalation path before the happy path
- 4. Observability is non-optional
- 5. Eval before launch, eval after launch
- Playbook7 min5 pages
RAG that actually grounds
Retrieval, reranking, citations, refusal. The pattern that stops hallucination.
Most RAG demos hallucinate within 10 minutes of real use. Production RAG doesn't. The difference is process: hybrid search, reranking, citation-required prompts, and refusal patterns. We ship this on every chatbot engagement.
Inside
- 1. Hybrid search beats vector-only
- 2. Reranking is the cheapest precision lift
- 3. Chunk smartly, not uniformly
- 4. Citation-required prompting
- 5. Refusal patterns
- Playbook6 min4 pages
The eval suite you actually keep running
20 test cases, replayed weekly, evolving with production. The minimum for shipping.
LLM evaluation isn't optional. An eval suite is the system that tells you whether your production AI is still doing what it did on launch day. Without it, you ship a great demo and watch it silently degrade. This is the eval pattern we build into every engagement.
Inside
- 1. Start with 20 cases. Not 200.
- 2. Score what matters
- 3. Run weekly, gate releases
- 4. Grow the suite from production
- 5. LLM-as-judge, then humans on the close calls
Read them in this order
They're independent, but they were written in a sequence, and the sequence is the argument. Each one assumes the failure the previous one prevents.
- 1. Production agent design — start here if you're building anything that takes actions
- The central move is designing exception-first: enumerate every action the agent can take before writing a single prompt, and build that list with the person currently doing the work by hand. They will name actions you'd miss and veto ones you'd have included. Everything downstream — escalation rules, confidence thresholds, observability — hangs off that list.
- 2. RAG that actually grounds — read this the moment retrieval enters the picture
- Most RAG demos hallucinate within ten minutes of real use, and the cause is almost never the model. It's a retrieval step that was tuned on a curated document set and then pointed at a real knowledge base. Hybrid search, a reranker, citation-required prompting, and an explicit refusal path when nothing relevant comes back — that combination is what separates a demo from a system.
- 3. The eval suite you actually keep running — read this before you start tuning
- Twenty cases, replayed weekly, growing as production surfaces new failures. Twenty is deliberate: it's the largest number a team will still maintain six months in, and a suite nobody runs is worth nothing. Without one, prompt iteration is guesswork — you fix one behaviour, quietly break another, and hear about it from a customer.
These are the patterns we build into every engagement rather than things we sell separately. If you'd rather not implement them yourself, book a 20-min scoping call →
