Support copilot with measured quality
A retrieval-grounded assistant for a support team, shipped with an evaluation harness that gates every prompt and model change.
Industry
B2B SaaS, US
Year
2026
Stack
- Python
- LangGraph
- pgvector
- Braintrust
- Next.js
- Modal
01Problem
Agents spent most of each ticket hunting through docs and past tickets. Earlier LLM pilots produced confident wrong answers and were abandoned.
02Approach
- 01
Built a hybrid retrieval layer over docs, release notes, and resolved tickets with freshness weighting.
- 02
Drafted responses with cited sources and confidence signals surfaced to the agent.
- 03
Created a 600-item golden set and rubric; every change runs offline evals before deploy.
- 04
Added model routing to hold cost per ticket within budget.
03Outcome
- Draft acceptance rate tracked weekly against the golden set instead of anecdotes.
- Hallucinated-citation rate driven to near zero through retrieval gating.
- Agents review and send instead of research and write.
Similar problem?
Let's talk about yours.
Next chapter
Code reasoning evaluation program