An evaluation harness before you ship the chatbot

LLM features drift. A prompt tweak that looks sharper in a demo can quietly fail the ugly tickets your customers already send. We do not ship a chatbot on vibes. We put a small golden set in continuous integration before the first public release.

The harness is not a research lab. It is the same idea as unit tests: freeze a contract, fail the build when the contract breaks, and make it cheap to add the next edge case after an incident.

What we measure

Task success, refusal quality, latency, and cost per resolved ticket. Fluency scores that reward long, confident paragraphs do not make the dashboard. A polite wrong refund is still a wrong refund.

We sample from production, not from imagination. Forty tickets that actually happened beat four thousand synthetic chats. Each fixture stores the input, the policy, and the acceptable outputs — including the right to refuse.

How it sits in the pipeline

Prompt or tool changes open a pull request like any other. CI runs the golden set against the candidate model and the current production prompt. If task success drops more than the agreed threshold, the pull request cannot merge.

That sounds slow until the first silent regression. Then it is the cheapest meeting you never had to book.