Treat prompt changes like failing tests

If a prompt can change without a test, it will. We store fixtures next to the feature and fail the build when the model drifts past a threshold. Yesterday's correct answer is allowed to become today's incident only if someone chose that risk in a review.

This is not research excellence. It is the same hygiene we already demand for tax calculation. Language models are not exempt because they sound confident.

Small sets, sharp cases

Forty ugly tickets beat four thousand generic chats. We sample from production: refunds, threats, mixed languages, incomplete orders. Each case has an acceptable band, including 'refuse and escalate'.

When a new failure appears in production, it becomes a fixture before we 'just tweak the prompt'. Otherwise we are playing whack-a-mole with a thesaurus.

Same pipeline as unit tests

The eval job is a step in CI, with a number on the pull request. Authors see the failing case. Reviewers see the delta. Nobody has to open a notebook to understand what changed.

Treat prompts like code and they start behaving like code: reviewable, revertible, and slightly less magical.