Enterprise AI: The Silent Prompt Rollback Nobody Owned
A prompt cleanup quietly dropped groundedness by nine points across the fleet. The eval dashboard that caught it and the ownership rule that stops it.
Every prompt in a shared prompt library is a production dependency, and dependencies get changed by people who do not know what depends on them. We lived that sentence on a Thursday: a colleague merged a tidy-up to the shared support-answer system prompt - shorter, friendlier, one fewer guardrail sentence - and by Friday afternoon our groundedness score had dropped 9.4 points across six product surfaces. Nobody was paged, because no single alert fired. The eval dashboard below is what finally told the story, four days later, in a weekly review.
The regression board: v214 rolls out at 09:40, groundedness slides for two days, and the rollback restores v211 in minutes. The embarrassing column is the last one - every failing suite had its owner listed as unassigned.
What actually happened
The timeline, reconstructed from the eval store and git history:
- Wednesday 09:40 - a well-reviewed pull request merges: “shorten support-answer system prompt, remove redundant phrasing”. It touches one template file. Reviewers are two senior engineers; the diff reads like an obvious improvement. Among the removals: one sentence that instructed the model to refuse when the retrieved context does not contain the answer.
- Wednesday 14:00 - the nightly eval suite runs against
v214in staging. Groundedness drops on three suites, but staging thresholds were set wide years ago (nobody wants flaky staging), so the run is green. - Thursday 10:00 - the weekly promotion job ships
v214to the production canary, then to full traffic by 14:00. No single surface drops enough to trip a per-service alert; each service sees a two-to-three-point slide, inside its individual noise band. - Thursday to Friday - support tickets mention the assistant “guessing” refund rules it should have declined to answer. Each ticket lands in a different team’s queue, so no queue looks unusual.
- Friday 16:20 - the weekly eval review opens the dashboard above. The fleet-wide groundedness view - the one panel nobody watches daily - shows a single clean step down at exactly 09:40 Wednesday, version
v214.
Total damage: four days of degraded answer quality on six surfaces, a measurable uptick in “unhelpful” thumbs-down feedback, and one very uncomfortable prompt-governance review.
Why the existing guardrails did not catch it
We were not naive going in. Prompt changes already went through code review, ran against nightly eval suites, and shipped through a canary. All three worked exactly as designed - and none of them fired:
- Code review judged prose, not behavior. A prompt diff reads like an English edit. Nobody diffs the answers the prompt produces against a golden set before merge.
- Staging thresholds were too wide. The eval suites detected the regression; the gate ignored it. A gate that never blocks is a report, not a gate.
- Alerts were per-service and absolute. The fleet-wide relative view - “groundedness down more than five points week-over-week across all surfaces” - existed only as a dashboard panel that a human opened on Fridays.
The gap was not any of those controls. The gap was that nobody owned a prompt version the way someone owns a deployment. A bad deploy has a name next to it and a rollback button; a bad prompt version had neither.
The fix: a fleet-wide regression gate and a named owner
The control we added has two halves, and both are deliberately boring.
1. A fleet-wide eval gate. Every prompt version promoted to production must beat the previous version’s groundedness score on the union of all suites, or a human signs the waiver. The gate runs in the promotion pipeline, not in staging, and its threshold is relative, not absolute:
1
2
3
4
5
6
7
8
9
10
11
12
13
# .prompt-gates/support-answer.yaml
prompt: support-answer-system
promotion_gate:
metric: groundedness
baseline: previous_prod_version
max_regression_points: 1.0 # fleet-wide, not per-suite
window: nightly_eval_run
waiver:
requires: prompt_owner_approval
ticket: required
on_breach:
action: auto_rollback
page: prompt_owner
2. Every prompt gets an owner, and the owner is a person, not a team alias. The eval store refuses to register a prompt version whose metadata has no owner field, and the nightly job pages that person directly when the regression gate trips. That one metadata rule is what turned the dashboard’s unassigned ⚠ column into names.
The first Friday after this shipped, the gate caught a real regression in 40 minutes - the on-page owner rolled back from their phone before the change reached half the canary. Same failure mode, fourteen-minute blast radius instead of four days.
What we would tell past us
- Treat prompt diffs like schema diffs. A removed sentence is a removed constraint. If you would not drop a
NOT NULLwithout a migration review, do not drop a refusal instruction without an eval review. - Fleet-wide relative metrics catch what per-service absolute alerts miss. A slow nine-point slide across six surfaces is invisible to every individual threshold and glaringly obvious in one aggregate chart.
- Ownership is a control, not an org chart. The single most effective line in this whole fix was the one that makes the eval store reject an ownerless prompt version. Dashboards inform people; gates and owners change what ships.
A silent rollback is a rollback that happens days late because nobody was watching the one number that mattered. Make the number a gate, give the version a name, and the silence ends.