Your deployed LLM app degrades quietly: inputs drift, quality slips, and users rarely file a ticket. Break a simulated production system on purpose, then tune a real changepoint detector to catch it, and feel the tradeoff between catching it fast and crying wolf.
Four signals that tell you before your users do
Judge score
Mean quality score from an automated LLM judge grading a sample of each hour's responses. The closest thing to ground truth you get in production.
Refusal rate
Share of requests the model declines or deflects. Rises when incoming traffic drifts away from what your prompts were tuned for.
p95 latency
95th percentile response time per hour. Longer inputs and longer outputs both push it up, so it moves when traffic drifts.
Tokens per request
Mean total tokens per request. Your spend signal: drifting inputs tend to be longer and provoke longer answers.
Offline evals grade a frozen model on a frozen dataset. Production is neither: prompts change, providers ship silent model updates, and the traffic itself drifts. Monitoring is the eval that never stops running. Everything below is simulated traffic from a fictional app, generated by a seeded random model you control.
Break production on purpose
You control the incident: pick when it starts and how hard it hits. Quality drop is a silent regression in answer quality, like a bad prompt deploy or a provider-side model change. Input drift is your traffic changing shape, which shows up as more refusals, slower responses, and fatter token bills.
Reseeds the simulated noise; the regression settings stay put. The regression ramps to full strength over 8 hours, like a gradual rollout or slow upstream drift.
No regression injected yet: raise the quality drop or the drift severity to break something.
Judge score
now 0.854
baseline mean 0.870 (hours 0 to 29, shaded)
Refusal rate
now 4.5%
baseline mean 4.2% (hours 0 to 29, shaded)
p95 latency
now 1905 ms
baseline mean 1828 ms (hours 0 to 29, shaded)
Tokens per request
now 687
baseline mean 625 (hours 0 to 29, shaded)
Simulated traffic: each hour draws from a normal distribution around the metric baselines (judge 0.86, refusal 4%, p95 1800 ms, 620 tokens) using a seeded PRNG. Quality drop pulls the judge score down; drift pushes refusals, latency, and token spend up. The pink dashed line marks the onset you chose.
The detector: CUSUM, a real changepoint method
Eyeballing dashboards does not scale, and a naive "alert when the metric crosses X" rule ignores how noisy each hour is. CUSUM (cumulative sum control chart) accumulates small standardized deviations from baseline, so a persistent shift builds up and fires even when no single hour looks scary. Pick a metric to monitor, then tune the threshold h. Lower it until it catches your regression, then keep lowering it and watch false alarms appear before the onset.
Mean quality score from an automated LLM judge grading a sample of each hour's responses. The closest thing to ground truth you get in production.
One-sided tabular CUSUM (Judge score, watching for drops): z_t = -(x_t - mu0) / sigma0 deviation from baseline, in sigmas S_t = max(0, S_t-1 + z_t - k) accumulate anything beyond the slack k alarm when S_t > h, then S resets to 0 baseline from hours 0 to 29: mu0 = 0.8697, sigma0 = 0.0285
Detection delay
n/a
inject a regression first
False alarms
1
at hour 109, with no real shift on this metric to explain them
Total alarms
1
each alarm resets the statistic, like an on-call ack
The honest tradeoff: detection delay vs false alarms
Inject a regression in the lab above to trace this curve. With nothing to detect, detection delay has no meaning.
Computed by Monte Carlo: 40 paired simulations per threshold with the same noise seeds, once with your regression and once without it. False-alarm probability counts in-control runs with at least one alarm; delay averages over detected regression runs. 9 of 24 thresholds never detected the regression and are not plotted.