Online evals and regression monitoring
Offline evals, the ones in the rest of this guide, run a dataset you built, before you ship. They answer “is this change safe to release?” But they can only test what you thought to put in the dataset. Production is where the inputs you did not anticipate show up: new phrasings, new topics, edge cases, slow drift as the world changes. A prompt that passed every offline case can still degrade on live traffic.
Online evals close that gap. Instead of a fixed dataset, they score your real production requests, continuously, as they happen.
What online evals catch
Two things offline evals cannot:
- Regressions after a change. You ship a new model or prompt. Offline it looked fine. Online evals watch the live score across the deploy, so if quality drops on real traffic you find out in minutes, not from a support ticket a week later.
- Drift and unknown failures. Real inputs shift over time, and users ask things no test case covered. Online evals surface the failures you never wrote a test for, because they score whatever actually comes in.
How it works
You cannot grade every production request; at volume that is too much cost and latency. So online scoring is built around a few pieces:
- A condition, which selects the traffic worth scoring (a feature, a model, a route, or just “everything from production”).
- A sampling rate, because a representative slice is enough to track quality without scoring all of it.
- An evaluator, run on each sampled request. Production has no gold answer attached, so these are reference-free evaluators: relevance, groundedness, tone, format validity, or an LLM judge on quality. (This is the “expected output is optional” point from earlier, made concrete: online, you almost never have one.)
- An automation, which runs the evaluator on the sampled traffic on an ongoing basis and tracks the score over time, so a drop can trigger an alert.
Regression monitoring
This is the online counterpart to gating a change in CI. Offline, you block a bad merge before it ships. Online, you watch the score across the release: if faithfulness or format compliance falls after a deploy, the trend line shows it and you can roll back. It catches the regressions your offline dataset simply did not have a case for, which, by definition, are the ones most likely to slip through.
Turn production failures into tests
The payoff compounds: every real failure online scoring catches is a case your offline dataset was missing. Pull those requests into your dataset, and the next release is tested against a failure that actually happened. Offline and online are not separate practices; online finds the gaps, offline hardens against them.
On the platform
In Respan this is an online evals automation: define a condition, set a sampling rate, pick an evaluator, and results stream in with the score tracked over time. See Online evals for the setup.
What this gives you
- Failures surface in production, from you, not from a user. You see the drop on a dashboard instead of in a churn report.
- Every deploy is watched. Regressions your offline set missed still get caught, live.
- Your offline dataset keeps improving, fed by the real failures online scoring finds.
Next steps
- When online scoring flags a systematic weakness, fix the prompt behind it, by hand with the single-call loop, or automatically with the optimization page next.