Online evals

Online evals run an evaluator against your live production traffic. You create an automation that points at a deployed evaluator pipeline, and Respan applies that evaluator to the spans flowing through your project, sampling, scoring, and writing the results back to each span in real time.

Unlike experiments, which score a fixed dataset offline, online evals score whatever your users actually send. They are best paired with the Gateway: once your calls route through Respan, every production span becomes something an automation can pick up and grade automatically.

How it works

An automation is made of three parts:

• Evaluator: a deployed evaluator pipeline that produces a score for a span.
• Filters: optional rules that narrow the automation to a subset of spans (by status, error type, customer, and more).
• Sampling rate: the percentage of matching spans to actually evaluate, so you control eval cost on high-volume traffic.

When a span matches the filters and is picked by the sampler, the automation runs the evaluator on it and attaches the resulting score. You can then watch scores trend over time and drill into individual results.

Prerequisites

To run your first online eval you need two things:

1. A deployed evaluator pipeline. Automations run a pipeline that has been deployed. If you don’t have one yet, follow Building your first LLM eval and deploy it.
2. Production traffic in Respan. The automation grades the spans your project receives. The simplest way to get production spans is to route your calls through the Gateway; any tracing integration that sends spans to Respan works too.

Walkthrough

1

Open Automations

In the sidebar, go to Evals → Automations. This page lists every automation in your project, with a Scores over time chart across the top and each automation’s Status and Version.

Click New automation in the top right.

2

Choose an evaluator pipeline

In the Create automation dialog, open the Evaluator dropdown and select a deployed evaluator pipeline. This is the evaluator that will score each sampled span.

3

Add filters (optional)

Under Filters, click + Filter to narrow the automation to a subset of spans. You can filter on span attributes such as status, status code, error type, customer, and thread. Only spans that match every filter are eligible for evaluation. Leave filters empty to consider all spans.

Filtering to error spans (for example, Status code ≥ 500) is a cheap way to grade only the traffic most likely to be a problem.

4

Set the sampling rate

Set Sampling rate, the percentage of matching spans to evaluate. On high-volume traffic, a low rate (for example, 2%) gives you a representative quality signal while keeping eval cost predictable. Raise it toward 100% for low-volume or high-stakes traffic where you want to grade nearly everything.

Click Create. The automation is deployed immediately and starts scoring new production spans as they arrive.

5

Read the results

Open the automation from the list to see its detail page. The Metrics tab shows a Scores over time chart and a table of every evaluated span; the JSON tab shows the raw configuration. Use the time-range selector (Today, Last 24 hours, Last 7 days, Last 30 days) to change the window.

The results table has one row per evaluated span:

Column	Description
Time	When the evaluation ran
Span ID	The production span that was scored
Input	The input sent to the model on that span
Output	The model’s output that was graded
Score	The score the evaluator produced
Eval cost	The cost of running the evaluator on that span

Click any row to open the span and its full score in the sidepanel.

Manage automations from the Spans page

You don’t have to leave your traffic to manage automations. Open Logs → Spans and click the automations icon in the toolbar to open the Automations panel, which shows how many automations are live versus paused and lets you act on each one without losing your place in the logs.

From the panel you can:

• Pause or resume an automation with the pause button. A paused automation stops scoring new spans until you resume it; a live/paused count at the top shows the current state of all your automations.
• Edit an automation with the pencil button. This opens the Configure automation dialog, which has the same evaluator, filters, and sampling rate controls as the create form; change any of them and click Update & redeploy to apply, or Pause the automation directly from this dialog.
• Create a new automation without leaving the Spans page.

Why pair online evals with the Gateway

Online evals grade whatever spans reach Respan, so the value scales with how much of your production traffic Respan can see. Routing your model calls through the Gateway is the easiest way to get complete, structured production spans: inputs, outputs, model, cost, latency, and status, with no extra instrumentation. Every one of those spans is then something an automation can sample and score, which is why the two features are designed to work together.

What’s next