EvaluationsDatasets

Datasets for production data

Score answers you already have. No generation happens.

Use this setup to grade outputs that already exist: answers captured from production, a previous run, or a hand-built set of good and bad examples. Respan scores the output already stored on each row, so nothing is generated.

This walkthrough uses the shared “world capitals” dataset from the Datasets overview, here with a stored output on every row.

What each row needs

FieldRequiredNotes
outputYesThe answer to grade. This is what the evaluator scores.
expected_outputUsuallyThe reference answer to compare against.
inputOptionalInclude it to show context alongside the results.
Flow: a stored output and the expected_output feed the Correctness evaluator to produce a score; input is optional context. Nothing is generated.

Create the dataset

Click New Dataset, give it a name, and you land on an empty dataset with two ways to add rows: Insert by sampling (pull answers from your production traffic) or Insert from CSV (upload a curated set). The required field here is output, so both paths center on getting a stored answer onto each row.

Empty dataset with Insert by sampling and Insert from CSV options
A new, empty dataset. Insert by sampling or Insert from CSV to add the first rows.

Sampling is the natural fit for this setup: your production spans already carry the response the model gave, and it lands directly in each row’s output. Nothing is generated, so you are grading real answers exactly as they shipped.

1

Open the sampling modal

On the empty dataset, click Insert by sampling.

2

Set the time range

Pick the window the answers you want to grade were produced in. The range defaults to a narrow recent window (the last 15 minutes), so widen it if Estimated rows reads 0.

3

Filter to the answers you want to grade

Click + Filter and set Status to a success value so you grade delivered answers rather than errored calls. Then narrow to the traffic you want to review: use the Filter… search to find a field such as Model, Customer ID, Thread ID, or a Custom ID / metadata value.

4

Set the sampling percentage and insert

Drag the Sampling percentage slider to control how many matching logs to pull, watch Estimated rows update, then click Insert. Each row arrives with its input, the stored output, and the original telemetry.

The Insert by sampling modal with the filter menu open
The Insert-by-sampling modal: a time range, the + Filter menu (Status, Model, Customer ID, and more via search), and the sampling-percentage slider with a live Estimated rows count.

With the rows in, run the dataset through an experiment with Task type = Dataset outputs. Nothing is generated: Respan scores each stored output against its expected_output.