EvaluationsDatasets

Datasets for model comparison

Score a raw model directly. No saved prompt in the loop.

Use this setup when you want to evaluate a model itself, or compare models against each other. Respan sends each row’s input straight to the model you configure, generates a fresh output, then scores it. Here the input is the request itself (the messages or payload), not template variables.

This walkthrough uses the shared “world capitals” dataset from the Datasets overview. Six rows, each an input question and its expected_output capital.

What each row needs

FieldRequiredNotes
inputYesThe model request, a self-contained question or message payload.
expected_outputOptionalAdd it when your grader compares against a known answer.
outputNoLeave empty. The run generates it.
Flow: the input request goes straight to the model, which generates an output; the output and expected_output feed the Correctness evaluator, producing a score.

Create the dataset

Click New Dataset, give it a name, and you land on an empty dataset with two ways to add rows: Insert by sampling (pull from your production traffic) or Insert from CSV (upload curated cases). Because the input here is a self-contained request, both paths are simple.

Empty dataset with Insert by sampling and Insert from CSV options
A new, empty dataset. Insert by sampling or Insert from CSV to add the first rows.

Any normal LLM-call span already has the shape a model dataset needs: the request becomes the row’s input. So you can sample your traced model calls directly, with no prompt template required.

1

Open the sampling modal

On the empty dataset, click Insert by sampling.

2

Set the time range

Pick the window the requests ran in. The range defaults to a narrow recent window (the last 15 minutes), so widen it if Estimated rows reads 0.

3

Filter to the requests you want to test

Click + Filter and set Status to a success value so you skip errored calls. Then narrow to the traffic you care about: use the Filter… search to find a field such as Model, Customer ID, Thread ID, or a Custom ID / metadata value. To evaluate one model’s real inputs, filter by that Model.

4

Set the sampling percentage and insert

Drag the Sampling percentage slider to control how many matching logs to pull, watch Estimated rows update, then click Insert. Each row’s input is the sampled request, alongside the original telemetry.

The Insert by sampling modal with the filter menu open
The Insert-by-sampling modal: a time range, the + Filter menu (Status, Model, Customer ID, and more via search), and the sampling-percentage slider with a live Estimated rows count.

With the rows in, run the dataset through an experiment with Task type = Model, where you pick the model to test and can add a second model to compare head-to-head on the same rows.