Ragas (tracing) | Respan Docs

Ragas is a framework for evaluating LLM and RAG applications. respan-instrumentation-ragas traces Ragas 0.4 metrics, dataset evaluations, and experiments as canonical Respan task spans, preserving the active workflow hierarchy around them.

The adapter covers both the preferred Ragas 0.4 metric collections API and the legacy evaluation surfaces. Nested internal calls are collapsed so one logical metric operation does not produce duplicate sync and async spans.

Set up Respan

Create an account at platform.respan.ai and grab an API key.

Run npx @respan/cli setup to set up with your coding agent.

Example projects

Setup

Install packages

pip install respan-ai respan-instrumentation-ragas

Set the Respan API key

export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

Metric-specific provider credentials are still required for metrics that call an LLM or embedding model. Deterministic metrics such as exact match do not require a provider key.

Initialize and score a sample

from ragas.metrics.collections import ExactMatch, StringPresence
from respan import Respan
from respan_instrumentation_ragas import RagasInstrumentor
respan = Respan(
    app_name="ragas-answer-evaluation",
    instrumentations=[RagasInstrumentor()],
)
try:
    exact_match = ExactMatch().score(
        reference="Paris",
        response="Paris",
    )
    string_presence = StringPresence().score(
        reference="Paris",
        response="Paris, France",
    )
    print(exact_match.value, string_presence.value)
finally:
    respan.shutdown()

View your trace

Open the Traces page to inspect one task span per metric with its inputs, score, timing, and status.

Supported operations

Ragas surface	Traced operation
Collection metrics	`score`, `ascore`, `batch_score`, and `abatch_score`
Metric base classes	Single-turn and multi-turn sync and async scoring
Dataset evaluation	`evaluate` and `aevaluate`
Experiments	Experiment `arun` plus each experiment row

All of these operations emit canonical Respan task spans. Successful spans include status_code=200. Failures are re-raised unchanged after recording OpenTelemetry error status, an exception event, backend-visible status_code, and error.message.

Async metrics

from ragas.metrics.collections import ExactMatch
async def score_answer() -> float:
    result = await ExactMatch().ascore(
        reference="Paris",
        response="Paris",
    )
    return result.value

Batch metric methods are traced as one logical operation, without an extra span for each internal sync-to-async delegation.

Dataset evaluation

The legacy evaluate and aevaluate entry points remain supported:

from ragas import EvaluationDataset, evaluate
from ragas.metrics import ExactMatch, StringPresence
dataset = EvaluationDataset.from_list(
    [
        {"response": "Paris", "reference": "Paris"},
        {"response": "Lyon", "reference": "Paris"},
    ]
)
result = evaluate(
    dataset,
    metrics=[ExactMatch(), StringPresence()],
    experiment_name="capital_answers",
    show_progress=False,
)
print(result.to_pandas())

Experiments and rows

Ragas experiments produce a task for the experiment run and a child task for every evaluated row:

from ragas import Dataset, experiment
from ragas.metrics.collections import ExactMatch
@experiment()
async def score_row(row: dict) -> dict:
    score = await ExactMatch().ascore(
        reference=row["reference"],
        response=row["response"],
    )
    return {**row, "exact_match": score.value}
async def run_experiment(dataset: Dataset) -> object:
    return await score_row.arun(dataset, name="answer_quality")

Configuration

Parameter	Type	Default	Description
`capture_content`	`bool`	`True`	Capture evaluation inputs, references, responses, scores, and outputs.
`instrumentations`	`list`	`[]`	Pass `[RagasInstrumentor()]` to activate Ragas tracing.
`api_key`	`str \| None`	`None`	Falls back to `RESPAN_API_KEY`.
`base_url`	`str \| None`	`None`	Falls back to `RESPAN_BASE_URL`.
`customer_identifier`	`str \| None`	`None`	Default customer identifier for all emitted spans.
`metadata`	`dict \| None`	`None`	Default metadata attached to all emitted spans.

Disable content capture

from respan import Respan
from respan_instrumentation_ragas import RagasInstrumentor
respan = Respan(
    instrumentations=[RagasInstrumentor(capture_content=False)],
)

With capture disabled, evaluation inputs and outputs are omitted. Operation names, duration, lifecycle, status, and exception type remain visible.

Group evaluations in a workflow

Use a Respan workflow or propagated attributes to group several Ragas operations into one trace:

from respan import Respan, workflow
@workflow(name="ragas_regression_suite")
def run_regression_suite() -> dict[str, float]:
    score = ExactMatch().score(reference="Paris", response="Paris")
    return {"exact_match": score.value}
with Respan.propagate_attributes(
    trace_group_identifier="ragas_regression_suite",
    metadata={"dataset": "capital-cities-v2"},
):
    results = run_regression_suite()