End-to-end: the recommended Respan workflow

This guide is the recommended path for adopting Respan in a real product. It follows one small example, a customer support chatbot, through four stages that feed into each other. It is intentionally light on code: each stage links to a focused quickstart when you want the full implementation.

$
export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

1. Observe: see what your app is doing

Start here. Before optimizing anything, you need visibility into what your app actually does in production: which steps run, what each LLM call costs, where latency comes from, and what users receive.

Install the SDK and initialize once at app start. Every supported LLM call is auto-traced from then on.

$
pip install respan-ai

1
from respan import Respan
2
3
Respan()
4
# All supported LLM calls are now auto-traced

For a multi-step app, wrap the logical workflow so the trace shows structure instead of isolated calls:

1
from respan import Respan, workflow, task
2
3
Respan()
4
5
@workflow(name="support_chatbot")
6
def support_chatbot(question: str):
7
    return answer(question)
8
9
@task(name="answer")
10
def answer(question: str):
11
    return client.chat.completions.create(...)

Full setup, frameworks, and options: Tracing quickstart.

2. Route: one gateway for every model

Once you can see your calls, route them through the gateway. Point any OpenAI-compatible SDK at the Respan endpoint and you get one key for 1000+ models, automatic logging, fallbacks, and caching, without changing how you call the model.

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="https://api.respan.ai/api/",
5
    api_key="YOUR_RESPAN_API_KEY",
6
)

Switch models by changing the model string (gpt-4o, claude-sonnet-4-20250514, gemini-2.0-flash). Add fallbacks, retries, and caching per request without touching the rest of your code.

Model switching, fallbacks, and caching: Gateway quickstart.

3. Evaluate: measure quality, do not guess

With traffic flowing and traced, start measuring output quality instead of eyeballing it. The offline workflow has three parts:

1. Dataset: the test inputs. Build it by sampling real production spans (recommended) or importing a CSV.
2. Evaluator: how outputs are scored. Use an LLM judge, a code check, or human review.
3. Experiment: runs a dataset through different prompt versions or models and compares the scores.

For the chatbot, sample real support conversations into a dataset, define a “Support Quality” LLM evaluator (does it answer the question, is the tone right, is it accurate), and run an experiment before shipping any prompt change.

Walkthrough: Evals quickstart. Concepts: datasets, evaluators, experiments.

4. Automate: score live traffic continuously

Offline tests do not catch everything. Set up an online evaluation so production traffic is scored automatically and you are alerted when quality drops.

Create an automation that filters to the feature you care about, attaches your evaluator, and samples a fraction of traffic. For the chatbot, scoring 20% of live conversations surfaces regressions within hours instead of after a user complains.

Set it up: Online evaluation.

5. Iterate: close the loop

This is where the stages connect into a flywheel:

• Low-scoring production traces become new rows in your dataset.
• You improve the prompt in the platform and test it with an experiment.
• You deploy the new version without a code change, and traffic picks it up immediately.
• New traffic flows back into tracing, and the cycle repeats.

The longer Respan runs, the better your dataset, your evaluations, and your prompts get, because each builds on real production data.

Recommended practices

The short version of “the proper way to use Respan”:

• Separate API keys for test and production. Cleaner data, better security.
• Use structured traces (@workflow / @task) for any multi-step app, not flat spans.
• Tag spans with customer_identifier and thread_identifier for per-user analytics and conversation grouping.
• Seed eval datasets from real production traces, not hand-written cases.
• Gate every prompt or model change behind an experiment before it reaches users.
• Manage prompts in the platform, not hardcoded, so you can iterate without redeploying.
• Add fallbacks and retries on critical gateway calls for reliability.
• For latency-critical paths, trace with the SDK instead of routing through the gateway.
• Disable logging (disable_log) on requests carrying sensitive data.
• In serverless, call flush() at the end of each invocation so spans are exported.

Next steps