Advanced

For the complete list of all request parameters, see Span Attributes and API reference.

1
{
2
  "mcpServers": {
3
    "respan-docs": {
4
      "url": "https://docs.respan.ai/mcp"
5
    }
6
  }
7
}

Load balancing

Load balancing allows you to balance the request load across different deployments. You can specify weights for each deployment based on their rate limit and your preference.

See all supported params here.

Load balancing between models

1

Go to the Load balancing page

Go to the Load balancing page and click on Create new load balancer

2

Add models

Click Add model to add models and specify the weight for each model and add your own credentials.

3

Copy group ID to your codebase

After you have added the models, copy the group ID (the blue text) to your codebase and use it in your requests.

The model parameter will overwrite the load_balance_group!

1
{
2
    "messages": [
3
        {
4
            "role": "user",
5
            "content": "Hi, how are you?"
6
        }
7
    ],
8
    "load_balance_group": {
9
        "group_id":"THE_GROUP_ID"
10
    }
11
}

4

Add load balancing group in code (Optional)

You can also add the load balancing group in your codebase directly. The models field will overwrite the load_balance_group you specified in the UI.

Example code

1
{
2
  "load_balance_group": {
3
      "group_id":"THE_GROUP_ID",
4
      "models": [
5
        {
6
          "model": "azure/gpt-35-turbo",
7
          "weight": 1
8
        },
9
        {
10
          "model": "azure/gpt-4",
11
          "credentials": {
12
              "api_base": "Your own Azure api_base",
13
              "api_version": "Your own Azure api_version",
14
              "api_key": "Your own Azure api_key"
15
          },
16
          "weight": 1
17
        }
18
      ]
19
  }
20
}

Load balancing between deployments

A deployment basically means a credential. If you add an OpenAI API key, you have one deployment. If you add 2 OpenAI API keys, you have 2 deployments.

You can go to the platform and add multiple deployments for the same provider, specifying load balancing weights for each deployment.

You can also load balance between deployments in your codebase using the customer_credentials field:

1
{
2
  "customer_credentials": [
3
    {
4
        "credentials": {
5
            "openai": {
6
                "api_key": "YOUR_OPENAI_API_KEY",
7
            }
8
        },
9
        "weight": 1.0
10
    },
11
    {
12
        "credentials": {
13
            "openai": {
14
                "api_key": "YOUR_OPENAI_API_KEY",
15
            }
16
        },
17
        "weight": 1.0
18
    },
19
  ],
20
}

1
{
2
  "customer_credentials": [
3
    {
4
        "credentials": {
5
            "openai": {
6
                "api_key": "YOUR_OPENAI_API_KEY",
7
            }
8
        },
9
        "weight": 1.0,
10
        "available_models": ["gpt-3.5-turbo"],
11
        "exclude_models": ["gpt-4"]
12
    },
13
    {
14
        "credentials": {
15
            "openai": {
16
                "api_key": "YOUR_OPENAI_API_KEY",
17
            }
18
        },
19
        "weight": 1.0,
20
    },
21
  ],
22
}

Retries

When an LLM call fails, the system detects the error and retries the request to prevent failovers.

Via UI

Via code

Go to the Retries page and enable retries and set the number of retries and the initial retry time.

1
model # User requested model
2
model_params = respan_models_data[model]
3
# Exponential backoff retry logic
4
for i in range(0, fallback_retries):
5
    try:
6
        response = respan_response_with_load_balance(model)
7
        return response
8
        break
9
    except RateLimitError:
10
        if model_params["fallback_models"]:
11
            for fallback_model in model_params["fallback_models"]:
12
                response = respan_response_with_load_balance(fallback_model)
13
                return response
14
        sleep(2 ** i)
15
    except Exception as e:
16
        raise e

Fallback models

Respan catches any errors occurring in a request and falls back to the list of models you specified in the fallback_models field. This is useful to avoid downtime and ensure availability.

See all Respan params here.

Via UI

OpenAI Python SDK

OpenAI TypeScript SDK

Standard API

Go to Settings -> Fallback -> Click on Add fallback models -> Select the models you want to add as fallbacks.

You can drag and drop the models to reorder them. The order of the models in the list is the order in which they will be tried.

Rate limit

You can set rate limits for each model and API key. See our rate limit configuration guide for detailed instructions.

Caches

Caches save and reuse exact LLM requests. Enable caches to reduce LLM costs and improve response times.

• Reduce latency: Serve stored responses instantly, eliminating repeated API calls.
• Save costs: Minimize expenses by reusing cached responses.

Turn on caches by setting cache_enabled to true. We will cache the whole conversation, including the system message, user message and the response.

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="https://api.respan.ai/api/",
5
    api_key="YOUR_RESPAN_API_KEY",
6
)
7
8
response = client.chat.completions.create(
9
    model="gpt-4o-mini",
10
    messages=[
11
        {"role": "user", "content": "Tell me a long story"}
12
    ],
13
    extra_body={
14
        "cache_enabled": True,
15
        "cache_ttl": 600,
16
        "cache_options": {
17
            "cache_by_customer": True
18
        }
19
    }
20
)

Cache parameters

1
{
2
    "cache_enabled": true
3
}

1
{
2
    "cache_ttl": 3600
3
}

1
{
2
    "cache_options": {
3
        "cache_by_customer": true,
4
        "is_cached_by_model": true,
5
        "omit_log": false
6
    }
7
}

View caches

You can view the caches on the Logs page. The model tag will be respan/cache. You can also filter the logs by the Cache hit field.

Omit logs when cache hit

Set the omit_logs parameter to true or go to Caches in Settings. This won’t generate a new LLM log when the cache is hit.

Prompt caching

Prompt caching stores the model’s intermediate computation state. This allows the model to generate diverse responses while still saving computational costs, as it doesn’t need to reprocess the entire prompt from scratch.

1
import anthropic
2
3
client = anthropic.Anthropic(
4
    base_url="https://api.respan.ai/api/anthropic/",
5
    api_key="Your_Respan_API_Key",
6
)
7
8
message = client.messages.create(
9
    model="claude-3-opus-20240229",
10
    system=[
11
      {
12
        "type": "text",
13
        "text": "You are an AI assistant tasked with analyzing literary works. Your goal is to provide insightful commentary on themes, characters, and writing style.\n",
14
      },
15
      {
16
        "type": "text",
17
        "text": "<the entire contents of 'Pride and Prejudice'>",
18
        "cache_control": {"type": "ephemeral"}
19
      }
20
    ],
21
    messages=[{"role": "user", "content": "Analyze the major themes in 'Pride and Prejudice'."}]
22
)
23
24
print(message.content)

Function calling

Function calling allows you to call a function from a model and get the result.

1
from openai import OpenAI
2
client = OpenAI(
3
    base_url="https://api.respan.ai/api/",
4
    api_key="YOUR_RESPAN_API_KEY",
5
)
6
7
tools = [
8
  {
9
    "type": "function",
10
    "function": {
11
      "name": "get_current_weather",
12
      "description": "Get the current weather in a given location",
13
      "parameters": {
14
        "type": "object",
15
        "properties": {
16
          "location": {
17
            "type": "string",
18
            "description": "The city and state, e.g. San Francisco, CA",
19
          },
20
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
21
        },
22
        "required": ["location"],
23
      },
24
    }
25
  }
26
]
27
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]
28
completion = client.chat.completions.create(
29
  model="gpt-4o",
30
  messages=messages,
31
  tools=tools,
32
  tool_choice="auto"
33
)
34
print(completion)

Enable thinking

Thinking mode allows supported models to show their reasoning process before providing the final answer.

1
payload = {
2
    "model": "claude-sonnet-4-20250514",
3
    "max_tokens": 16000,
4
    "thinking": {
5
        "type": "enabled",
6
        "budget_tokens": 10000
7
    },
8
    "messages": [
9
        {
10
            "role": "user",
11
            "content": "Are there an infinite number of prime numbers such that n mod 4 == 3?"
12
        }
13
    ]
14
}

Parameters:

• type: Set to "enabled" to activate thinking mode
• budget_tokens: Maximum number of tokens allocated for the thinking process (optional)

Choose models that support thinking like gpt-5, claude-sonnet-4-20250514. See the Log Thinking documentation for details on the response structure.

Upload PDF

To help models understand PDF content, we put into the model’s context both the extracted text and an image of each page.

1
import os
2
import base64
3
import requests
4
from openai import OpenAI
5
6
openai_client = OpenAI()
7
pdf_url = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
8
response = requests.get(pdf_url)
9
file_data = response.content
10
file = openai_client.files.create(file=file_data, purpose="user_data")
11
12
client = OpenAI(
13
    base_url="https://api.respan.ai/api",
14
    api_key=os.getenv("RESPAN_API_KEY_TEST"),
15
)
16
17
model = "gpt-4.1"
18
19
file_content = [
20
    {"type": "text", "text": "What's this file about?"},
21
    {
22
        "type": "file",
23
        "file": {
24
            "file_id": file.id,
25
        },
26
    }
27
]
28
29
response = client.chat.completions.create(
30
    model=model,
31
    messages=[
32
        {
33
            "role": "user",
34
            "content": file_content,
35
        }
36
    ],
37
)

Upload image

You can upload images to the LLM request. We support base64 or url format for image variables.

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="https://api.respan.ai/api/",
5
    api_key="YOUR_RESPAN_API_KEY",
6
)
7
8
response = client.chat.completions.create(
9
    model="gpt-4o-mini",
10
    messages=[{"role":"user", "content":"Tell me a long story"}],
11
    extra_body={
12
        "variables": {
13
            "image_variable": {"_type": "image_url", "value": "url_string"}
14
        }
15
    },
16
)

Disable logging

At Respan, data privacy is our priority. Set the disable_log parameter to true to disable logging for sensitive data.

The following fields will not be logged: full_request, full_response, messages, prompt_messages, completion_message, tools.

See all supported parameters here.

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="https://api.respan.ai/api/",
5
    api_key="YOUR_RESPAN_API_KEY",
6
)
7
8
response = client.chat.completions.create(
9
    model="gpt-4o-mini",
10
    messages=[
11
        {"role": "user", "content": "Tell me a long story"}
12
    ],
13
    extra_body={
14
        "disable_log": True
15
    }
16
)

Streaming

When streaming is enabled, Respan forwards the streaming response to your end token by token. This is useful when you want to process the output as soon as it is available, rather than waiting for the entire response.

See all params here.

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="https://api.respan.ai/api/",
5
    api_key="YOUR_RESPAN_API_KEY",
6
)
7
8
response = client.chat.completions.create(
9
    model="gpt-4o-mini",
10
    messages=[{"role": "user", "content": "Hello"}],
11
    stream=True,
12
)
13
14
for chunk in response:
15
    print(chunk)