Retries and fallback

When a call fails, Respan retries it on the same model after a short delay. If it still fails, Respan tries your fallback models one at a time and returns the first response that succeeds. Retries are on by default, with up to 3 attempts starting 0.2 seconds apart.

Set them for your organization

1

Open the settings

Go to Gateway › Routing config and open the Retries and fallback tab.

2

Set retries

With Retry the same request on, pick how many times to try (1 time to 3 times) and the first delay (0.2 seconds to 2 seconds), then click Save.

3

Build a fallback chain

Turn on Retry with a different model, click New model, and pick a model. To add a model that isn’t listed, type its name and click Custom model: …. Use the arrows to reorder the chain and the red × to remove a model, then click Save.

The retry count includes the first call, so 3 times means at most three calls to each model. On /api/chat/completions, each wait doubles the one before, plus some jitter, up to 60 seconds. On /api/responses, every wait is the first delay. To call each model only once, turn off Retry the same request and confirm with Disable.

Only organization admins can change the fallback chain. It applies to every request that doesn’t send its own fallback_models.

Set them per request

fallback_models replaces your organization’s chain for that request:

import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.respan.ai/api/",
    api_key=os.environ["RESPAN_API_KEY"],
)
response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"fallback_models": ["claude-sonnet-4-5-20250929", "gemini-2.5-pro"]},
)

On /api/chat/completions, your organization’s retry settings always apply. On POST /api/responses, you can override them per request with retry_params, where num_retries is the total number of attempts, including the first, and retry_after is the wait in seconds before each retry. Send all three fields: one you leave out takes its default (retry_enabled true, num_retries 3, retry_after 0.2), not your organization’s setting.

{
  "model": "gpt-5.5",
  "input": "Summarize this incident.",
  "respan_params": {
    "retry_params": { "retry_enabled": true, "num_retries": 2, "retry_after": 0.5 }
  }
}

What happens when a call fails

1. Checks. Respan validates the request, your API key, and your limits. A request rejected here never reaches a model, so it isn’t retried and doesn’t fall back.
2. Retries. Respan calls the model up to the retry count, waiting between attempts as described above.
3. Fallback. When the attempts run out, Respan moves to the next fallback model and retries that one the same way. It skips fallback models that can’t take the request, such as ones with too small a context window or without image input.
4. Result. Respan returns the first success. If every model fails, it returns the last error.

Some errors end the request at once, with no retry and no fallback, because repeating the call can’t fix them:

• Input larger than the model’s context window.
• An invalid provider key of your own (401).
• A model that timed out while generating.

Other provider errors, such as rate limits, server errors, and dropped connections, are retried.

Each fallback model uses your provider key or Respan credits, chosen the same way as for the first model. Requests to span-router don’t use your organization’s chain. In Logs, a request that needed a retry or a fallback is marked as a warning, and its log shows which model served it.

Troubleshooting

• The request failed without trying a fallback. A check rejected it, or it hit one of the errors above. The log’s error message says which.
• Requests are slow while a provider is struggling. Each fallback model gets its own attempts, so the worst case grows with both. Lower the retry count or shorten the fallback chain.
• You want to spread traffic rather than recover from failures. Use load balancing.