Respan caching

Caches save and reuse exact LLM requests. Enable caches to reduce LLM costs and improve response times.

• Reduce latency: Serve stored responses instantly, eliminating repeated API calls.
• Save costs: Minimize expenses by reusing cached responses.

Turn on caches by setting cache_enabled to true. We will cache the whole conversation, including the system message, user message and the response.

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="https://api.respan.ai/api/",
5
    api_key="YOUR_RESPAN_API_KEY",
6
)
7
8
response = client.chat.completions.create(
9
    model="gpt-5.4",
10
    messages=[
11
        {"role": "user", "content": "Tell me a long story"}
12
    ],
13
    extra_body={
14
        "cache_enabled": True,
15
        "cache_ttl": 600,
16
        "cache_options": {
17
            "cache_by_customer": True
18
        }
19
    }
20
)

Cache parameters

1
{
2
    "cache_enabled": true
3
}

1
{
2
    "cache_ttl": 3600
3
}

1
{
2
    "cache_options": {
3
        "cache_by_customer": true,
4
        "is_cached_by_model": true,
5
        "omit_log": false
6
    }
7
}

View caches

Open Caches in the platform to configure response caching. To inspect cache usage, use the Logs page: cached responses use the respan/cache model tag, and logs can be filtered by the Cache hit field.

Omit logs when cache hit

Set cache_options.omit_log to true when you do not want a new LLM log created for a cache hit.

1
{
2
  "cache_enabled": true,
3
  "cache_options": {
4
    "omit_log": true
5
  }
6
}

Understand cache identity and expiry

Response caching matches the request information used to create the entry. A changed message or cache option can produce a different entry. Use:

• cache_ttl to control how long the cached response remains eligible for reuse.
• cache_by_customer when identical requests from different customer_identifier values must not share an entry.
• is_cached_by_model when switching the requested model should produce a separate entry.

Choose a TTL that matches how long a response can remain valid for your application. Do not cache data that must always be recomputed or content whose storage would violate your privacy requirements.

Troubleshooting

A repeated request misses the cache

• Confirm that cache_enabled is true on both requests.
• Compare the full messages and relevant cache options; even a small request change can prevent a hit.
• Confirm the entry has not passed its cache_ttl.
• If cache_by_customer or is_cached_by_model is enabled, confirm the customer identifier or model is unchanged.

A response comes from an unexpected model

Enable is_cached_by_model when the model must be part of cache identity. Without it, the same prompt can reuse a response created for another requested model.

No cache-hit log appears

Check whether cache_options.omit_log is enabled before assuming that the cache missed.