Skip to navigation
IntegrationIntegrationsAgent Frameworks

Haystack (gateway)

Route Haystack’s OpenAI-compatible LLM calls through the Respan gateway to use 1000+ models from different providers. No separate OpenAI provider key is required.

Setup

1

Install packages

pip install haystack-ai
2

Set environment variables

export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"
3

Point Haystack to the Respan gateway

import os
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
os.environ["OPENAI_API_KEY"] = os.environ["RESPAN_API_KEY"]
os.environ["OPENAI_BASE_URL"] = "https://api.respan.ai/api"
pipeline = Pipeline()
pipeline.add_component(
    "prompt_builder",
    ChatPromptBuilder(
        template=[ChatMessage.from_user("Answer the following question: {{question}}")],
        required_variables=["question"],
    ),
)
pipeline.add_component("generator", OpenAIChatGenerator(model="gpt-5-mini"))
pipeline.connect("prompt_builder", "generator")
result = pipeline.run(
    {"prompt_builder": {"question": "What is the capital of France?"}}
)
print(result["generator"]["replies"][0].text)

Haystack’s OpenAI-compatible generators read OpenAI-shaped environment variables. The example maps those values from RESPAN_API_KEY; users do not need an OpenAI provider key.

Switch models

Change the model parameter on OpenAIChatGenerator to use another OpenAI model through the same gateway-backed endpoint.

pipeline.add_component(
    "generator",
    OpenAIChatGenerator(model="gpt-5.5"),
)

OpenAIChatGenerator is Haystack’s OpenAI-compatible chat generator. Haystack 3 removed the older OpenAIGenerator, so use the chat generator with ChatPromptBuilder and ChatMessage as shown above. This page avoids showing Claude or Gemini inside that OpenAI-named component; use the Respan API or OpenAI SDK gateway pages for provider-neutral Claude and Gemini examples.

See the full model list.

Prompt management

Use Respan prompt management to store the prompt template on the platform, then pass the managed prompt_id and variables through Haystack’s generation_kwargs.extra_body.prompt. Use schema_version: 2 for all new integrations.

import os
from haystack import Pipeline
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
PROMPT_ID = "YOUR_PROMPT_ID"
os.environ["OPENAI_API_KEY"] = os.environ["RESPAN_API_KEY"]
os.environ["OPENAI_BASE_URL"] = "https://api.respan.ai/api"
pipeline = Pipeline()
pipeline.add_component(
    "generator",
    OpenAIChatGenerator(model="gpt-5-mini"),
)
result = pipeline.run(
    {
        "generator": {
            "messages": [
                ChatMessage.from_user("Who created Python?"),
            ],
            "generation_kwargs": {
                "temperature": 0.0,
                "extra_body": {
                    "prompt": {
                        "prompt_id": PROMPT_ID,
                        "schema_version": 2,
                        "variables": {
                            "question": "Who created Python?",
                            "context": "Python was created by Guido van Rossum and first released in 1991.",
                        },
                        "override": True,
                    }
                },
            },
        }
    }
)
print(result["generator"]["replies"][0].text)

Haystack still requires a runtime message for OpenAIChatGenerator. The Respan gateway reads extra_body.prompt, renders the managed prompt from the platform, and uses the prompt variables for the final model request. The prompt must have a deployed version, otherwise the gateway returns a 404.

See the full Haystack examples.