Configure plugins

Everything the gateway does beyond relaying a request is set in one file, gateway.toml. Each block of the file turns on one capability. The gateway reads gateway.toml from the directory it starts in, or the file you pass with -c.

A complete example

This file connects three providers, protects the gateway with a key, routes a model name to two providers that race, gives every model web search, and traces every request in Respan.

client_keys = ["env:GATEWAY_KEY"]   # Who may call the gateway
[providers.openai]                  # A vendor and its key
vendor = "openai"
key = "env:OPENAI_API_KEY"
[providers.gemini]
vendor = "gemini"
key = "env:GEMINI_API_KEY"
[providers.respan]                  # 1000+ models with one key
vendor = "respan"
key = "env:RESPAN_API_KEY"
[routes.smart]                      # Clients ask for "smart"
to = ["respan/claude-sonnet-4-5-20250929", "openai/gpt-5.5"]
hedge = "800ms"                     # Race them; the first answer wins
[mcp.search]                        # An MCP server
url = "https://mcp.exa.ai/mcp"
[plugins.tool_loop]                 # Its tools, for every model
servers = ["search"]
[plugins.respan]                    # A trace in Respan per request
key = "env:RESPAN_API_KEY"

env:NAME reads a value from the environment, or from a .env file in the current directory or a parent. A literal value works too. Settings that aren’t in a [table], such as client_keys, go at the top of the file.

Check the file before you start the gateway. check prints what the gateway will serve, and --online also calls each provider and MCP server with its key:

respan-gateway check --online
respan-gateway

To apply an edit without a restart, send the process SIGHUP, start it with --watch, or click Reload on the web UI’s Config page. Requests already in flight finish on the old config, and an invalid file is refused while the old one keeps serving.

Providers

Each [providers.NAME] block adds one provider. vendor brings the vendor’s URLs, formats and model list, so a key is all you add.

[providers.groq]
vendor = "groq"
key = "env:GROQ_API_KEY"

Built-in vendors: openai, anthropic, gemini, respan, groq, deepseek, openrouter, xai, mistral, together, fireworks, ollama, azure, vllm, cloudflare and typesafe.

Any other endpoint that speaks an OpenAI-, Anthropic- or Gemini-style API works with a base_url and its format:

[providers.lmstudio]
base_url = "http://localhost:1234/v1"
formats = ["openai-chat"]

A request’s model name picks its provider: the provider whose model list has the model, or the one you name with provider/model.

Client keys

Set client_keys to make every request carry a gateway key. The gateway strips it before calling a provider, so providers never see it. Without client keys the gateway answers only on your machine, and it refuses to start on a public address.

client_keys = [
  { name = "ops", key = "env:OPS_KEY", admin = true },
  { name = "app", key = "env:APP_KEY", tools = ["search__*"] },
]

respan-gateway init --client-key writes a strong key into a new gateway.toml. You can also pass keys as RESPAN_CLIENT_KEYS, separated by commas.

Routing

A route gives a model name to one or more candidates. Clients ask for the route’s name as the model.

[routes.cheap-and-good]
to = ["openai/gpt-5.5", "respan/claude-sonnet-4-5-20250929", "gemini/gemini-3.6-flash"]
pick = "cheapest"
needs = ["tools"]
tries = 2

A route’s name can be a glob, such as [routes."llama-*"], to catch a family of models. to = "groq" sends them to a provider under the name asked for.

A request goes to the first match of: a route of that exact name, a provider/model prefix, the first glob route that matches, the provider whose model list has the model, and finally the default provider. Set the default with:

[plugins.routing]
default = "openai"

To see where a model goes and why, open http://localhost:8080/ui/api/routes/explain?model=smart.

To price models the gateway can’t find a price for, declare it in US dollars per million tokens. cheapest and the request log’s cost use it:

[models."lmstudio/qwen3-8b"]
pricing = { input = 0.05, output = 0.2 }

MCP servers

Each [mcp.NAME] block puts an MCP server behind the gateway at /mcp/NAME. Its fields match an MCP client’s mcp.json, so you can copy an entry across. The server’s credentials stay on the gateway; clients use a gateway key.

[mcp.github]
url = "https://api.githubcopilot.com/mcp/"
auth = { bearer = "env:GITHUB_TOKEN" }

To serve every MCP server at one endpoint, /mcp, turn on the aggregate. Tools are named server__tool:

[plugins.mcp_aggregate]
deny = ["files__write_*"]

Tool loop

The tool loop gives every chat request the tools of your MCP servers. When the model calls one, the gateway runs it, sends the result back to the model, and returns only the final answer. It works with any chat format and any provider, and in realtime voice sessions. A client needs no MCP support.

[plugins.tool_loop]
servers = ["search", "github"]
allow = ["search__*", "github__search_*"]
deny = ["github__delete_*"]

A tool that fails becomes a result the model reads, not a failed request. A client skips the loop for one request with the header x-respan-tool-loop: off.

Respan tracing

Turn on the respan plugin to send each request to Respan as a trace, with its model, provider, tokens, cost and latency. Find them in Logs.

[plugins.respan]
key = "env:RESPAN_API_KEY"
content = false            # Send metadata only, without messages or replies

Request log

The web UI’s request history is always on. Add a file to keep it across restarts:

[plugins.request_log]
history = 1000
file = "./requests.jsonl"

Conversation state

When a Responses API client is translated to another provider, the gateway keeps each response in memory so previous_response_id keeps working. It is on by default.

[plugins.responses]
keep = "1h"              # How long after it was last read
# enabled = false

State lives in one gateway’s memory: a client whose requests reach several gateways must reach the one that kept its conversation.

Server settings

These go at the top of the file, above every [table].

The full list of options is in gateway.example.toml.