Together AI (tracing)

The Together AI SDKs are official clients for Together’s inference platform. respan-instrumentation-together and @respan/instrumentation-together-ai emit Respan spans for direct Together AI calls.

Set up Respan

Sign up - Create an account at platform.respan.ai
Create an API key - Generate one on the API keys page

Use Respan Gateway

See Together AI gateway setup to route Together AI model calls through the Respan gateway.

Example projects

Setup

Install packages

pip install respan-ai respan-instrumentation-together together python-dotenv

Set environment variables

export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"
export TOGETHER_API_KEY="YOUR_TOGETHER_API_KEY"

Optional:

export TOGETHER_MODEL="meta-llama/Llama-3.2-3B-Instruct-Turbo"

TOGETHER_API_KEY is used by the Together AI SDK for direct provider calls. RESPAN_API_KEY is used by Respan for trace export.

Initialize and run

import os
from dotenv import load_dotenv
from respan import Respan, workflow
from respan_instrumentation_together import TogetherInstrumentor
from together import Together
load_dotenv()
respan = Respan(
    api_key=os.environ["RESPAN_API_KEY"],
    instrumentations=[TogetherInstrumentor()],
)
client = Together(api_key=os.environ["TOGETHER_API_KEY"])
@workflow(name="together_chat_completion")
def run_chat() -> str:
    response = client.chat.completions.create(
        model=os.getenv("TOGETHER_MODEL", "meta-llama/Llama-3.2-3B-Instruct-Turbo"),
        messages=[
            {
                "role": "user",
                "content": "Reply with one concise sentence about tracing.",
            }
        ],
    )
    return response.choices[0].message.content or ""
try:
    print(run_chat())
finally:
    respan.shutdown()

View your trace

Open the Traces page and search for the workflow name together_chat_completion.

Configuration

Parameter	Type	Default	Description
`api_key`	`str \| None`	`None`	Respan API key. Falls back to `RESPAN_API_KEY`.
`base_url`	`str \| None`	`None`	Respan API base URL. Falls back to `RESPAN_BASE_URL`.
`instrumentations`	`list`	`[]`	Plugin instrumentations to activate, such as `TogetherInstrumentor()` or `new TogetherAIInstrumentor()`.
`customer_identifier`	`str \| None`	`None`	Default customer identifier for all exported spans.
`metadata`	`dict \| None`	`None`	Default metadata attached to all exported spans.
`environment`	`str \| None`	`None`	Environment tag, such as `"production"`.

Supported calls

SDK call	Traced
`client.chat.completions.create(...)`	Chat completion spans
`client.chat.completions.create(..., stream=True)` / `stream: true`	Streaming chat completion spans after the stream is consumed or closed
`await async_client.chat.completions.create(...)`	Async chat completion spans
`client.completions.create(...)`	Text completion spans
`client.embeddings.create(...)`	Embedding spans with vector payloads summarized
`client.rerank.create(...)`	Rerank spans
`client.images.generate(...)`	Image generation spans
`tools=[...]` / `tool_calls` responses	Tool definitions and assistant tool calls

Attributes

Attach customer identifiers, thread IDs, workflow names, and metadata to Together AI calls with propagate_attributes or propagateAttributes.

from respan import propagate_attributes
with propagate_attributes(
    customer_identifier="user_123",
    thread_identifier="conversation_456",
    trace_group_identifier="together_support_chat",
    metadata={"plan": "pro", "workflow_name": "together_support_chat"},
):
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.2-3B-Instruct-Turbo",
        messages=[{"role": "user", "content": "Summarize our support policy."}],
    )

Attribute	Type	Description
`customer_identifier`	`str`	Identifies the end user in Respan analytics.
`thread_identifier`	`str`	Groups related messages into a conversation.
`trace_group_identifier`	`str`	Groups spans by workflow name.
`metadata`	`dict`	Custom key-value pairs merged with default metadata.

Examples

Streaming chat

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.2-3B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Write a short haiku about traces."}],
    stream=True,
)
for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Embeddings

response = client.embeddings.create(
    model="BAAI/bge-base-en-v1.5",
    input=[
        "Respan traces Together AI chat calls.",
        "Embeddings should not export vector payloads.",
    ],
)
print(len(response.data))

Tool calling

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-3B-Instruct-Turbo",
    messages=[{"role": "user", "content": "What is the weather in Tokyo?"}],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather for a city.",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
)
print(response.choices[0].message.tool_calls)