← Back to dashboard
together-aiaifresh

Together AI Integration Guide

What is Together AI?

The real model

OpenAI-compatible base URL + native SDK over the full open-model catalogue.

Point `openai.OpenAI(base_url="https://api.together.ai/v1")` and chat/vision/embeddings/images Just Work; reach for the first-party `together-ai` / `together` SDK for rerank, fine-tuning, batches, dedicated endpoints, and richer image controls. The client exposes `chat`, `embeddings`, `images`, `rerank`, `fine_tuning`, `audio`, `endpoints` and more. For codeAmani, Together is the open-weight tier between Anthropic (frontier) and DeepSeek (budget reasoning): route bulk SME traffic to Llama/Qwen, generate localized FLUX imagery, or fine-tune a house model — and because the weights are open, the same models can later move to self-hosted GPUs for KDPA data-sovereignty with only a base-URL change.

Five Together AI primitives

One OpenAI-shaped endpoint fronts the whole open-model catalogue — chat, vision, embeddings, images, and reserved capacity.

Text
████████╗ ██████╗  ██████╗ ███████╗████████╗██╗  ██╗███████╗██████╗      █████╗ ██╗
╚══██╔══╝██╔═══██╗██╔════╝ ██╔════╝╚══██╔══╝██║  ██║██╔════╝██╔══██╗    ██╔══██╗██║
   ██║   ██║   ██║██║  ███╗█████╗     ██║   ███████║█████╗  ██████╔╝    ███████║██║
   ██║   ██║   ██║██║   ██║██╔══╝     ██║   ██╔══██║██╔══╝  ██╔══██╗    ██╔══██║██║
   ██║   ╚██████╔╝╚██████╔╝███████╗   ██║   ██║  ██║███████╗██║  ██║    ██║  ██║██║
   ╚═╝    ╚═════╝  ╚═════╝ ╚══════╝   ╚═╝   ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝    ╚═╝  ╚═╝╚═╝

Together AI Integration Guide

Focus: Serverless inference for 200+ open-weight models — chat, vision, embeddings, rerank, and FLUX image generation — through an OpenAI-compatible API plus a native SDK.

Overview

Together AI is an inference platform for open-source models. A single API key unlocks chat/completion models (Llama, Qwen, DeepSeek, gpt-oss, MiniMax), multimodal vision models, embedding + rerank models, audio (speech-to-text / TTS), and FLUX image + video generation — all billed per token / per image with no GPU management. The API is OpenAI-compatible, so existing OpenAI-SDK code works after changing the API key and base URL; for richer features (images, rerank, fine-tuning, dedicated endpoints) there's a first-party together-ai / together SDK.

For codeAmani, Together AI is the open-weight lever: it sits alongside Anthropic (frontier quality) and DeepSeek (budget reasoning) as the place to reach for open models, image generation, or a fine-tuned house model — without rewriting integration code.

The same OpenAI call shape simply points at Together and fans out to any open model:

Official Documentation

ResourceURL
Docs home / Quickstarthttps://docs.together.ai/docs/quickstart
OpenAI compatibilityhttps://docs.together.ai/docs/inference/openai-compatibility
API referencehttps://docs.together.ai/reference
Models listhttps://docs.together.ai/docs/serverless-models
Dedicated endpointshttps://docs.together.ai/docs/dedicated-endpoints

Setup

Option A — OpenAI SDK (drop-in)

Together's API mirrors OpenAI's REST schema. Keep the OpenAI SDK; change the key and base URL.

Bash
npm install openai
TypeScript
import OpenAI from "openai";

const together = new OpenAI({
  apiKey: process.env.TOGETHER_API_KEY!,
  baseURL: "https://api.together.ai/v1",
});

const res = await together.chat.completions.create({
  model: "meta-llama/Llama-3.3-70B-Instruct-Turbo",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(res.choices[0].message.content);

Option B — Native Together SDK (full surface)

Use the first-party SDK for images, rerank, fine-tuning, batches, and dedicated endpoints. The client reads TOGETHER_API_KEY from the environment by default.

Bash
npm install together-ai     # Node / TypeScript
pip install together        # Python
TypeScript
import Together from "together-ai";

const client = new Together(); // picks up TOGETHER_API_KEY

const chat = await client.chat.completions.create({
  model: "meta-llama/Llama-3.3-70B-Instruct-Turbo",
  messages: [{ role: "user", content: "Say this is a test!" }],
});
console.log(chat.choices);
Python
from together import Together

client = Together()  # picks up TOGETHER_API_KEY

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
)
print(resp.choices[0].message.content)

Core Patterns

Streaming

TypeScript
const stream = await client.chat.completions.create({
  model: "meta-llama/Llama-3.3-70B-Instruct-Turbo",
  messages: [{ role: "user", content: prompt }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
// Cancel anytime with stream.controller.abort()

Vision (multimodal)

Python
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-72B-Instruct",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/receipt.png"}},
        ],
    }],
)
print(resp.choices[0].message.content)

Embeddings

Python
resp = client.embeddings.create(
    model="intfloat/multilingual-e5-large-instruct",
    input=["The cat sat on the mat", "A dog played in the park"],
)
for row in resp.data:
    print(len(row.embedding), "dimensions")

Image generation (FLUX)

TypeScript
const image = await client.images.generate({
  model: "black-forest-labs/FLUX.2-pro",
  prompt: "A vibrant Nairobi street market at golden hour, photorealistic",
  width: 1024,
  height: 768,
});
console.log(image.data[0].url); // hosted URL (or b64_json if requested)
// Budget / fastest: model "black-forest-labs/FLUX.1-schnell" with steps: 4

Next.js App Router streaming route

TypeScript
// app/api/ai/together/route.ts
import OpenAI from "openai";
import { NextRequest } from "next/server";

const together = new OpenAI({
  apiKey: process.env.TOGETHER_API_KEY!,
  baseURL: "https://api.together.ai/v1",
});

export async function POST(req: NextRequest) {
  const { messages } = await req.json();

  const stream = await together.chat.completions.create({
    model: "meta-llama/Llama-3.3-70B-Instruct-Turbo",
    stream: true,
    messages,
  });

  const encoder = new TextEncoder();
  return new Response(
    new ReadableStream({
      async start(controller) {
        for await (const chunk of stream) {
          const text = chunk.choices[0]?.delta?.content ?? "";
          if (text) controller.enqueue(encoder.encode(text));
        }
        controller.close();
      },
    }),
    { headers: { "Content-Type": "text/plain; charset=utf-8" } },
  );
}

Serverless vs. Dedicated Endpoints

Together runs models two ways. Pick by traffic shape:

  • Serverless — pay per token/image on a shared pool. Default for development and bursty workloads. Just call the model ID.
  • Dedicated endpoints — reserved GPU capacity billed hourly; predictable latency for steady production traffic. Provision via the dashboard or the client.endpoints API.

AI Routing: Where Together AI Fits

In codeAmani's routing strategy, Together AI is the open-model + image tier:

TaskRecommended provider/model
Complex reasoning, agentsAnthropic claude-sonnet-4-6
Budget chain-of-thoughtDeepSeek deepseek-reasoner
Open-weight chat at scaleTogether meta-llama/Llama-3.3-70B-Instruct-Turbo
Open-weight reasoningTogether deepseek-ai/DeepSeek-V3.1 or openai/gpt-oss-120b
Vision / document understandingTogether Qwen/Qwen2.5-VL-72B-Instruct
Image generationTogether black-forest-labs/FLUX.2-pro (or FLUX.1-schnell for speed)
Fast structured JSONOpenAI gpt-4o
TypeScript
// lib/ai.ts — Together slots in as the open-model tier
function selectModel(task: "reason" | "openchat" | "vision" | "image") {
  switch (task) {
    case "reason":   return { provider: "anthropic", model: "claude-sonnet-4-6" };
    case "openchat": return { provider: "together", model: "meta-llama/Llama-3.3-70B-Instruct-Turbo" };
    case "vision":   return { provider: "together", model: "Qwen/Qwen2.5-VL-72B-Instruct" };
    case "image":    return { provider: "together", model: "black-forest-labs/FLUX.2-pro" };
  }
}

Because Together is OpenAI-shaped, the resilient-fallback pattern from the DeepSeek guide applies unchanged — retry transient 429/5xx with jittered backoff, then fall back to Anthropic Claude.

Environment Variables

Bash
# Required
TOGETHER_API_KEY=...

# Optional — the native SDK reads TOGETHER_BASE_URL (defaults to
# https://api.together.ai/v1). Point it at a self-hosted / proxied
# deployment without touching code — the open-weights on-ramp story.
TOGETHER_BASE_URL=https://api.together.ai/v1

# Base URL is set in code (OpenAI SDK): https://api.together.ai/v1
# The native SDK reads TOGETHER_API_KEY automatically.

codeAmani Notes

  • Secrets server-side only. TOGETHER_API_KEY lives in .env.local / Vercel env vars and is used only from API routes, server actions, or edge functions — never shipped to the browser. Proxy all calls through app/api/**.
  • Open weights = data-sovereignty option. For KDPA-sensitive or government workloads, the same open models (Llama, Qwen, DeepSeek) Together hosts can later be self-hosted with no code change beyond the base URL — Together is the low-friction on-ramp before committing to GPUs.
  • Images for African-market UI. FLUX on Together is a cheap way to generate localized marketing/illustration assets (Nairobi scenes, Swahili signage prompts) without a separate image provider.
  • Multilingual embeddings for Swahili RAG. intfloat/multilingual-e5-large-instruct embeds Swahili and English in one model — a better fit for code-switched East African text than English-only embeddings. Store vectors in Supabase pgvector or Pinecone.
  • Cost discipline. Open models are far cheaper than frontier APIs for high-volume SME tasks (classification, summarization, RAG synthesis) — route the bulk there and reserve Claude for quality-critical calls.
  • Mobile-first latency. For 2G/3G users, prefer smaller models (Meta-Llama-3.1-8B-Instruct-Turbo, 8B-class) and FLUX.1-schnell (4-step) to keep response and image-generation times low; stream chat responses so the UI paints early.

Troubleshooting

IssueFix
401 UnauthorizedVerify TOGETHER_API_KEY; the native SDK trims surrounding quotes, the OpenAI SDK does not — store the raw key
model not foundUse exact IDs from the models list (e.g. meta-llama/Llama-3.3-70B-Instruct-Turbo), case-sensitive
Need rerank via OpenAI SDKRerank is native-together-ai-SDK-only; image generation + embeddings are on the OpenAI-compat surface now, but Together-specific image params (steps, img-to-img image_url) still need the native SDK
429 rate limitedBack off + retry, or move steady traffic to a dedicated endpoint
Slow first token on rare modelsCold serverless start — pre-warm or use a dedicated endpoint for production