eleven-labsaifresh

ElevenLabs Integration Guide

What is ElevenLabs?

The real model

Multilingual TTS + voice cloning + audio dubbing + conversational agents.

Models: `eleven_v3` is the headline (most expressive), `eleven_multilingual_v2` for 29 languages incl. Swahili, `eleven_turbo_v2_5` for low-latency streaming (~300ms first byte). Voice cloning offers Instant (zero-shot, 30s sample) and Professional (more training, paid tier). Voices are addressed by ID and parametrised by `stability` (0-1, low = more emotional variance) and `similarity_boost`. For codeAmani, ElevenLabs is the East African voice layer — IVR for M-Pesa-adjacent flows, accessibility audio in Kiswahili, voice-first interfaces for users with low literacy.

Five ElevenLabs primitives

Voice as a first-class input/output. Swahili-capable, sub-second when needed.

Text

███████╗██╗     ███████╗██╗   ██╗███████╗███╗   ██╗██╗      █████╗ ██████╗ ███████╗
██╔════╝██║     ██╔════╝██║   ██║██╔════╝████╗  ██║██║     ██╔══██╗██╔══██╗██╔════╝
█████╗  ██║     █████╗  ██║   ██║█████╗  ██╔██╗ ██║██║     ███████║██████╔╝███████╗
██╔══╝  ██║     ██╔══╝  ╚██╗ ██╔╝██╔══╝  ██║╚██╗██║██║     ██╔══██║██╔══██╗╚════██║
███████╗███████╗███████╗ ╚████╔╝ ███████╗██║ ╚████║███████╗██║  ██║██████╔╝███████║
╚══════╝╚══════╝╚══════╝  ╚═══╝  ╚══════╝╚═╝  ╚═══╝╚══════╝╚═╝  ╚═╝╚═════╝ ╚══════╝

ElevenLabs Integration Guide

Focus: Text-to-speech, voice cloning, speech-to-text, and voice design via ElevenLabs APIs and Claude Code tooling.

Overview

ElevenLabs provides state-of-the-art AI voice generation — synthesize speech, clone voices, transcribe audio, and design custom voices. Used in codeAmani products for voice UI features, audio notifications, and multilingual TTS (including Swahili).

Here is the big picture — text flows out as audio, and user audio flows back in as text, all through one API:

Official Documentation

Resource	URL
API Reference	https://elevenlabs.io/docs/api-reference
Node.js SDK	https://github.com/elevenlabs/elevenlabs-js
Python SDK	https://github.com/elevenlabs/elevenlabs-python
Voice Library	https://elevenlabs.io/voice-library
Models Reference	https://elevenlabs.io/docs/models

MCP Server Setup

ElevenLabs provides an official MCP server for direct use in Claude Code.

Bash

# Add the ElevenLabs MCP server
claude mcp add elevenlabs -- npx -y @elevenlabs/elevenlabs-mcp

Set the API key in your MCP environment:

JSON

// .mcp.json
{
  "mcpServers": {
    "elevenlabs": {
      "command": "npx",
      "args": ["-y", "@elevenlabs/elevenlabs-mcp"],
      "env": {
        "ELEVENLABS_API_KEY": "${ELEVENLABS_API_KEY}"
      }
    }
  }
}

MCP Capabilities

Tool	Description
`text_to_speech`	Convert text to audio using a specified voice
`list_voices`	Browse available and cloned voices
`get_voice`	Get details and settings for a voice
`speech_to_text`	Transcribe audio files
`sound_generation`	Generate sound effects from text

SDK Setup

Node.js / TypeScript

Bash

npm install elevenlabs

TypeScript

import { ElevenLabsClient } from "elevenlabs";

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY,
});

Python

Bash

pip install elevenlabs

Python

from elevenlabs.client import ElevenLabs

client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])

Core Patterns

You are about to wire these up — here is how a streaming TTS request travels from your Next.js route to the listener:

Text-to-Speech (Streaming)

TypeScript

import { ElevenLabsClient } from "elevenlabs";

const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });

// Stream audio to a file
const audioStream = await client.textToSpeech.convertAsStream(
  "JBFqnCBsd6RMkjVDRZzb", // Voice ID (Rachel — default)
  {
    text: "Karibu! Welcome to your codeAmani dashboard.",
    model_id: "eleven_multilingual_v2",
    voice_settings: {
      stability: 0.5,
      similarity_boost: 0.75,
    },
  }
);

// Write to disk
import { createWriteStream } from "fs";
const writer = createWriteStream("output.mp3");
for await (const chunk of audioStream) {
  writer.write(chunk);
}
writer.end();

Next.js App Router API Route

TypeScript

// app/api/tts/route.ts
import { ElevenLabsClient } from "elevenlabs";
import { NextRequest } from "next/server";

const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });

export async function POST(req: NextRequest) {
  const { text, voiceId = "JBFqnCBsd6RMkjVDRZzb" } = await req.json();

  const audioStream = await client.textToSpeech.convertAsStream(voiceId, {
    text,
    model_id: "eleven_multilingual_v2",
  });

  // Collect chunks
  const chunks: Buffer[] = [];
  for await (const chunk of audioStream) {
    chunks.push(Buffer.from(chunk));
  }

  return new Response(Buffer.concat(chunks), {
    headers: {
      "Content-Type": "audio/mpeg",
      "Cache-Control": "no-store",
    },
  });
}

List Available Voices

TypeScript

const voices = await client.voices.getAll();

for (const voice of voices.voices) {
  console.log(`${voice.voice_id}: ${voice.name} (${voice.labels?.language ?? "multi"})`);
}

Speech-to-Text (Transcription)

TypeScript

import { createReadStream } from "fs";

const transcription = await client.speechToText.convert({
  file: createReadStream("recording.mp3"),
  model_id: "scribe_v1",
  language_code: "sw", // Swahili
});

console.log(transcription.text);

Voice cloning

Instant Voice Cloning (IVC) turns a short audio sample into a reusable voice. You upload one or more recordings, get back a voice_id, then synthesize speech with it like any other voice. This is how you give a codeAmani product its own branded voice — or let a user respond in their own voice for voice-note replies.

The flow is two API calls — clone once, reuse the voice_id forever:

Create a cloned voice, then synthesize with it

TypeScript

import { ElevenLabsClient } from "elevenlabs";
import fs from "fs";

const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });

// 1. Clone a voice from one or more audio samples
const cloned = await client.voices.ivc.create({
  name: "codeAmani Brand Voice",
  files: [
    fs.createReadStream("samples/sample-1.mp3"),
    fs.createReadStream("samples/sample-2.mp3"),
  ],
});

const voiceId = cloned.voice_id;
console.log("Cloned voice id:", voiceId);
// Persist voiceId in your DB so you can reuse it without re-cloning.

// 2. Synthesize speech using the returned voice_id
const audio = await client.textToSpeech.convert(voiceId, {
  text: "Karibu! Hii ni sauti yako mpya kwenye codeAmani.",
  model_id: "eleven_multilingual_v2",
  outputFormat: "mp3_44100_128",
});

const chunks: Buffer[] = [];
for await (const chunk of audio) {
  chunks.push(Buffer.from(chunk));
}
fs.writeFileSync("cloned-output.mp3", Buffer.concat(chunks));

The create call returns an AddVoiceIVCResponseModel with voice_id (use this everywhere) and requires_verification (whether the voice must be verified before high-volume use). The cloned voice also appears in client.voices.getAll().

Gotcha — consent and sample quality. Only clone voices you have explicit permission to use; cloning a real person's voice without consent breaks ElevenLabs' terms (and KDPA-style consent expectations for biometric/voice data). Quality is bounded by your samples: use clean, single-speaker recordings with no background noise or music — a minute of crisp audio beats ten minutes of noisy phone audio. Keep the API key server-side; never expose it to the client doing the upload.

Model Reference

Model ID	Use Case	Latency
`eleven_multilingual_v2`	High quality, 29 languages	~1s
`eleven_turbo_v2_5`	Low latency, English	~250ms
`eleven_flash_v2_5`	Lowest latency, real-time	~75ms
`scribe_v1`	Speech-to-text transcription	—

For Swahili TTS, use eleven_multilingual_v2 — it includes Swahili in its language support.

Webhooks

ElevenLabs sends webhook events for async operations. Verify signatures using the secret from your dashboard:

TypeScript

// app/api/webhooks/elevenlabs/route.ts
import { NextRequest } from "next/server";
import crypto from "crypto";

export async function POST(req: NextRequest) {
  const body = await req.text();
  const signature = req.headers.get("xi-signature") ?? "";
  const secret = process.env.ELEVENLABS_WEBHOOK_SECRET!;

  const hmac = crypto
    .createHmac("sha256", secret)
    .update(body)
    .digest("hex");

  if (`sha256=${hmac}` !== signature) {
    return new Response("Unauthorized", { status: 401 });
  }

  const event = JSON.parse(body);
  // Handle event.type: "speech_history.item_created", etc.

  return new Response("OK");
}

Environment Variables

Bash

# Required
ELEVENLABS_API_KEY=sk_...

# Optional
ELEVENLABS_WEBHOOK_SECRET=whsec_...
ELEVENLABS_DEFAULT_VOICE_ID=JBFqnCBsd6RMkjVDRZzb

Common Use Cases

Use Case	Approach
Voice notifications	POST to `/api/tts`, play on client
Swahili audio content	`eleven_multilingual_v2` + `language_code: "sw"`
Voice cloning	`voices.ivc.create()` with audio samples → get `voice_id` → use in TTS (see Voice cloning)
Audio transcription	`scribe_v1` model + `speech_to_text.convert()`
Real-time voice AI	WebSocket streaming with `eleven_flash_v2_5`

Troubleshooting

Issue	Fix
`401 Unauthorized`	Check `ELEVENLABS_API_KEY` is set and valid
Audio sounds robotic	Increase `stability` (0.7–0.9) and `similarity_boost` (0.8)
Swahili not accurate	Use `eleven_multilingual_v2` — turbo models have limited Swahili
Large audio files slow	Use streaming (`convertAsStream`) instead of buffered response
MCP server not connecting	Run `claude mcp list` and check `ELEVENLABS_API_KEY` in env