Transcription

The AI SDK provides the transcribe function to transcribe audio using a transcription model.

import { transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const transcript = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
});

The audio property can be a Uint8Array, ArrayBuffer, Buffer, string (base64 encoded audio data), or a URL.

To access the generated transcript:

const text = transcript.text; // transcript text e.g. "Hello, world!"
const segments = transcript.segments; // array of segments with start and end times, if available
const language = transcript.language; // language of the transcript e.g. "en", if available
const durationInSeconds = transcript.durationInSeconds; // duration of the transcript in seconds, if available

Streaming Transcription

Streaming transcription is an experimental feature.

Use experimental_streamTranscribe when you have live raw audio and need transcript updates before the full audio stream is complete. The function uses transcription models with streaming support; provider options configure provider-specific behavior, but the streaming operation is selected by the function itself.

import { openai } from '@ai-sdk/openai';
import { experimental_streamTranscribe as streamTranscribe } from 'ai';

const result = streamTranscribe({
  model: openai.transcription('gpt-realtime-whisper'),
  audio: audioStream, // ReadableStream<Uint8Array | string>
  inputAudioFormat: { type: 'audio/pcm', rate: 24000 },
  providerOptions: {
    openai: {
      language: 'en',
      streaming: {
        delay: 'low',
      },
    },
  },
});

for await (const part of result.fullStream) {
  if (part.type === 'transcript-delta') {
    process.stdout.write(part.delta);
  }

  if (part.type === 'transcript-partial') {
    console.log('partial:', part.text);
  }

  if (part.type === 'transcript-final') {
    console.log('final:', part.text);
  }
}

console.log(await result.text);

fullStream is a single-consumer live stream and can only be accessed once. When you need both stream parts and final results, access fullStream first and await the result promises while or after consuming it. Accessing a result promise first consumes the stream internally, so fullStream is no longer available. This avoids retaining an unbounded replay buffer for live audio.

To access the final transcript metadata:

const text = await result.text; // final transcript text
const segments = await result.segments; // final segments with timing, if available
const language = await result.language; // language of the transcript, if available
const durationInSeconds = await result.durationInSeconds; // duration in seconds, if available

The audio stream must contain raw audio chunks. Uint8Array chunks are raw bytes; string chunks are base64-encoded raw bytes. Always set inputAudioFormat to match the chunks you send.

String model IDs resolve through the global provider (AI Gateway by default). AI Gateway supports streaming transcription for supported models (e.g. openai/gpt-realtime-whisper, elevenlabs/eleven-scribe-2-realtime, xai/grok-stt), so string IDs work: experimental_streamTranscribe({ model: 'openai/gpt-realtime-whisper', ... }). You can also pass a provider model instance (e.g. openai.transcription('gpt-realtime-whisper')) to stream directly against the provider.

OpenAI streaming transcription uses openai.transcription('gpt-realtime-whisper'). Cartesia uses cartesia.transcription('ink-2') for streaming-only Ink 2 transcription. ElevenLabs uses elevenLabs.transcription('scribe_v2_realtime') for Scribe v2 Realtime. xAI uses the same xai.transcription() model for request/response and streaming transcription; experimental_streamTranscribe selects the provider's WebSocket STT transport.

import { xai } from '@ai-sdk/xai';
import { experimental_streamTranscribe as streamTranscribe } from 'ai';

const result = streamTranscribe({
  model: xai.transcription(),
  audio: audioStream,
  inputAudioFormat: { type: 'audio/pcm', rate: 16000 },
  providerOptions: {
    xai: {
      language: 'en',
      keyterm: ['AI SDK', 'Grok'],
      streaming: {
        interimResults: true,
        endpointing: 500,
      },
    },
  },
});

Some providers require WebSocket headers for direct streaming STT. In those runtimes, pass a provider-specific webSocket implementation when creating the provider.

Settings

Provider-Specific settings

Transcription models often have provider or model-specific settings which you can set using the providerOptions parameter.

import { transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const transcript = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
  providerOptions: {
    openai: {
      timestampGranularities: ['word'],
    },
  },
});

Download Size Limits

When audio is a URL, the SDK downloads the file with a default 2 GiB size limit. You can customize this using createDownload:

import { transcribe, createDownload } from 'ai';
import { openai } from '@ai-sdk/openai';

const transcript = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  download: createDownload({ maxBytes: 50 * 1024 * 1024 }), // 50 MB limit
});

You can also provide a fully custom download function:

import { transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';

const transcript = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  download: async ({ url }) => {
    const res = await myAuthenticatedFetch(url);
    return {
      data: new Uint8Array(await res.arrayBuffer()),
      mediaType: res.headers.get('content-type') ?? undefined,
    };
  },
});

If a download exceeds the size limit, a DownloadError is thrown:

import { transcribe, DownloadError } from 'ai';
import { openai } from '@ai-sdk/openai';

try {
  await transcribe({
    model: openai.transcription('whisper-1'),
    audio: new URL('https://example.com/audio.mp3'),
  });
} catch (error) {
  if (DownloadError.isInstance(error)) {
    console.log('Download failed:', error.message);
  }
}

Abort Signals and Timeouts

transcribe accepts an optional abortSignal parameter of type AbortSignal that you can use to abort the transcription process or set a timeout.

This is particularly useful when combined with URL downloads to prevent long-running requests:

import { openai } from '@ai-sdk/openai';
import { transcribe } from 'ai';

const transcript = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  abortSignal: AbortSignal.timeout(5000), // Abort after 5 seconds
});

Custom Headers

transcribe accepts an optional headers parameter of type Record<string, string> that you can use to add custom headers to the transcription request.

import { openai } from '@ai-sdk/openai';
import { transcribe } from 'ai';
import { readFile } from 'fs/promises';

const transcript = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
  headers: { 'X-Custom-Header': 'custom-value' },
});

Warnings

Warnings (e.g. unsupported parameters) are available on the warnings property.

import { openai } from '@ai-sdk/openai';
import { transcribe } from 'ai';
import { readFile } from 'fs/promises';

const transcript = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
});

const warnings = transcript.warnings;

Error Handling

When transcribe cannot generate a valid transcript, it throws a AI_NoTranscriptGeneratedError.

This error can arise for any of the following reasons:

The model failed to generate a response
The model generated a response that could not be parsed

The error preserves the following information to help you log the issue:

responses: Metadata about the transcription model responses, including timestamp, model, and headers.
cause: The cause of the error. You can use this for more detailed error handling.

import { transcribe, NoTranscriptGeneratedError } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

try {
  await transcribe({
    model: openai.transcription('whisper-1'),
    audio: await readFile('audio.mp3'),
  });
} catch (error) {
  if (NoTranscriptGeneratedError.isInstance(error)) {
    console.log('NoTranscriptGeneratedError');
    console.log('Cause:', error.cause);
    console.log('Responses:', error.responses);
  }
}

Transcription Models

Provider	Model
OpenAI	`whisper-1`
OpenAI	`gpt-4o-transcribe`
OpenAI	`gpt-4o-mini-transcribe`
ElevenLabs	`scribe_v1`
ElevenLabs	`scribe_v1_experimental`
ElevenLabs	`scribe_v2`
ElevenLabs	`scribe_v2_realtime`
Groq	`whisper-large-v3-turbo`
Groq	`whisper-large-v3`
Mistral	`voxtral-mini-latest`
Azure OpenAI	`whisper-1`
Azure OpenAI	`gpt-4o-transcribe`
Azure OpenAI	`gpt-4o-mini-transcribe`
Rev.ai	`machine`
Rev.ai	`low_cost`
Rev.ai	`fusion`
Deepgram	`base` (+ variants)
Deepgram	`enhanced` (+ variants)
Deepgram	`nova` (+ variants)
Deepgram	`nova-2` (+ variants)
Deepgram	`nova-3` (+ variants)
Gladia	`default`
AssemblyAI	`universal-3-5-pro`
AssemblyAI	`universal-3-pro`
Fal	`whisper`
Fal	`wizper`
Google Vertex	`chirp_2`
Google Vertex	`chirp_3`
Google Vertex	`telephony`
xAI	`default`
Cartesia	`ink-whisper`
Cartesia	`ink-2`
Fish Audio	`transcribe-1`

Above are a small subset of the transcription models supported by the AI SDK providers. For more, see the respective provider documentation.