From be2446e1887140fa8a4f790c556b406848061e73 Mon Sep 17 00:00:00 2001 From: Shoubhit Dash Date: Sun, 27 Sep 2026 19:34:36 +0530 Subject: [PATCH] feat(ai): add ElevenLabs Scribe transcription route (#51641) --- packages/ai/AGENTS.md | 2 +- packages/ai/README.md | 22 +- packages/ai/docs/media-design.md | 38 ++-- .../src/protocols/deepgram-transcription.ts | 15 +- .../src/protocols/elevenlabs-transcription.ts | 211 ++++++++++++++++++ .../ai/src/protocols/openai-transcription.ts | 2 +- .../ai/src/protocols/utils/media-input.ts | 11 +- .../ai/src/protocols/utils/speaker-turns.ts | 10 + packages/ai/src/providers/elevenlabs.ts | 5 + packages/ai/src/transcription.ts | 2 +- packages/ai/test/exports.test.ts | 5 + ...ups-diarized-words-into-speaker-turns.json | 32 +++ ...ranscribes-audio-with-word-timestamps.json | 50 +++++ .../elevenlabs-transcription.recorded.test.ts | 50 +++++ packages/ai/test/transcription.test.ts | 114 +++++++++- 15 files changed, 525 insertions(+), 44 deletions(-) create mode 100644 packages/ai/src/protocols/elevenlabs-transcription.ts create mode 100644 packages/ai/src/protocols/utils/speaker-turns.ts create mode 100644 packages/ai/test/fixtures/recordings/elevenlabs-transcription/groups-diarized-words-into-speaker-turns.json create mode 100644 packages/ai/test/fixtures/recordings/elevenlabs-transcription/transcribes-audio-with-word-timestamps.json create mode 100644 packages/ai/test/provider/elevenlabs-transcription.recorded.test.ts diff --git a/packages/ai/AGENTS.md b/packages/ai/AGENTS.md index 49e3ea0f046..3547ca09211 100644 --- a/packages/ai/AGENTS.md +++ b/packages/ai/AGENTS.md @@ -102,7 +102,7 @@ Media does not fit the SSE-frames-to-event-state-machine LLM route. `MediaRoute. `MediaProtocol.stream` is the incremental kind every speech route uses, with the same discipline as LLM protocols. `MediaRoute.stream` submits the caller's request as `MediaProtocol.Addressed` (`{ ...request, mode }`, `mode: "generate" | "stream"`), so one provider stays one protocol: `body.from`, the endpoint path, and `frames` read `request.mode` to pick the body, path, and framing. `frames(bytes, context)` returns frames — `Framing.sse`, `Framing.lines`, `Framing.document` (a single-document response shaped like a streamed record), or the raw `bytes` for chunked audio. `initial()` is fresh per-response parser state; `step` folds each frame into it and emits modality events; `finish(state, context)` runs once after the last frame with the request, body, and observed `http` (header-only usage lives there) and emits exactly one terminal event or fails with `route.incomplete()`. Keep parser state to real accumulators and derive anything the request or body determines in `finish`. `generate` runs the same stream and folds it with the modality's `collect`. Request-derived URL parameters go on the body's `query` (array values repeat the parameter), applied before route and caller `http.query`. Decode frames with `route.decodeFrame` and raise stream-time failures with `route.frameError` (the frame stays on `reason.body`); protocols never thread HTTP context, because the route fills `reason.http` on stream errors that lack it. Speech protocols share `protocols/utils/speech-stream.ts` for deltas, timestamps, voice ids, PCM and container descriptions, and the terminal asset. -Every modality route is the inline | stream | queued union (transcription uses all three: OpenAI and Gemini stream, Deepgram is inline, AssemblyAI is queued), every client is `MediaClient.make(Service, { modality, responseEvents })` (`src/media-client.ts`), which dispatches on the route's `kind`, and every model composes through `composeRoute`. fal queue protocols come from `protocols/utils/fal-queue.ts`, bodies are `json`, `multipart`, or `binary` (a raw upload), and a queued protocol that must upload media before submitting implements `start.prepare` (`MediaProtocol.Prepare`; AssemblyAI `/v2/upload`). +Every modality route is the inline | stream | queued union (transcription uses all three: OpenAI and Gemini stream, Deepgram and ElevenLabs are inline, AssemblyAI is queued), every client is `MediaClient.make(Service, { modality, responseEvents })` (`src/media-client.ts`), which dispatches on the route's `kind`, and every model composes through `composeRoute`. fal queue protocols come from `protocols/utils/fal-queue.ts`, bodies are `json`, `multipart`, or `binary` (a raw upload), and a queued protocol that must upload media before submitting implements `start.prepare` (`MediaProtocol.Prepare`; AssemblyAI `/v2/upload`). ### URL Construction diff --git a/packages/ai/README.md b/packages/ai/README.md index c8dc9ab98f3..c96298c88bd 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -873,11 +873,12 @@ for await (const event of ai.speech.stream({ model, text: "Hello from OpenCode." ## Transcription Transcription (speech-to-text) is the one modality whose providers use every route kind: OpenAI and Gemini stream, -Deepgram answers inline, and AssemblyAI is queued. `Transcription.generate` and `Transcription.stream` work on all of -them; `Transcription.start` / `resume` return a `Generation` on queued routes and fail with `UnsupportedOperation` -elsewhere. Models come from `.transcription(...)` selectors on the `OpenAI`, `Google`, `Deepgram`, and `AssemblyAI` -facades. Common fields (`language`, `prompt`, `timestamps: "none" | "segment" | "word"`, `diarize`, `speakers`) lower -natively or fail with a typed `AIError` before any network call; a route may return more than asked. +Deepgram and ElevenLabs answer inline, and AssemblyAI is queued. `Transcription.generate` and `Transcription.stream` +work on all of them; `Transcription.start` / `resume` return a `Generation` on queued routes and fail with +`UnsupportedOperation` elsewhere. Models come from `.transcription(...)` selectors on the `OpenAI`, `Google`, +`Deepgram`, `ElevenLabs`, and `AssemblyAI` facades. Common fields (`language`, `prompt`, +`timestamps: "none" | "segment" | "word"`, `diarize`, `speakers`) lower natively or fail with a typed `AIError` before +any network call; a route may return more than asked. ```ts import { Console, Effect, Stream } from "effect" @@ -889,7 +890,7 @@ const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY }) const program = Effect.gen(function* () { const audio = yield* Media.file("./call.mp3") - // Speaker-labelled segments; labels are provider-native strings ("A", "0", "spk:0"). + // Speaker-labelled segments; labels are provider-native strings ("A", "0", "spk:0", "speaker_0"). const response = yield* Transcription.generate({ model: Deepgram.configure({ apiKey }).transcription("nova-3"), audio, @@ -899,7 +900,7 @@ const program = Effect.gen(function* () { response.text // "Hello from OpenCode." response.segments // [{ text, startSeconds, endSeconds, speaker: "0" }] response.words // [{ text, startSeconds, endSeconds, speaker, confidence }] - response.language // the provider's own value, lowercased ("en", "english", "en_us") + response.language // the provider's own value, lowercased ("en", "eng", "english", "en_us") // Text deltas as the model transcribes, then one finish carrying the whole transcript. yield* Transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio }).pipe( @@ -923,7 +924,12 @@ Provider notes: - **OpenAI** takes inline audio only; `diarize` needs `gpt-4o-transcribe-diarize`, timestamps need `whisper-1`, and `whisper-1` does not stream. - **Gemini** needs a transcribe model (`gemini-3.5-transcribe`); `prompt` and `speakers` fail typed. - **Deepgram** detects the language unless `language` is set; vocabulary goes in `providerOptions.keyterm`. -- **AssemblyAI** uploads inline audio before submitting and is the only route that accepts `speakers`. +- **ElevenLabs** (`scribe_v2`) uploads inline audio as the multipart `file` and sends a URL as `source_url`. Words + always carry timestamps, and segments are speaker turns, so `diarize`, `timestamps: "segment"`, or `speakers` turns + on diarization. `speakers` is an upper bound (`num_speakers`); `prompt` fails typed (vocabulary goes in + `providerOptions.keyterms`), as do webhook delivery and per-channel output (`use_multi_channel` without + `multichannel_output_style: "combined"`). +- **AssemblyAI** uploads inline audio before submitting and treats `speakers` as the exact speaker count. The promise client mirrors the Effect API: diff --git a/packages/ai/docs/media-design.md b/packages/ai/docs/media-design.md index 19ccebc9243..f98ff4eb7c9 100644 --- a/packages/ai/docs/media-design.md +++ b/packages/ai/docs/media-design.md @@ -1,7 +1,6 @@ # Media generation in `@opencode/ai` — public API direction -Status: phases 1–4 implemented (through Image queued routes and partial images; ElevenLabs Scribe transcription -pending); phase 5 proposal. +Status: phases 1–4 implemented (through Image queued routes and partial images); phase 5 proposal. ## Goal @@ -271,8 +270,8 @@ Deferred: `Speech.session(...)` — input-streaming TTS where text arrives incre #### Transcription (STT) Shipped as the second half of phase 3 (`src/transcription.ts`, `src/transcription-client.ts`, protocols -`openai-transcription`, `google-transcription`, `deepgram-transcription`, `assemblyai-transcription`; new `AssemblyAI` -facade). +`openai-transcription`, `google-transcription`, `deepgram-transcription`, `elevenlabs-transcription`, +`assemblyai-transcription`; new `AssemblyAI` facade). ```ts const request = Transcription.request({ @@ -281,7 +280,7 @@ const request = Transcription.request({ language: "en", // provider-native passthrough timestamps: "segment", // none | segment | word diarize: true, - speakers: 2, // exact speaker count (AssemblyAI only) + speakers: 2, // speaker count (AssemblyAI exact, ElevenLabs maximum) providerOptions: { known_speaker_names: ["agent"] }, }) @@ -309,17 +308,23 @@ upload); `packages/ai/AGENTS.md` (Media Routes) describes both. Settled rules: - **Timestamps.** A granularity the selected route or model cannot produce fails as `UnsupportedOperation` - (`media.timestamps`), following Speech; a route that returns more than asked (Deepgram and AssemblyAI always return - words) is not stripped. Segments always carry start and end times: Gemini times each transcription part from its + (`media.timestamps`), following Speech; a route that returns more than asked (Deepgram, ElevenLabs, and AssemblyAI + always return words) is not stripped. Segments always carry start and end times: Gemini times each transcription part from its word offsets, so segment timestamps and diarization also request word offsets there. - **Diarization.** `diarize` means segments (and words, where the provider labels them) carry `speaker`. Labels are - provider-native strings — OpenAI `A` or a known speaker name, Deepgram `0`, Gemini `spk:0`, AssemblyAI `A` — with no - cross-provider speaker model. `speakers` is the exact number of speakers to label, which AssemblyAI (`speakers_expected`, the only route that - accepts it) treats as a constraint rather than a hint. + provider-native strings — OpenAI `A` or a known speaker name, Deepgram `0`, Gemini `spk:0`, AssemblyAI `A`, + ElevenLabs `speaker_0` — with no cross-provider speaker model. `speakers` is the number of speakers to label: + AssemblyAI (`speakers_expected`) treats it as an exact constraint rather than a hint, and ElevenLabs + (`num_speakers`) as the maximum. Both turn on diarization for it; the other routes reject it. +- **Segments from words.** ElevenLabs returns only a token list (`word`, `spacing`, `audio_event`), so its segments + are speaker turns: consecutive words and spacing with one `speaker_id`, text joined from the provider's own spacing + tokens. `words` drops spacing and audio events. Segments therefore need diarization, which `timestamps: "segment"` + turns on, as AssemblyAI's utterances need speaker labels. - **Language** is passed through (`language`, OpenAI `gpt-transcribe` `languages[]`, Gemini `languageCodes`, - AssemblyAI `language_code`). `response.language` is the provider's own value, lowercased but not normalized: an - ISO code on most routes (AssemblyAI's detection returns `en`), `english` from whisper-1. Deepgram and AssemblyAI - assume English unless asked to detect, so a missing `language` enables their detection. + AssemblyAI and ElevenLabs `language_code`). `response.language` is the provider's own value, lowercased but not + normalized: an ISO code on most routes (AssemblyAI's detection returns `en`, ElevenLabs ISO 639-3 `eng`), `english` + from whisper-1. Deepgram and AssemblyAI assume English unless asked to detect, so a missing `language` enables their + detection. - **Gemini** requires a transcribe model; other model ids fail with `UnsupportedOperation` before the call, because general models ignore `audioTranscriptionConfig` and answer conversationally. Streamed chunks carry whole speaker turns (one part per turn), which join with a space. @@ -332,11 +337,12 @@ Settled rules: | OpenAI | stream (`stream: true` in `stream` mode; `whisper-1` ignores `stream`, so it emits only `finish`) | multipart `file` (inline only) | `whisper-1` (`verbose_json`); diarize model: `segment` | `gpt-4o-transcribe-diarize` (`diarized_json`) | `speakers`; `prompt` on the diarize model | `tokens` or `seconds` | | Gemini | stream (`generateContent` / `streamGenerateContent`) | `inlineData` or Gemini Files `fileData` | `audioTranscriptionConfig.wordTimestamp` | `audioTranscriptionConfig.diarization` | `prompt`, `speakers` | `tokens` | | Deepgram | inline | raw body, or JSON `{ url }` | words always; `segment` → `utterances` | `diarize_model=latest` + `utterances` | `prompt`, `speakers` | `seconds` (`metadata.duration`) | +| ElevenLabs | inline | multipart `file`, or `source_url` | words always; `segment` → `diarize` (speaker turns) | `diarize` | `prompt`; `webhook`, per-channel `use_multi_channel` | `seconds` (`audio_duration_secs`) | | AssemblyAI | queued (upload → submit → poll) | `/v2/upload` then `audio_url`, or a URL | words always; `segment` → `speaker_labels` | `speaker_labels` | — | `seconds` (`audio_duration`) | Deferred: `Transcription.session(...)` — realtime STT over WebSocket (Deepgram live, AssemblyAI streaming, ElevenLabs realtime, OpenAI realtime transcription) — is the same future scoped `session` shape as input-streaming TTS and ships -with the realtime work in phase 5. ElevenLabs Scribe is not implemented yet. +with the realtime work in phase 5. ### `Generation` — shared async execution @@ -414,7 +420,7 @@ implemented): | `OpenAI` | responses (default), chat | Images API (stream) | *Sora skipped (decision 8)* | ✓ | ✓ | | | `Google` | Gemini | Gemini-native | Veo | Gemini TTS | `gemini-3.5-transcribe` | | | `XAI` | ✓ | ✓ | ✓ | | | | -| `ElevenLabs` | | | | ✓ | *Scribe (pending)* | *soundEffect, music (phase 5)* | +| `ElevenLabs` | | | | ✓ | Scribe | *soundEffect, music (phase 5)* | | `Cartesia` | | | | ✓ | | | | `Deepgram` | | | | Aura | ✓ | | | `Fal` | | ✓ (queued) | ✓ | | | | @@ -467,7 +473,7 @@ Foundation + Image ship together as the reference implementation, serially. Vide 1. **Foundation** — per-modality selectors, `Media`, `Generation`, `Poll`, `Usage` union, `MediaProtocol` kinds, `@opencode/ai/promise` with `llm` + `image`. Port the five existing image protocols onto it. Unify `MediaPart` and add the `media` LLM event (fixes Gemini image output being dropped). 2. **Video** — ✅ Veo, xAI, fal, Runway shipped (`MediaProtocol.queued`, `Video.start/generate/resume/stream`, promise `ai.video`). Deferred: `Video.complete` (webhooks), Luma, Kling, MiniMax, Replicate. -3. **Speech + Transcription** — ✅ Speech: OpenAI, Gemini TTS, ElevenLabs, Cartesia, Deepgram shipped (`MediaProtocol.stream`, `Speech.generate/stream`, promise `ai.speech`). ✅ Transcription: OpenAI, Gemini, Deepgram, AssemblyAI shipped across all three route kinds (`Transcription.generate/stream/start/resume`, promise `ai.transcription`). Pending: ElevenLabs Scribe. Deferred: `Speech.session` and `Transcription.session` (WebSocket streaming). +3. **Speech + Transcription** — ✅ Speech: OpenAI, Gemini TTS, ElevenLabs, Cartesia, Deepgram shipped (`MediaProtocol.stream`, `Speech.generate/stream`, promise `ai.speech`). ✅ Transcription: OpenAI, Gemini, Deepgram, ElevenLabs Scribe, AssemblyAI shipped across all three route kinds (`Transcription.generate/stream/start/resume`, promise `ai.transcription`). Deferred: `Speech.session` and `Transcription.session` (WebSocket streaming). 4. **Image queued routes and partials** — ✅ BFL, fal, Replicate, and Stability creative upscale queued; Stability generate inline; OpenAI `partial_images` streaming (`image-partial` restored). Imagen dropped: shut down on the Gemini API and discontinued on Vertex (2026-06-30). Deferred: Stability's synchronous edit and fast/conservative upscale endpoints. 5. **Later** — ElevenLabs music/SFX, Lyria, `Speech.session` / `Transcription.session`, realtime. diff --git a/packages/ai/src/protocols/deepgram-transcription.ts b/packages/ai/src/protocols/deepgram-transcription.ts index fe345ed67ed..1c6e8517c8c 100644 --- a/packages/ai/src/protocols/deepgram-transcription.ts +++ b/packages/ai/src/protocols/deepgram-transcription.ts @@ -6,6 +6,7 @@ import { mergeJsonRecords, type OpenString } from "../schema/index.js" import { TranscriptionModel, TranscriptionResponse, type TranscriptionRequestFor } from "../transcription.js" import { ProviderShared } from "./shared.js" import { MediaInput } from "./utils/media-input.js" +import { SpeakerTurns } from "./utils/speaker-turns.js" const route = MediaProtocol.identity({ id: "deepgram-transcription", name: "Deepgram", provider: "deepgram" }) export const DEFAULT_BASE_URL = "https://api.deepgram.com" @@ -115,16 +116,6 @@ const speaker = (value: number | undefined) => (value === undefined ? undefined const wordText = (word: typeof Word.Type) => word.punctuated_word ?? word.word -// Utterances split on pauses, not speakers: the v2 diarizer labels a whole utterance with one speaker even when its -// words change speaker, so segments split each utterance at speaker changes. -const speakerTurns = (words: ReadonlyArray) => - words.reduce>>((turns, word) => { - const last = turns.at(-1) - if (last === undefined || last[0].speaker !== word.speaker) return [...turns, [word]] - last.push(word) - return turns - }, []) - const decodeResponse = Effect.fn("DeepgramTranscription.decodeResponse")(function* ( response: HttpClientResponse.HttpClientResponse, ) { @@ -136,6 +127,8 @@ const decodeResponse = Effect.fn("DeepgramTranscription.decodeResponse")(functio const requestID = output.value.metadata?.request_id return new TranscriptionResponse({ text: alternative.transcript, + // Utterances split on pauses, not speakers: the v2 diarizer labels a whole utterance with one speaker even when + // its words change speaker, so segments split each utterance at speaker changes. segments: output.value.results.utterances?.flatMap((utterance) => utterance.words === undefined || utterance.words.length === 0 ? [ @@ -146,7 +139,7 @@ const decodeResponse = Effect.fn("DeepgramTranscription.decodeResponse")(functio speaker: speaker(utterance.speaker), }, ] - : speakerTurns(utterance.words).map((turn) => ({ + : SpeakerTurns.group(utterance.words, (word) => word.speaker).map((turn) => ({ text: turn.map(wordText).join(" "), startSeconds: turn[0].start, endSeconds: turn[turn.length - 1].end, diff --git a/packages/ai/src/protocols/elevenlabs-transcription.ts b/packages/ai/src/protocols/elevenlabs-transcription.ts new file mode 100644 index 00000000000..626df5e8e0a --- /dev/null +++ b/packages/ai/src/protocols/elevenlabs-transcription.ts @@ -0,0 +1,211 @@ +import { Effect, Schema } from "effect" +import type { HttpClientResponse } from "effect/unstable/http" +import { MediaProtocol } from "../route/media-protocol.js" +import { MediaRoute } from "../route/media.js" +import { mergeJsonRecords, type OpenString } from "../schema/index.js" +import { TranscriptionModel, TranscriptionResponse, type TranscriptionRequestFor } from "../transcription.js" +import { mediaTypeExtension } from "../utils/media-type.js" +import { ProviderShared, optionalNull } from "./shared.js" +import { MediaInput } from "./utils/media-input.js" +import { SpeakerTurns } from "./utils/speaker-turns.js" + +const route = MediaProtocol.identity({ + id: "elevenlabs-transcription", + name: "ElevenLabs Transcription", + provider: "elevenlabs", +}) +export const DEFAULT_BASE_URL = "https://api.elevenlabs.io" +export const PATH = "/v1/speech-to-text" + +// --------------------------------------------------------------------------- +// 1. Public model input +// --------------------------------------------------------------------------- + +export type ElevenLabsTranscriptionOptions = { + readonly tag_audio_events?: boolean + readonly timestamps_granularity?: OpenString<"none" | "word" | "character"> + readonly diarization_threshold?: number + readonly file_format?: OpenString<"pcm_s16le_16" | "other"> + readonly temperature?: number + readonly seed?: number + readonly keyterms?: ReadonlyArray + readonly no_verbatim?: boolean + readonly detect_speaker_roles?: boolean + readonly use_speaker_library?: boolean + readonly entity_detection?: string | ReadonlyArray + readonly entity_redaction?: string | ReadonlyArray + readonly entity_redaction_mode?: OpenString<"redacted" | "entity_type" | "enumerated_entity_type"> +} & Record + +export type Request = TranscriptionRequestFor + +// --------------------------------------------------------------------------- +// 2. Response schema +// --------------------------------------------------------------------------- + +/** `type` is `word`, `spacing` (the whitespace between words), or `audio_event` (`(laughter)`). */ +const Token = Schema.Struct({ + text: Schema.String, + type: Schema.String, + start: optionalNull(Schema.Number), + end: optionalNull(Schema.Number), + speaker_id: optionalNull(Schema.String), + logprob: optionalNull(Schema.Number), +}) +type Token = Schema.Schema.Type + +const Transcript = Schema.Struct({ + language_code: optionalNull(Schema.String), + text: Schema.String, + words: optionalNull(Schema.Array(Token)), + transcription_id: optionalNull(Schema.String), + audio_duration_secs: optionalNull(Schema.Number), +}) + +// --------------------------------------------------------------------------- +// 5. Request body construction +// --------------------------------------------------------------------------- + +/** Speaker turns are the only segments ElevenLabs can produce, and `num_speakers` only applies to diarization. */ +const diarizes = (request: Request) => + request.diarize === true || request.timestamps === "segment" || request.speakers !== undefined + +const RESERVED_FORM_FIELDS = new Set([ + "file", + "cloud_storage_url", + "source_url", + "model_id", + "language_code", + "diarize", + "num_speakers", +]) + +const validate = (request: Request, overlay: Record) => { + // Webhook requests return 202 with no transcript; the result arrives at a configured webhook instead. + if (overlay.webhook === true) + return Effect.fail(route.unsupported("transcription.webhook", `${route.name} does not deliver to webhooks`)) + // Separate multichannel output replaces the transcript with one transcript per channel. + if (overlay.use_multi_channel === true && overlay.multichannel_output_style !== "combined") + return Effect.fail( + route.unsupported( + "transcription.multichannel", + `${route.name} returns a single transcript; set multichannel_output_style: "combined" to merge channels`, + ), + ) + if (overlay.timestamps_granularity === "none" && (request.timestamps === "word" || diarizes(request))) + return Effect.fail( + route.unsupported( + "media.timestamps", + `${route.name} cannot return word timestamps or speaker turns with timestamps_granularity: "none"`, + ), + ) + return Effect.void +} + +const fromRequest = Effect.fn("ElevenLabsTranscription.fromRequest")(function* (request: Request) { + const overlay = mergeJsonRecords(request.providerOptions, request.http?.body) ?? {} + yield* validate(request, overlay) + const form = new FormData() + const url = ProviderShared.mediaUrl(request.audio) + if (url === undefined) { + const extension = mediaTypeExtension(request.audio.mediaType) + const audio = yield* MediaInput.inlineBytes(route.id, request.audio) + form.append( + "file", + MediaInput.blob(audio, request.audio.mediaType), + extension === undefined ? "audio" : `audio.${extension}`, + ) + } + MediaInput.appendFields( + form, + { + model_id: request.model.id, + // `cloud_storage_url` is deprecated in favor of `source_url`, which accepts any hosted audio or video URL. + source_url: url, + language_code: request.language, + diarize: diarizes(request) ? true : undefined, + num_speakers: request.speakers, + }, + { overlay, reserved: RESERVED_FORM_FIELDS, repeatArrays: "key" }, + ) + return MediaProtocol.multipart(form) +}) + +// --------------------------------------------------------------------------- +// 6. Response decoding +// --------------------------------------------------------------------------- + +const decodeTranscript = route.decodeJson(Transcript) + +type TimedWord = Token & { readonly start: number; readonly end: number } + +const isTimedWord = (token: Token): token is TimedWord => + token.type === "word" && typeof token.start === "number" && typeof token.end === "number" + +/** Turn text keeps the provider's own spacing tokens, so languages written without spaces are not re-spaced. */ +const speakerTurns = (tokens: ReadonlyArray) => + SpeakerTurns.group( + tokens.filter((token) => token.type === "word" || token.type === "spacing"), + (token) => token.speaker_id, + ).flatMap((turn) => { + const words = turn.filter(isTimedWord) + if (words.length === 0) return [] + return [ + { + text: turn + .map((token) => token.text) + .join("") + .trim(), + startSeconds: words[0].start, + endSeconds: words[words.length - 1].end, + speaker: turn[0].speaker_id ?? undefined, + }, + ] + }) + +const decodeResponse = Effect.fn("ElevenLabsTranscription.decodeResponse")(function* ( + response: HttpClientResponse.HttpClientResponse, + context: MediaProtocol.DecodeContext, +) { + const output = yield* decodeTranscript(response) + const transcript = output.value + const tokens = transcript.words ?? [] + const duration = transcript.audio_duration_secs ?? undefined + const transcriptionID = transcript.transcription_id ?? undefined + return new TranscriptionResponse({ + text: transcript.text, + segments: diarizes(context.request) ? speakerTurns(tokens) : undefined, + words: tokens.filter(isTimedWord).map((word) => ({ + text: word.text, + startSeconds: word.start, + endSeconds: word.end, + speaker: word.speaker_id ?? undefined, + confidence: typeof word.logprob === "number" ? Math.exp(word.logprob) : undefined, + })), + language: transcript.language_code?.toLowerCase(), + durationSeconds: duration, + usage: duration === undefined ? undefined : { type: "seconds", seconds: duration }, + providerMetadata: transcriptionID === undefined ? undefined : { elevenlabs: { transcriptionId: transcriptionID } }, + }) +}) + +// --------------------------------------------------------------------------- +// 7. Protocol and route +// --------------------------------------------------------------------------- + +export const protocol = MediaProtocol.inline(route, { + unsupported: ["prompt"], + body: { from: fromRequest }, + response: { decode: decodeResponse }, +}) + +export const model = (input: MediaRoute.ModelInput) => + TranscriptionModel.fromRoute( + { protocol, baseURL: DEFAULT_BASE_URL, path: PATH }, + input, + ) + +export const ElevenLabsTranscription = { + protocol, + model, +} as const diff --git a/packages/ai/src/protocols/openai-transcription.ts b/packages/ai/src/protocols/openai-transcription.ts index 9d267842ae5..0890ce836dd 100644 --- a/packages/ai/src/protocols/openai-transcription.ts +++ b/packages/ai/src/protocols/openai-transcription.ts @@ -193,7 +193,7 @@ const fromRequest = Effect.fn("OpenAITranscription.fromRequest")(function* (requ { overlay: mergeJsonRecords(request.providerOptions, request.http?.body), reserved: RESERVED_FORM_FIELDS, - repeatArrays: true, + repeatArrays: "key[]", }, ) return MediaProtocol.multipart(form) diff --git a/packages/ai/src/protocols/utils/media-input.ts b/packages/ai/src/protocols/utils/media-input.ts index e397d172bc2..2116b8cb542 100644 --- a/packages/ai/src/protocols/utils/media-input.ts +++ b/packages/ai/src/protocols/utils/media-input.ts @@ -71,8 +71,9 @@ export const imageOutput = ( } /** - * Append multipart text fields: strings as-is, other values as JSON, or arrays as repeated `key[]` parts with - * `repeatArrays`. `overlay` keys in `reserved` are dropped so `http.body` cannot replace route-owned fields. + * Append multipart text fields: strings as-is, other values as JSON, or scalar arrays as one part per item with + * `repeatArrays`, named `key[]` or `key`. `overlay` keys in `reserved` are dropped so `http.body` cannot replace + * route-owned fields. */ export const appendFields = ( form: FormData, @@ -80,13 +81,13 @@ export const appendFields = ( options: { readonly overlay?: Record readonly reserved: ReadonlySet - readonly repeatArrays?: true + readonly repeatArrays?: "key[]" | "key" }, ) => { const overlay = Object.entries(options.overlay ?? {}).filter(([key]) => !options.reserved.has(key)) Object.entries(mergeJsonRecords(fields, Object.fromEntries(overlay)) ?? {}).forEach(([key, value]) => { - if (Array.isArray(value) && options.repeatArrays) - return value.forEach((item) => form.append(`${key}[]`, String(item))) + if (Array.isArray(value) && value.every(isScalar) && options.repeatArrays !== undefined) + return value.forEach((item) => form.append(options.repeatArrays === "key[]" ? `${key}[]` : key, String(item))) form.append(key, typeof value === "string" ? value : encodeJson(value)) }) } diff --git a/packages/ai/src/protocols/utils/speaker-turns.ts b/packages/ai/src/protocols/utils/speaker-turns.ts new file mode 100644 index 00000000000..512a7758bbe --- /dev/null +++ b/packages/ai/src/protocols/utils/speaker-turns.ts @@ -0,0 +1,10 @@ +/** Split an ordered token list into runs of consecutive tokens with the same speaker. */ +export const group = (items: ReadonlyArray, speaker: (item: Item) => unknown) => + items.reduce>>((turns, item) => { + const last = turns.at(-1) + if (last === undefined || speaker(last[0]) !== speaker(item)) return [...turns, [item]] + last.push(item) + return turns + }, []) + +export * as SpeakerTurns from "./speaker-turns.js" diff --git a/packages/ai/src/providers/elevenlabs.ts b/packages/ai/src/providers/elevenlabs.ts index 4f0976ed557..410e246002f 100644 --- a/packages/ai/src/providers/elevenlabs.ts +++ b/packages/ai/src/providers/elevenlabs.ts @@ -3,8 +3,10 @@ import type { ProviderAuthOption } from "../route/auth-options.js" import { MediaRoute } from "../route/media.js" import { type HttpOptions, ProviderID, type ModelID } from "../schema/index.js" import { ElevenLabsSpeech } from "../protocols/elevenlabs-speech.js" +import { ElevenLabsTranscription } from "../protocols/elevenlabs-transcription.js" export type { ElevenLabsOutputFormat, ElevenLabsSpeechOptions } from "../protocols/elevenlabs-speech.js" +export type { ElevenLabsTranscriptionOptions } from "../protocols/elevenlabs-transcription.js" export const id = ProviderID.make("elevenlabs") @@ -24,12 +26,15 @@ const auth = (options: ProviderAuthOption<"optional">) => { export const configure = (input: Config = {}) => { const media = MediaRoute.deployment(input, auth(input)) const speech = (modelID: string | ModelID) => ElevenLabsSpeech.model({ ...media, id: modelID }) + const transcription = (modelID: string | ModelID) => ElevenLabsTranscription.model({ ...media, id: modelID }) return { id, speech, + transcription, configure, } } export const provider = configure() export const speech = provider.speech +export const transcription = provider.transcription diff --git a/packages/ai/src/transcription.ts b/packages/ai/src/transcription.ts index 9a5e01b1be5..1302d72fccd 100644 --- a/packages/ai/src/transcription.ts +++ b/packages/ai/src/transcription.ts @@ -103,7 +103,7 @@ export type TranscriptionRequestInput { expect(Google.configure({ apiKey: "fixture" }).transcription("gemini-3.5-transcribe").route.kind).toBe("stream") expect(Deepgram.configure({ apiKey: "fixture" }).transcription("nova-3").route.kind).toBe("inline") expect(AssemblyAI.configure({ apiKey: "fixture" }).transcription("universal-3-5-pro").route.kind).toBe("queued") + expect(ElevenLabs.configure({ apiKey: "fixture" }).transcription("scribe_v2").route.id).toBe( + "elevenlabs-transcription", + ) + expect(ElevenLabs.configure({ apiKey: "fixture" }).transcription("scribe_v2").route.kind).toBe("inline") + expect(ElevenLabs.provider.transcription).toBe(ElevenLabs.transcription) }) test("protocol barrels expose supported low-level routes", () => { diff --git a/packages/ai/test/fixtures/recordings/elevenlabs-transcription/groups-diarized-words-into-speaker-turns.json b/packages/ai/test/fixtures/recordings/elevenlabs-transcription/groups-diarized-words-into-speaker-turns.json new file mode 100644 index 00000000000..4463ec5a66a --- /dev/null +++ b/packages/ai/test/fixtures/recordings/elevenlabs-transcription/groups-diarized-words-into-speaker-turns.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:elevenlabs-transcription", + "provider:elevenlabs", + "protocol:elevenlabs-transcription" + ], + "name": "elevenlabs-transcription/groups-diarized-words-into-speaker-turns", + "recordedAt": "2026-09-27T09:35:28.265Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.elevenlabs.io/v1/speech-to-text", + "headers": { + "content-type": "multipart/form-data; boundary=----WebKitFormBoundary356bdc14864a477dbacbfcf60d1ecceb" + }, + "body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model_id\"\r\n\r\nscribe_v2\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"diarize\"\r\n\r\ntrue\r\n--BOUNDARY--\r\n" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"language_code\":\"eng\",\"language_probability\":0.9495430588722229,\"text\":\"Did the release ship? Yes, it shipped this morning\",\"words\":[{\"text\":\"Did\",\"start\":0.34,\"end\":0.44,\"type\":\"word\",\"speaker_id\":\"speaker_0\",\"logprob\":-1.7881377516459906e-6},{\"text\":\" \",\"start\":0.44,\"end\":0.48,\"type\":\"spacing\",\"speaker_id\":\"speaker_0\",\"logprob\":-1.1920928244535389e-7},{\"text\":\"the\",\"start\":0.48,\"end\":0.56,\"type\":\"word\",\"speaker_id\":\"speaker_0\",\"logprob\":-1.1920928244535389e-7},{\"text\":\" \",\"start\":0.56,\"end\":0.6,\"type\":\"spacing\",\"speaker_id\":\"speaker_0\",\"logprob\":-7.152531907195225e-6},{\"text\":\"release\",\"start\":0.6,\"end\":0.92,\"type\":\"word\",\"speaker_id\":\"speaker_0\",\"logprob\":-7.152531907195225e-6},{\"text\":\" \",\"start\":0.92,\"end\":0.94,\"type\":\"spacing\",\"speaker_id\":\"speaker_0\",\"logprob\":-8.344646857949556e-7},{\"text\":\"ship?\",\"start\":0.94,\"end\":1.26,\"type\":\"word\",\"speaker_id\":\"speaker_0\",\"logprob\":-7.414704032271402e-6},{\"text\":\" \",\"start\":1.26,\"end\":1.26,\"type\":\"spacing\",\"speaker_id\":\"speaker_0\",\"logprob\":-0.0009363081189803779},{\"text\":\"Yes,\",\"start\":1.68,\"end\":2.02,\"type\":\"word\",\"speaker_id\":\"speaker_1\",\"logprob\":-0.003542040009030245},{\"text\":\" \",\"start\":2.02,\"end\":2.48,\"type\":\"spacing\",\"speaker_id\":\"speaker_1\",\"logprob\":-3.933898824470816e-6},{\"text\":\"it\",\"start\":2.48,\"end\":2.62,\"type\":\"word\",\"speaker_id\":\"speaker_1\",\"logprob\":-3.933898824470816e-6},{\"text\":\" \",\"start\":2.62,\"end\":2.64,\"type\":\"spacing\",\"speaker_id\":\"speaker_1\",\"logprob\":-0.000013589766240329482},{\"text\":\"shipped\",\"start\":2.66,\"end\":2.9,\"type\":\"word\",\"speaker_id\":\"speaker_1\",\"logprob\":-0.000013589766240329482},{\"text\":\" \",\"start\":2.9,\"end\":2.94,\"type\":\"spacing\",\"speaker_id\":\"speaker_1\",\"logprob\":0.0},{\"text\":\"this\",\"start\":2.94,\"end\":3.12,\"type\":\"word\",\"speaker_id\":\"speaker_1\",\"logprob\":0.0},{\"text\":\" \",\"start\":3.12,\"end\":3.18,\"type\":\"spacing\",\"speaker_id\":\"speaker_1\",\"logprob\":-3.576278118089249e-7},{\"text\":\"morning\",\"start\":3.18,\"end\":3.5,\"type\":\"word\",\"speaker_id\":\"speaker_1\",\"logprob\":-3.576278118089249e-7}],\"transcription_id\":\"cs3I2282TH8hjw12brNg\",\"audio_duration_secs\":3.5526875}" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/elevenlabs-transcription/transcribes-audio-with-word-timestamps.json b/packages/ai/test/fixtures/recordings/elevenlabs-transcription/transcribes-audio-with-word-timestamps.json new file mode 100644 index 00000000000..5a6db64b9b1 --- /dev/null +++ b/packages/ai/test/fixtures/recordings/elevenlabs-transcription/transcribes-audio-with-word-timestamps.json @@ -0,0 +1,50 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:elevenlabs-transcription", + "provider:elevenlabs", + "protocol:elevenlabs-transcription" + ], + "name": "elevenlabs-transcription/transcribes-audio-with-word-timestamps", + "recordedAt": "2026-09-27T09:35:27.686Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.elevenlabs.io/v1/speech-to-text", + "headers": { + "content-type": "multipart/form-data; boundary=----WebKitFormBoundarye2be7b31e94441bbbeb35a9c890a9d74" + }, + "body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model_id\"\r\n\r\nscribe_v2\r\n--BOUNDARY--\r\n" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"language_code\":\"eng\",\"language_probability\":0.6618340611457825,\"text\":\"Hello from OpenCode\",\"words\":[{\"text\":\"Hello\",\"start\":0.4,\"end\":0.66,\"type\":\"word\",\"logprob\":-0.000014781842764932662},{\"text\":\" \",\"start\":0.66,\"end\":0.74,\"type\":\"spacing\",\"logprob\":-3.814689989667386e-6},{\"text\":\"from\",\"start\":0.74,\"end\":0.84,\"type\":\"word\",\"logprob\":-3.814689989667386e-6},{\"text\":\" \",\"start\":0.84,\"end\":0.9,\"type\":\"spacing\",\"logprob\":-0.018268775194883347},{\"text\":\"OpenCode\",\"start\":0.9,\"end\":1.44,\"type\":\"word\",\"logprob\":-0.1251817401498556}],\"transcription_id\":\"D4VfnANM2ArCHTujIb9q\",\"audio_duration_secs\":1.54125}" + } + }, + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.elevenlabs.io/v1/speech-to-text", + "headers": { + "content-type": "multipart/form-data; boundary=----WebKitFormBoundaryfb80d0e44d9e44d299416ed546a04056" + }, + "body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model_id\"\r\n\r\nscribe_v2\r\n--BOUNDARY--\r\n" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"language_code\":\"eng\",\"language_probability\":0.6618340611457825,\"text\":\"Hello from OpenCode\",\"words\":[{\"text\":\"Hello\",\"start\":0.4,\"end\":0.66,\"type\":\"word\",\"logprob\":-0.000023007127310847864},{\"text\":\" \",\"start\":0.66,\"end\":0.74,\"type\":\"spacing\",\"logprob\":-2.3841830625315197e-6},{\"text\":\"from\",\"start\":0.74,\"end\":0.84,\"type\":\"word\",\"logprob\":-2.3841830625315197e-6},{\"text\":\" \",\"start\":0.84,\"end\":0.9,\"type\":\"spacing\",\"logprob\":-0.008306833915412426},{\"text\":\"OpenCode\",\"start\":0.9,\"end\":1.44,\"type\":\"word\",\"logprob\":-0.1075385226868093}],\"transcription_id\":\"SkYplzfq1DW8Ae3bWnoy\",\"audio_duration_secs\":1.54125}" + } + } + ] +} diff --git a/packages/ai/test/provider/elevenlabs-transcription.recorded.test.ts b/packages/ai/test/provider/elevenlabs-transcription.recorded.test.ts new file mode 100644 index 00000000000..60962226a12 --- /dev/null +++ b/packages/ai/test/provider/elevenlabs-transcription.recorded.test.ts @@ -0,0 +1,50 @@ +import { describe, expect } from "bun:test" +import { Effect, Stream } from "effect" +import { Transcription } from "../../src/index.js" +import { ElevenLabs } from "../../src/providers.js" +import { recordedTests } from "../recorded-test.js" +import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js" + +const model = ElevenLabs.configure({ apiKey: process.env.ELEVENLABS_API_KEY ?? "fixture" }).transcription("scribe_v2") + +const recorded = recordedTests({ + prefix: "elevenlabs-transcription", + provider: "elevenlabs", + protocol: "elevenlabs-transcription", + requires: ["ELEVENLABS_API_KEY"], + options: audioRecording, +}) + +describe("ElevenLabs Transcription recorded", () => { + recorded.effect("transcribes audio with word timestamps", () => + Effect.gen(function* () { + const request = Transcription.request({ model, audio: yield* audio, timestamps: "word" }) + const response = yield* Transcription.generate(request) + + expect(response.text).toMatch(TRANSCRIPT) + expect(response.words?.map((word) => word.text)).toEqual(["Hello", "from", "OpenCode"]) + expect(response.words?.every((word) => word.speaker === undefined && (word.confidence ?? 0) > 0)).toBe(true) + expect(response.segments).toBeUndefined() + expect(response.language).toBe("eng") + expect(response.durationSeconds).toBeGreaterThan(0) + expect(response.usage).toEqual({ type: "seconds", seconds: response.durationSeconds }) + expect(response.providerMetadata?.elevenlabs?.transcriptionId).toEqual(expect.any(String)) + + const events = Array.from(yield* Stream.runCollect(Transcription.stream(request))) + expect(events.map((event) => event.type)).toEqual(["finish"]) + }), + ) + + recorded.effect("groups diarized words into speaker turns", () => + Effect.gen(function* () { + const response = yield* Transcription.generate({ model, audio: yield* dialog, diarize: true }) + + expect(response.segments?.map((segment) => segment.speaker)).toEqual(["speaker_0", "speaker_1"]) + expect(response.segments?.[0].text).toMatch(/^Did the release ship\?$/) + expect(response.segments?.[1].text).toMatch(/^Yes, it shipped this morning\.?$/) + expect(response.segments?.map((segment) => segment.text).join(" ")).toBe(response.text) + expect(response.words?.some((word) => word.text.trim() === "")).toBe(false) + expect(new Set(response.words?.map((word) => word.speaker))).toEqual(new Set(["speaker_0", "speaker_1"])) + }), + ) +}) diff --git a/packages/ai/test/transcription.test.ts b/packages/ai/test/transcription.test.ts index 71ac90434d5..8a980afc92a 100644 --- a/packages/ai/test/transcription.test.ts +++ b/packages/ai/test/transcription.test.ts @@ -3,7 +3,7 @@ import { Effect, Fiber, Layer, Stream } from "effect" import * as TestClock from "effect/testing/TestClock" import { HttpClientRequest } from "effect/unstable/http" import { Media, Transcription, TranscriptionClient, type TranscriptionEvent } from "../src/index.js" -import { AssemblyAI, Deepgram, Google, OpenAI } from "../src/providers.js" +import { AssemblyAI, Deepgram, ElevenLabs, Google, OpenAI } from "../src/providers.js" import { it } from "./lib/effect.js" import { dynamicResponse, json, observe, type Call } from "./lib/http.js" import { sseEvents } from "./lib/sse.js" @@ -35,6 +35,9 @@ const formFields = (call: Call) => const assemblyai = AssemblyAI.configure({ apiKey: "aai-key", baseURL: "https://assemblyai.test" }).transcription( "universal-3-5-pro", ) +const elevenlabs = ElevenLabs.configure({ apiKey: "test", baseURL: "https://elevenlabs.test" }).transcription( + "scribe_v2", +) describe("Transcription", () => { it.effect("rejects what a route cannot honor before sending anything", () => @@ -459,6 +462,115 @@ describe("Transcription", () => { }), ) + it.effect("rejects ElevenLabs prompts, webhooks, per-channel transcripts, and untimed diarization", () => + Effect.gen(function* () { + const errors = yield* Effect.all( + [ + Transcription.generate({ model: elevenlabs, audio, prompt: "OpenCode" }), + Transcription.generate({ model: elevenlabs, audio, providerOptions: { webhook: true } }), + Transcription.generate({ model: elevenlabs, audio, http: { body: { use_multi_channel: true } } }), + Transcription.generate({ + model: elevenlabs, + audio, + diarize: true, + providerOptions: { timestamps_granularity: "none" }, + }), + Transcription.generate({ + model: elevenlabs, + audio: Media.ref("file_1", { provider: "elevenlabs", mediaType: "audio/mpeg" }), + }), + ].map((effect) => Effect.flip(effect)), + ) + expect(errors.map((error) => [error.reason._tag, "operation" in error.reason && error.reason.operation])).toEqual( + [ + ["UnsupportedOperation", "media.prompt"], + ["UnsupportedOperation", "transcription.webhook"], + ["UnsupportedOperation", "transcription.multichannel"], + ["UnsupportedOperation", "media.timestamps"], + ["InvalidRequest", false], + ], + ) + }).pipe(Effect.provide(layer(() => Effect.die("an unsupported request reached the network")))), + ) + + it.effect("sends ElevenLabs URL audio as source_url and groups diarized words into speaker turns", () => + Effect.gen(function* () { + const calls: Array = [] + const token = (text: string, type: string, start: number, end: number, speaker_id?: string) => ({ + text, + type, + start, + end, + speaker_id, + logprob: 0, + }) + const response = yield* Transcription.generate({ + model: elevenlabs, + audio: Media.url("https://a.test/call.mp3"), + language: "en", + speakers: 2, + providerOptions: { keyterms: ["OpenCode", "Scribe"], tag_audio_events: true, diarize: false }, + }).pipe( + Effect.provide( + layer((input) => + observe(calls, input).pipe( + Effect.as( + json(input, { + language_code: "ENG", + text: "Ready? (laughs) Yes. Go", + words: [ + token("Ready?", "word", 0, 0.5, "speaker_0"), + token(" ", "spacing", 0.5, 0.6, "speaker_0"), + token("(laughs)", "audio_event", 0.6, 1, "speaker_0"), + token(" ", "spacing", 1, 1.1, "speaker_0"), + token("Yes.", "word", 1.2, 1.5, "speaker_1"), + token(" ", "spacing", 1.5, 1.6, "speaker_1"), + token("Go", "word", 1.6, 1.9, "speaker_0"), + ], + transcription_id: "tr_1", + audio_duration_secs: 2, + }), + ), + ), + ), + ), + ) + + // `observe` re-encodes the FormData with a new boundary, so read the boundary from the sent body. + const boundary = /^--(\S+)/.exec(calls[0].body)?.[1] + const form = yield* Effect.promise(() => + new Response(calls[0].body, { + headers: { "content-type": `multipart/form-data; boundary=${boundary}` }, + }).formData(), + ) + expect(calls[0].url).toBe("https://elevenlabs.test/v1/speech-to-text") + expect(calls[0].headers.get("xi-api-key")).toBe("test") + expect(Array.from(form.entries())).toEqual([ + ["model_id", "scribe_v2"], + ["source_url", "https://a.test/call.mp3"], + ["language_code", "en"], + ["diarize", "true"], + ["num_speakers", "2"], + ["keyterms", "OpenCode"], + ["keyterms", "Scribe"], + ["tag_audio_events", "true"], + ]) + expect(response.segments).toEqual([ + { text: "Ready?", startSeconds: 0, endSeconds: 0.5, speaker: "speaker_0" }, + { text: "Yes.", startSeconds: 1.2, endSeconds: 1.5, speaker: "speaker_1" }, + { text: "Go", startSeconds: 1.6, endSeconds: 1.9, speaker: "speaker_0" }, + ]) + expect(response.words?.map((word) => [word.text, word.speaker, word.confidence])).toEqual([ + ["Ready?", "speaker_0", 1], + ["Yes.", "speaker_1", 1], + ["Go", "speaker_0", 1], + ]) + expect(response.language).toBe("eng") + expect(response.usage).toEqual({ type: "seconds", seconds: 2 }) + expect(response.providerMetadata).toEqual({ elevenlabs: { transcriptionId: "tr_1" } }) + }), + ) + it.effect("rejects reading an AssemblyAI result before the transcript finishes", () => Effect.gen(function* () { const generation = yield* Transcription.resume(assemblyai, { transcriptID: "tr_1" })