diff --git a/packages/ai/AGENTS.md b/packages/ai/AGENTS.md index 52dd8a33449..b21c89a6f95 100644 --- a/packages/ai/AGENTS.md +++ b/packages/ai/AGENTS.md @@ -100,7 +100,9 @@ Media does not fit the SSE-frames-to-event-state-machine LLM route. `MediaRoute. `MediaProtocol.queued` is the submit-then-poll kind every video route uses: `start` (body + decode into `{ token, snapshot }`), `status`, `result`, and optional `cancel`, each addressed by a route-owned `token` whose `Schema.Codec` makes it serializable. `MediaRoute.inline` and `MediaRoute.queued` compose the two kinds with `Endpoint` and `Auth`; the queued route decodes the token once at the boundary (`start` output or `resume` input) and closes over it in a token-free `GenerationRoute` (`status`/`result`/`cancel` are plain Effects), so `Generation` never sees the token's shape and only carries the encoded JSON for persistence. Polls reuse the route's auth and deployment headers plus the request's `http` overlay after `start`, and resolve relative paths against the route base URL (provider-issued absolute URLs such as fal's `status_url` pass through). `result` is always its own GET even when the provider returns output inside the status document, so `Generation.await` behaves the same after `start` and after `resume`. `PollContext.auth` carries only what `Auth` added so protocols can hand download credentials to output assets as transient `Media.Asset.headers` (Veo) — never part of `source` or JSON. Status strings map through a per-protocol `STATUS` table via `MediaProtocol.status`; terminal generations without output fail through `output.ended` / `output.contentPolicy` with the provider document on `reason.body`. `Generation.AwaitOptions` (`{ poll?: Poll }`) is the one options type for `await`, `events`, `Video.generate`, and `Video.stream`. -`MediaProtocol.stream` is the incremental kind every speech route uses, with the same discipline as LLM protocols. `MediaRoute.stream` submits the caller's request as `MediaProtocol.Addressed` (`{ ...request, mode }`, `mode: "generate" | "stream"`), so one provider stays one protocol: `body.from`, the endpoint path, and `frames` read `request.mode` to pick the body, path, and framing. `frames(bytes, context)` returns frames — `Framing.sse`, `Framing.lines`, `Framing.document` (a single-document response shaped like a streamed record), or the raw `bytes` for chunked audio. `initial()` is fresh per-response parser state; `step` folds each frame into it and emits modality events; `finish(state, context)` runs once after the last frame with the request, body, and observed `http` (header-only usage lives there) and emits exactly one terminal event or fails with `MediaProtocol.incomplete`. Keep parser state to real accumulators and derive anything the request or body determines in `finish`. `generate` runs the same stream and folds it with the modality's `collect`. Request-derived URL parameters go on the JSON body's `query`, applied before route and caller `http.query`. Decode frames with `MediaProtocol.decodeFrame` and raise stream-time failures with `MediaProtocol.frameError` (the frame stays on `reason.body`); protocols never thread HTTP context, because the route fills `reason.http` on stream errors that lack it. Speech protocols share `protocols/utils/speech-stream.ts` for deltas, timestamps, voice ids, PCM and container descriptions, and the terminal asset. +`MediaProtocol.stream` is the incremental kind every speech route uses, with the same discipline as LLM protocols. `MediaRoute.stream` submits the caller's request as `MediaProtocol.Addressed` (`{ ...request, mode }`, `mode: "generate" | "stream"`), so one provider stays one protocol: `body.from`, the endpoint path, and `frames` read `request.mode` to pick the body, path, and framing. `frames(bytes, context)` returns frames — `Framing.sse`, `Framing.lines`, `Framing.document` (a single-document response shaped like a streamed record), or the raw `bytes` for chunked audio. `initial()` is fresh per-response parser state; `step` folds each frame into it and emits modality events; `finish(state, context)` runs once after the last frame with the request, body, and observed `http` (header-only usage lives there) and emits exactly one terminal event or fails with `MediaProtocol.incomplete`. Keep parser state to real accumulators and derive anything the request or body determines in `finish`. `generate` runs the same stream and folds it with the modality's `collect`. Request-derived URL parameters go on the body's `query` (array values repeat the parameter), applied before route and caller `http.query`. Decode frames with `MediaProtocol.decodeFrame` and raise stream-time failures with `MediaProtocol.frameError` (the frame stays on `reason.body`); protocols never thread HTTP context, because the route fills `reason.http` on stream errors that lack it. Speech protocols share `protocols/utils/speech-stream.ts` for deltas, timestamps, voice ids, PCM and container descriptions, and the terminal asset. + +Transcription uses all three kinds (OpenAI and Gemini stream, Deepgram is inline, AssemblyAI is queued): every route carries its `kind` and `TranscriptionClient` dispatches on it. Bodies are `json`, `multipart`, or `binary` (a raw upload), and a queued protocol that must upload media before submitting implements `start.prepare` (`MediaProtocol.Prepare`; AssemblyAI `/v2/upload`). ### URL Construction diff --git a/packages/ai/README.md b/packages/ai/README.md index 564492ece77..27c43b16bb5 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -766,6 +766,69 @@ for await (const event of ai.speech.stream({ model, text: "Hello from OpenCode." } ``` +## Transcription + +Transcription (speech-to-text) is the one modality whose providers use every route kind: OpenAI and Gemini stream, +Deepgram answers inline, and AssemblyAI is queued. `Transcription.generate` and `Transcription.stream` work on all of +them; `Transcription.start` / `resume` return a `Generation` on queued routes and fail with `UnsupportedOperation` +elsewhere. Models come from `.transcription(...)` selectors on the `OpenAI`, `Google`, `Deepgram`, and `AssemblyAI` +facades. Common fields (`language`, `prompt`, `timestamps: "none" | "segment" | "word"`, `diarize`, `speakers`) lower +natively or fail with a typed `AIError` before any network call; a route may return more than asked. + +```ts +import { Media, Transcription, TranscriptionEvent } from "@opencode/ai" +import { AssemblyAI, Deepgram, OpenAI } from "@opencode/ai/providers" + +const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY }) + +const program = Effect.gen(function* () { + const audio = yield* Media.file("./call.mp3") + + // Speaker-labelled segments; labels are provider-native strings ("A", "0", "spk:0"). + const response = yield* Transcription.generate({ + model: Deepgram.configure({ apiKey }).transcription("nova-3"), + audio, + diarize: true, + timestamps: "word", + }) + response.text // "Hello from OpenCode." + response.segments // [{ text, startSeconds, endSeconds, speaker: "0" }] + response.words // [{ text, startSeconds, endSeconds, speaker, confidence }] + response.language // the provider's own value, lowercased ("en", "english", "en_us") + + // Text deltas as the model transcribes, then one finish carrying the whole transcript. + yield* Transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio }).pipe( + Stream.tap((event) => (TranscriptionEvent.is.textDelta(event) ? Console.log(event.delta) : Effect.void)), + Stream.runDrain, + ) + + // Queued: persist the token, resume from another process, and await. + const model = AssemblyAI.configure({ apiKey }).transcription("universal-3-5-pro") + const generation = yield* Transcription.start({ model, audio }) + const resumed = yield* Transcription.resume(model, JSON.parse(JSON.stringify(generation.token))) + const transcript = yield* resumed.await({ poll: { interval: "3 seconds" } }) +}) +``` + +Inline routes emit only `finish` from `stream` (no faked deltas); queued routes emit `generation-queued` / +`generation-progress` before it. `TranscriptionClient.layer` needs `RequestExecutor.Service`. + +Provider notes: + +- **OpenAI** takes inline audio only; `diarize` needs `gpt-4o-transcribe-diarize`, timestamps need `whisper-1`, and `whisper-1` does not stream. +- **Gemini** needs a transcribe model (`gemini-3.5-transcribe`); `prompt` and `speakers` fail typed. +- **Deepgram** detects the language unless `language` is set; vocabulary goes in `providerOptions.keyterm`. +- **AssemblyAI** uploads inline audio before submitting and is the only route that accepts `speakers`. + +The promise client mirrors the Effect API: + +```ts +const text = (await ai.transcription.generate({ model, audio })).text +for await (const event of ai.transcription.stream({ model, audio })) if (event.type === "text-delta") write(event.delta) +const generation = await ai.transcription.start({ model: assemblyai, audio }) +const transcript = await generation.await({ poll: { interval: 3_000 } }) +``` + ## Public API - **`LLM.request({...})`** — build a provider-neutral `LLMRequest`. Accepts ergonomic inputs (`system: string`, `prompt: string`) that normalize into the canonical Schema classes. @@ -778,7 +841,8 @@ for await (const event of ai.speech.stream({ model, text: "Hello from OpenCode." - **`Media`** — the shared asset type (`Media.Asset`, `Media.Source`) and constructors used by messages, tool results, and media requests. - **`Generation`** — provider-neutral handle for an in-flight media generation (`await`, `refresh`, `cancel`, `events`) used by queued media routes. - **`Speech.request` / `Speech.generate` / `Speech.stream`** — text-to-speech through a provider-neutral request; `SpeechClient` is its Effect service and layer. -- **`@opencode/ai/promise`** — `AI.make({ layer? })` and a default `ai` client exposing `llm`, `image`, `video`, and `speech` as Promise / `AsyncIterable` APIs. +- **`Transcription.request` / `generate` / `stream` / `start` / `resume`** — speech-to-text over inline, streaming, and queued routes; `TranscriptionClient` is its Effect service and layer. +- **`@opencode/ai/promise`** — `AI.make({ layer? })` and a default `ai` client exposing `llm`, `image`, `video`, `speech`, and `transcription` as Promise / `AsyncIterable` APIs. ## Testing diff --git a/packages/ai/docs/media-design.md b/packages/ai/docs/media-design.md index 8a7a4a2559a..4d0fd121c7f 100644 --- a/packages/ai/docs/media-design.md +++ b/packages/ai/docs/media-design.md @@ -1,6 +1,6 @@ # Media generation in `@opencode/ai` — public API direction -Status: phases 1–2 implemented; phase 3 Speech implemented, Transcription pending; phases 4–5 proposal. +Status: phases 1–3 implemented (Speech and Transcription); phases 4–5 proposal. ## Goal @@ -249,23 +249,71 @@ Deferred: `Speech.session(...)` — input-streaming TTS where text arrives incre #### Transcription (STT) +Shipped as the second half of phase 3 (`src/transcription.ts`, `src/transcription-client.ts`, protocols +`openai-transcription`, `google-transcription`, `deepgram-transcription`, `assemblyai-transcription`; new `AssemblyAI` +facade). + ```ts const request = Transcription.request({ - model: openai.transcription("gpt-4o-transcribe"), - audio: Media.file("./call.wav"), - language: "en", - prompt: "Names: Shoubhit, OpenCode.", - timestamps: "word", // none | segment | word + model: openai.transcription("gpt-4o-transcribe-diarize"), + audio: yield* Media.file("./call.wav"), + language: "en", // provider-native passthrough + timestamps: "segment", // none | segment | word diarize: true, - providerOptions: { chunkingStrategy: "auto" }, + speakers: 2, // expected count, hint only (AssemblyAI) + providerOptions: { known_speaker_names: ["agent"] }, }) const response = yield* Transcription.generate(request) -response.text; response.segments; response.words; response.language; response.durationSeconds -yield* Transcription.stream(request) // Stream: text-delta | segment | finish +response.text; response.segments; response.words; response.language; response.durationSeconds; response.usage +yield* Transcription.stream(request) // Stream: generation-queued | generation-progress | text-delta | segment | finish +const generation = yield* Transcription.start(request) // queued routes only +yield* Transcription.resume(model, token) ``` -Realtime STT over WebSocket is the same future `session` shape as input-streaming TTS. +Transcription is the first modality whose providers span all three protocol kinds, and it needed no fourth kind. +Every `MediaRoute` now carries its `kind`; `TranscriptionRoute` is the union of the inline, stream, and queued routes; +`TranscriptionModel.fromRoute` is overloaded per protocol kind (arity picks the overload: ``, +``, ``) and composes through `MediaRoute.inline` / `stream` / `queued`; and +`TranscriptionClient` dispatches on `route.kind`. `generate` on a queued route is `start` then `await`; `stream` on an +inline route is the response as a single `finish`, and on a queued route it is the status observations followed by +`finish`. `start` / `resume` on a non-queued route fail with `UnsupportedOperation` (`transcription.start`). The +`finish` event carries the whole transcript (text, segments, words, language, duration, usage), so the stream route's +`collect` is just "take `finish`". + +The route layer gained a `binary` body with array-valued `query` (Deepgram) and `Queued.start.prepare` (AssemblyAI's +upload); `packages/ai/AGENTS.md` (Media Routes) describes both. + +Settled rules: + +- **Timestamps.** A granularity the selected route or model cannot produce fails as `UnsupportedOperation` + (`media.timestamps`), following Speech; a route that returns more than asked (Deepgram and AssemblyAI always return + words) is not stripped. Segments always carry start and end times: Gemini times each transcription part from its + word offsets, so segment timestamps and diarization also request word offsets there. +- **Diarization.** `diarize` means segments (and words, where the provider labels them) carry `speaker`. Labels are + provider-native strings — OpenAI `A` or a known speaker name, Deepgram `0`, Gemini `spk:0`, AssemblyAI `A` — with no + cross-provider speaker model. `speakers` is a hint; only AssemblyAI (`speakers_expected`) accepts it. +- **Language** is passed through (`language`, OpenAI `gpt-transcribe` `languages[]`, Gemini `languageCodes`, + AssemblyAI `language_code`). `response.language` is the provider's own value, lowercased but not normalized: an + ISO code on most routes, `english` from whisper-1, `en_us` from AssemblyAI. Deepgram and AssemblyAI assume English + unless asked to detect, so a missing `language` enables their detection. +- **Gemini** requires a transcribe model; other model ids fail with `UnsupportedOperation` before the call, because + general models ignore `audioTranscriptionConfig` and answer conversationally. Streamed chunks carry whole speaker + turns (one part per turn), which join with a space. +- **Streaming inline providers** emit only `finish`; deltas are never faked. +- **Units.** AssemblyAI milliseconds and Gemini protobuf durations (`"0.400s"`) are normalized to seconds at the + protocol boundary. + +| Provider | Kind | Audio input | `timestamps` | `diarize` | Unsupported | Usage | +|---|---|---|---|---|---|---| +| OpenAI | stream (`stream: true` in `stream` mode) | multipart `file` (inline only) | `whisper-1` (`verbose_json`); diarize model: `segment` | `gpt-4o-transcribe-diarize` (`diarized_json`) | `speakers`; `prompt` on the diarize model; streaming on `whisper-1` | `tokens` or `seconds` | +| Gemini | stream (`generateContent` / `streamGenerateContent`) | `inlineData` or Gemini Files `fileData` | `audioTranscriptionConfig.wordTimestamp` | `audioTranscriptionConfig.diarization` | `prompt`, `speakers` | `tokens` | +| Deepgram | inline | raw body, or JSON `{ url }` | words always; `segment` → `utterances` | `diarize_model=latest` + `utterances` | `prompt`, `speakers` | `seconds` (`metadata.duration`) | +| AssemblyAI | queued (upload → submit → poll) | `/v2/upload` then `audio_url`, or a URL | words always; `segment` → `speaker_labels` | `speaker_labels` | — | `seconds` (`audio_duration`) | + +Deferred: `Transcription.session(...)` — realtime STT over WebSocket (Deepgram live, AssemblyAI streaming, ElevenLabs +realtime, OpenAI realtime transcription) — is the same future scoped `session` shape as input-streaming TTS and ships +with the realtime work in phase 5. ElevenLabs Scribe is not implemented yet. ### `Generation` — shared async execution @@ -338,13 +386,14 @@ Existing facades gain per-modality selectors; the modality routes each facade pr | Facade | llm | image | video | speech | transcription | other | |---|---|---|---|---|---|---| | `OpenAI` | responses (default), chat | Images API | Sora (deprecated 2026-09-24) | ✓ | ✓ | | -| `Google` | Gemini | Gemini-native (default), `imagen` | Veo | Gemini TTS | Gemini transcribe | | +| `Google` | Gemini | Gemini-native (default), `imagen` | Veo | Gemini TTS | `gemini-3.5-transcribe` | | | `XAI` | ✓ | ✓ | ✓ | | | | | `ElevenLabs` | | | | ✓ | Scribe | soundEffect, music | | `Cartesia` | | | | ✓ | | | | `Deepgram` | | | | Aura | ✓ | | | `Fal` | | ✓ | ✓ | | | | -| `Replicate`, `Runway`, `Luma`, `Kling`, `MiniMax`, `AssemblyAI`, `BlackForestLabs`, `Stability` | | per provider | | | | | +| `AssemblyAI` | | | | | ✓ (queued) | | +| `Replicate`, `Runway`, `Luma`, `Kling`, `MiniMax`, `BlackForestLabs`, `Stability` | | per provider | | | | | New facades follow the existing one-file-per-provider rule. Package entrypoints are modality-specific, such as `@opencode/ai/providers/openai/images`, and return the concrete model. @@ -358,7 +407,7 @@ Media does not fit the LLM four-axis route (SSE frames → event state machine) - `MediaProtocol.queued` — `start` (body + decode to `{ token, snapshot }`), `status`, `result`, optional `cancel`, `pollHint`, and a `token` codec. `result` is always a separate GET (against the status document for Veo/xAI/Runway, fal's `response_url` otherwise) so `await` after `start` and after `resume` share one path. `PollContext.auth` hands the auth headers the route sent to the protocol for output URLs that need them (Veo downloads); they become transient `Media.Asset.headers`, never part of `source`. There is no separate `download` step: `Media.Asset.bytes()` downloads through the executor with those headers. `MediaRoute.inline(...)` / `MediaRoute.queued(...)` compose each kind with endpoint and auth; the queued route decodes the token once and hands `Generation` a token-free `{ status, result, cancel? }`. - `MediaProtocol.stream` — `body.from(request)` over the request plus its `mode`, `frames` (a function that picks the framing for the call: `Framing.sse`, `lines`, `document`, or the raw bytes), fresh per-response `initial()` state, `step` emitting modality events, and `finish(state, context)` — with the observed response for header-only usage — emitting exactly one terminal event or failing as an incomplete stream. The route fills `reason.http` on stream errors. `MediaRoute.stream(...)` exposes `stream` and `generate` (the same stream folded by the modality's `collect`). -`MediaRoute.inline` / `MediaRoute.queued` / `MediaRoute.stream` compose one protocol kind with endpoint/auth; `ImageModel`/`VideoModel`/`SpeechModel` share the `MediaModel` base (`src/media-model.ts`). +`MediaRoute.inline` / `MediaRoute.queued` / `MediaRoute.stream` compose one protocol kind with endpoint/auth and tag the route with its `kind`; `ImageModel`/`VideoModel`/`SpeechModel`/`TranscriptionModel` share the `MediaModel` base (`src/media-model.ts`). ### LLM integration @@ -388,7 +437,7 @@ Foundation + Image ship together as the reference implementation, serially. Vide 1. **Foundation** — per-modality selectors, `Media`, `Generation`, `Poll`, `Usage` union, `MediaProtocol` kinds, `@opencode/ai/promise` with `llm` + `image`. Port the five existing image protocols onto it. Unify `MediaPart` and add the `media` LLM event (fixes Gemini image output being dropped). 2. **Video** — ✅ Veo, xAI, fal, Runway shipped (`MediaProtocol.queued`, `Video.start/generate/resume/stream`, promise `ai.video`). Deferred: `Video.complete` (webhooks), Luma, Kling, MiniMax, Replicate. -3. **Speech + Transcription** — ✅ Speech: OpenAI, Gemini TTS, ElevenLabs, Cartesia, Deepgram shipped (`MediaProtocol.stream`, `Speech.generate/stream`, promise `ai.speech`). Pending: Transcription (OpenAI, ElevenLabs Scribe, Gemini, Deepgram, AssemblyAI). Deferred: `Speech.session` (WebSocket input streaming). +3. **Speech + Transcription** — ✅ Speech: OpenAI, Gemini TTS, ElevenLabs, Cartesia, Deepgram shipped (`MediaProtocol.stream`, `Speech.generate/stream`, promise `ai.speech`). ✅ Transcription: OpenAI, Gemini, Deepgram, AssemblyAI shipped across all three route kinds (`Transcription.generate/stream/start/resume`, promise `ai.transcription`). Pending: ElevenLabs Scribe. Deferred: `Speech.session` and `Transcription.session` (WebSocket streaming). 4. **Image queued routes and partials** — BFL, fal, Replicate, Stability; OpenAI `partial_images` streaming. 5. **Later** — ElevenLabs music/SFX, Lyria, `Speech.session` / `Transcription.session`, realtime. diff --git a/packages/ai/script/setup-recording-env.ts b/packages/ai/script/setup-recording-env.ts index ec47dea8e95..8e5d9be701f 100644 --- a/packages/ai/script/setup-recording-env.ts +++ b/packages/ai/script/setup-recording-env.ts @@ -161,7 +161,7 @@ const PROVIDERS: ReadonlyArray = [ id: "deepgram", label: "Deepgram", tier: "canary", - note: "Deepgram Aura text-to-speech recorded tests", + note: "Deepgram Aura text-to-speech and Nova transcription recorded tests", vars: [{ name: "DEEPGRAM_API_KEY" }], validate: (env) => HttpClientRequest.get("https://api.deepgram.com/v1/projects").pipe( @@ -169,6 +169,18 @@ const PROVIDERS: ReadonlyArray = [ executeRequest, ), }, + { + id: "assemblyai", + label: "AssemblyAI", + tier: "canary", + note: "AssemblyAI queued transcription recorded tests", + vars: [{ name: "ASSEMBLYAI_API_KEY" }], + validate: (env) => + HttpClientRequest.get("https://api.assemblyai.com/v2/transcript?limit=1").pipe( + HttpClientRequest.setHeader("authorization", Redacted.value(Redacted.make(env.ASSEMBLYAI_API_KEY))), + executeRequest, + ), + }, { id: "cloudflare-ai-gateway", label: "Cloudflare AI Gateway", diff --git a/packages/ai/src/generation.ts b/packages/ai/src/generation.ts index f1296b1473b..b2e93dac220 100644 --- a/packages/ai/src/generation.ts +++ b/packages/ai/src/generation.ts @@ -40,10 +40,21 @@ export interface AwaitOptions { export const DEFAULT_POLL_INTERVAL = Duration.seconds(5) export const DEFAULT_POLL_TIMEOUT = Duration.minutes(10) -export type Event = - | { readonly type: "generation-queued"; readonly id: string; readonly position?: number } - | { readonly type: "generation-progress"; readonly id: string; readonly progress?: number } - | { readonly type: "generation-finished"; readonly id: string; readonly status: Status } +export const QueuedEvent = Schema.Struct({ + type: Schema.tag("generation-queued"), + id: Schema.String, + position: Schema.optional(Schema.Number), +}).annotate({ identifier: "Generation.Event.Queued" }) + +export const ProgressEvent = Schema.Struct({ + type: Schema.tag("generation-progress"), + id: Schema.String, + progress: Schema.optional(Schema.Number), +}).annotate({ identifier: "Generation.Event.Progress" }) + +export type Observation = Schema.Schema.Type | Schema.Schema.Type + +export type Event = Observation | { readonly type: "generation-finished"; readonly id: string; readonly status: Status } const TERMINAL: ReadonlySet = new Set(["completed", "failed", "cancelled", "expired"]) @@ -168,3 +179,13 @@ export class Generation { ) } } + +export const resultEvents = ( + generation: Generation, + expand: (response: Response) => ReadonlyArray, + options?: AwaitOptions, +): Stream.Stream => + generation.events(options).pipe( + Stream.filter((event): event is Observation => event.type !== "generation-finished"), + Stream.concat(Stream.fromIterableEffect(Effect.map(generation.result(), expand))), + ) diff --git a/packages/ai/src/index.ts b/packages/ai/src/index.ts index d4907125bcf..594ae5d6d58 100644 --- a/packages/ai/src/index.ts +++ b/packages/ai/src/index.ts @@ -67,6 +67,25 @@ export type { SpeechRoute, } from "./speech.js" export { Speech } from "./speech.js" +export { TranscriptionClient } from "./transcription-client.js" +export { + TranscriptionEvent, + TranscriptionModel, + TranscriptionModelSchema, + TranscriptionRequest, + TranscriptionResponse, + TranscriptionSegment, + TranscriptionTimestamps, + TranscriptionWord, +} from "./transcription.js" +export type { + TranscriptionModelOptions, + TranscriptionOptions, + TranscriptionRequestFor, + TranscriptionRequestInput, + TranscriptionRoute, +} from "./transcription.js" +export { Transcription } from "./transcription.js" export { Media } from "./media.js" export { Generation } from "./generation.js" export type { diff --git a/packages/ai/src/promise.ts b/packages/ai/src/promise.ts index 703c693cc96..873e5fa2ec9 100644 --- a/packages/ai/src/promise.ts +++ b/packages/ai/src/promise.ts @@ -9,12 +9,21 @@ import { LanguageModel, LLMRequest } from "./schema/index.js" import type { RequestInput } from "./llm.js" import { Speech, SpeechModel, SpeechRequest, type SpeechRequestInput } from "./speech.js" import { SpeechClient } from "./speech-client.js" +import { + Transcription, + TranscriptionModel, + TranscriptionRequest, + type TranscriptionOptions, + type TranscriptionRequestInput, +} from "./transcription.js" +import { TranscriptionClient } from "./transcription-client.js" import { Video, VideoModel, VideoRequest, type VideoOptions, type VideoRequestInput } from "./video.js" import { VideoClient } from "./video-client.js" /** - * Promise-first entrypoint for scripts and non-Effect callers. One `ManagedRuntime` hosts the LLM, image, video, and - * speech clients over a request executor; every method runs the corresponding Effect API and rethrows `AIError` unchanged. + * Promise-first entrypoint for scripts and non-Effect callers. One `ManagedRuntime` hosts the LLM, image, video, speech, + * and transcription clients over a request executor; every method runs the corresponding Effect API and rethrows + * `AIError` unchanged. */ export interface Options { /** Executor layer; defaults to `RequestExecutor.fetchLayer`. Inject a recorder or middleware here. */ @@ -30,6 +39,7 @@ export type Services = | Layer.Success | Layer.Success | Layer.Success + | Layer.Success | RequestExecutor.Service /** Promise view of a `Generation`: its snapshot plus `await`, `refresh`, and `cancel` returning promises. */ @@ -56,9 +66,13 @@ const abortEffect = (signal: AbortSignal | undefined) => export const make = (options: Options = {}) => { const runtime = ManagedRuntime.make( - Layer.mergeAll(LLMClient.layer, ImageClient.layer, VideoClient.layer, SpeechClient.layer).pipe( - Layer.provideMerge(options.layer ?? RequestExecutor.fetchLayer), - ), + Layer.mergeAll( + LLMClient.layer, + ImageClient.layer, + VideoClient.layer, + SpeechClient.layer, + TranscriptionClient.layer, + ).pipe(Layer.provideMerge(options.layer ?? RequestExecutor.fetchLayer)), ) /** Run any package Effect (for example `asset.bytes()`) inside this runtime. */ @@ -93,6 +107,8 @@ export const make = (options: Options = {}) => { input instanceof VideoRequest ? input : Video.request(input) const speechRequest = (input: SpeechRequestInput | SpeechRequest) => input instanceof SpeechRequest ? input : Speech.request(input) + const transcriptionRequest = (input: TranscriptionRequestInput | TranscriptionRequest) => + input instanceof TranscriptionRequest ? input : Transcription.request(input) return { run, @@ -138,6 +154,26 @@ export const make = (options: Options = {}) => { options?: RunOptions, ) => iterate(Speech.stream(speechRequest(input)), options), }, + transcription: { + request: Transcription.request, + generate: ( + input: TranscriptionRequestInput | TranscriptionRequest, + options?: AwaitOptions & RunOptions, + ) => run(Transcription.generate(transcriptionRequest(input), { poll: options?.poll }), options), + stream: ( + input: TranscriptionRequestInput | TranscriptionRequest, + options?: AwaitOptions & RunOptions, + ) => iterate(Transcription.stream(transcriptionRequest(input), { poll: options?.poll }), options), + start: ( + input: TranscriptionRequestInput | TranscriptionRequest, + options?: RunOptions, + ) => run(Transcription.start(transcriptionRequest(input)), options).then(handle), + resume: ( + model: TranscriptionModel, + token: unknown, + options?: RunOptions, + ) => run(Transcription.resume(model, token), options).then(handle), + }, dispose: () => runtime.dispose(), } } diff --git a/packages/ai/src/protocols/assemblyai-transcription.ts b/packages/ai/src/protocols/assemblyai-transcription.ts new file mode 100644 index 00000000000..b3172b89980 --- /dev/null +++ b/packages/ai/src/protocols/assemblyai-transcription.ts @@ -0,0 +1,211 @@ +import { Effect, Schema } from "effect" +import type { HttpClientResponse } from "effect/unstable/http" +import type { Status } from "../generation.js" +import { Media } from "../media.js" +import { MediaProtocol } from "../route/media-protocol.js" +import { MediaRoute } from "../route/media.js" +import { ProviderID, mergeJsonRecords } from "../schema/index.js" +import { TranscriptionModel, TranscriptionResponse, type TranscriptionRequestFor } from "../transcription.js" +import { ProviderShared, optionalNull } from "./shared.js" +import { MediaInput } from "./utils/media-input.js" + +const ADAPTER = "assemblyai-transcription" +const NAME = "AssemblyAI" +const PROVIDER = ProviderID.make("assemblyai") +export const DEFAULT_BASE_URL = "https://api.assemblyai.com" +export const PATH = "/v2/transcript" +export const UPLOAD_PATH = "/v2/upload" + +// --------------------------------------------------------------------------- +// 1. Public model input +// --------------------------------------------------------------------------- + +export type AssemblyAITranscriptionOptions = { + readonly keyterms_prompt?: ReadonlyArray + readonly punctuate?: boolean + readonly format_text?: boolean + readonly disfluencies?: boolean + readonly filter_profanity?: boolean + readonly temperature?: number + readonly speaker_options?: { readonly min_speakers_expected?: number; readonly max_speakers_expected?: number } + readonly language_detection_options?: { + readonly expected_languages?: ReadonlyArray + readonly fallback_language?: string + readonly code_switching?: boolean + } + readonly speech_models?: ReadonlyArray<"universal-3-5-pro" | "universal-2" | (string & {})> +} & Record + +export type Request = TranscriptionRequestFor + +// --------------------------------------------------------------------------- +// 2. Token and response schemas +// --------------------------------------------------------------------------- + +export const Token = Schema.Struct({ transcriptID: Schema.String }) +export type Token = Schema.Schema.Type + +const Upload = Schema.Struct({ upload_url: Schema.String }) + +/** Word and utterance times are milliseconds. */ +const Transcript = Schema.Struct({ + id: Schema.String, + status: Schema.String, + text: optionalNull(Schema.String), + words: optionalNull( + Schema.Array( + Schema.Struct({ + text: Schema.String, + start: Schema.Number, + end: Schema.Number, + confidence: optionalNull(Schema.Number), + speaker: optionalNull(Schema.String), + }), + ), + ), + utterances: optionalNull( + Schema.Array( + Schema.Struct({ + text: Schema.String, + start: Schema.Number, + end: Schema.Number, + speaker: optionalNull(Schema.String), + }), + ), + ), + language_code: optionalNull(Schema.String), + audio_duration: optionalNull(Schema.Number), + speech_model_used: optionalNull(Schema.String), + error: optionalNull(Schema.String), +}) + +const STATUS = { + queued: "queued", + processing: "running", + completed: "completed", + error: "failed", +} as const satisfies Record + +// --------------------------------------------------------------------------- +// 5. Request body construction +// --------------------------------------------------------------------------- + +const decodeUpload = MediaProtocol.decodeJson(ADAPTER, NAME, Upload) + +/** `/v2/transcript` only takes a URL, so inline audio is uploaded to `/v2/upload` first. */ +const prepare = Effect.fn("AssemblyAITranscription.prepare")(function* (request: Request, send: MediaProtocol.Send) { + if (request.audio.source.type !== "bytes" && request.audio.source.type !== "base64") return request + const audio = yield* MediaInput.inlineBytes(ADAPTER, request.audio) + const uploaded = yield* send(UPLOAD_PATH, MediaProtocol.binary(audio, "application/octet-stream")).pipe( + Effect.flatMap(decodeUpload), + ) + return { ...request, audio: Media.url(uploaded.value.upload_url, { mediaType: request.audio.mediaType }) } +}) + +const fromRequest = Effect.fn("AssemblyAITranscription.fromRequest")(function* (request: Request) { + const audio = yield* ProviderShared.mediaReference(request.audio, PROVIDER, NAME) + return MediaProtocol.json( + mergeJsonRecords( + { + audio_url: audio.value, + speech_models: [request.model.id], + language_code: request.language, + language_detection: request.language === undefined ? true : undefined, + prompt: request.prompt, + // Turn-level `utterances`, the only segments AssemblyAI returns, require speaker labels. + speaker_labels: request.diarize === true || request.timestamps === "segment" ? true : undefined, + speakers_expected: request.speakers, + }, + request.providerOptions, + request.http?.body, + ) ?? {}, + ) +}) + +// --------------------------------------------------------------------------- +// 6. Response decoding +// --------------------------------------------------------------------------- + +const decodeTranscript = MediaProtocol.decodeJson(ADAPTER, NAME, Transcript) + +const decodeStart = Effect.fn("AssemblyAITranscription.decodeStart")(function* ( + response: HttpClientResponse.HttpClientResponse, +) { + const output = yield* decodeTranscript(response) + const status = yield* MediaProtocol.status(STATUS, output.value.status, output) + return { token: { transcriptID: output.value.id }, snapshot: { id: output.value.id, status } } +}) + +const decodeStatus = Effect.fn("AssemblyAITranscription.decodeStatus")(function* ( + response: HttpClientResponse.HttpClientResponse, + context: MediaProtocol.PollContext, +) { + const output = yield* decodeTranscript(response) + const status = yield* MediaProtocol.status(STATUS, output.value.status, output) + return { id: context.token.transcriptID, status } +}) + +const seconds = (milliseconds: number) => milliseconds / 1000 + +const decodeResult = Effect.fn("AssemblyAITranscription.decodeResult")(function* ( + response: HttpClientResponse.HttpClientResponse, + context: MediaProtocol.PollContext, +) { + const output = yield* decodeTranscript(response) + const transcript = output.value + const status = yield* MediaProtocol.status(STATUS, transcript.status, output) + const error = transcript.error ?? undefined + if (status === "failed") + return yield* output.ended("failed", `${NAME} transcription failed${error === undefined ? "" : `: ${error}`}`) + if (status !== "completed") + return yield* output.invalid(`${NAME} transcript ${context.token.transcriptID} has not finished`) + const duration = transcript.audio_duration ?? undefined + return new TranscriptionResponse({ + text: transcript.text ?? "", + segments: transcript.utterances?.map((utterance) => ({ + text: utterance.text, + startSeconds: seconds(utterance.start), + endSeconds: seconds(utterance.end), + speaker: utterance.speaker ?? undefined, + })), + words: transcript.words?.map((word) => ({ + text: word.text, + startSeconds: seconds(word.start), + endSeconds: seconds(word.end), + speaker: word.speaker ?? undefined, + confidence: word.confidence ?? undefined, + })), + language: transcript.language_code?.toLowerCase(), + durationSeconds: duration, + usage: duration === undefined ? undefined : { type: "seconds", seconds: duration }, + providerMetadata: { + assemblyai: { transcriptId: transcript.id, speechModel: transcript.speech_model_used ?? undefined }, + }, + }) +}) + +// --------------------------------------------------------------------------- +// 7. Protocol and route +// --------------------------------------------------------------------------- + +const transcriptPath = (token: Token) => `${PATH}/${token.transcriptID}` + +export const protocol = MediaProtocol.queued({ + id: ADAPTER, + name: NAME, + token: Token, + start: { prepare, body: { from: fromRequest }, decode: decodeStart }, + status: { path: transcriptPath, decode: decodeStatus }, + result: { path: transcriptPath, decode: decodeResult }, +}) + +export const model = (input: MediaRoute.ModelInput) => + TranscriptionModel.fromRoute( + { id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: PATH }, + input, + ) + +export const AssemblyAITranscription = { + protocol, + model, +} as const diff --git a/packages/ai/src/protocols/deepgram-speech.ts b/packages/ai/src/protocols/deepgram-speech.ts index f58cc026cd6..15926314a09 100644 --- a/packages/ai/src/protocols/deepgram-speech.ts +++ b/packages/ai/src/protocols/deepgram-speech.ts @@ -3,6 +3,7 @@ import { MediaProtocol } from "../route/media-protocol.js" import { MediaRoute } from "../route/media.js" import { ProviderID, mergeJsonRecords } from "../schema/index.js" import { SpeechModel, type SpeechEvent, type SpeechRequestFor } from "../speech.js" +import { MediaInput } from "./utils/media-input.js" import { SpeechStream } from "./utils/speech-stream.js" const ADAPTER = "deepgram-speech" @@ -49,26 +50,24 @@ const FORMATS: Readonly - typeof value === "string" || typeof value === "number" || typeof value === "boolean" ? String(value) : undefined +const audioFormat = (request: Request) => { + const format = request.format === undefined ? undefined : FORMATS[request.format] + return { + encoding: request.providerOptions?.encoding ?? format?.encoding, + container: request.providerOptions?.container ?? format?.container, + } +} const queryParameters = (request: Request) => { - const format = request.format === undefined ? undefined : FORMATS[request.format] - const { encoding, container, sampleRate, bitRate, ...native } = request.providerOptions ?? {} - return Object.fromEntries( - Object.entries({ - ...native, - model: request.model.id, - encoding: encoding ?? format?.encoding, - container: container ?? format?.container, - sample_rate: sampleRate, - bit_rate: bitRate, - speed: request.speed, - }).flatMap(([key, value]) => { - const text = queryValue(value) - return text === undefined ? [] : [[key, text]] - }), - ) + const { encoding: _encoding, container: _container, sampleRate, bitRate, ...native } = request.providerOptions ?? {} + return MediaInput.query(ADAPTER, { + ...native, + model: request.model.id, + ...audioFormat(request), + sample_rate: sampleRate, + bit_rate: bitRate, + speed: request.speed, + }) } const fromRequest = Effect.fn("DeepgramSpeech.fromRequest")(function* (request: Request) { @@ -84,7 +83,7 @@ const fromRequest = Effect.fn("DeepgramSpeech.fromRequest")(function* (request: ) return MediaProtocol.json( mergeJsonRecords({ text: request.text }, request.http?.body) ?? {}, - queryParameters(request), + yield* queryParameters(request), ) }) @@ -101,15 +100,15 @@ const HEADERLESS_ENCODINGS: Readonly> = const finish = (state: State, context: MediaProtocol.ResponseContext) => { const headers = context.http.headers const mediaType = headers["content-type"] - const query = queryParameters(context.request) - const encoding = HEADERLESS_ENCODINGS[query.encoding ?? ""] + const format = audioFormat(context.request) + const encoding = HEADERLESS_ENCODINGS[format.encoding ?? ""] const requestID = headers["dg-request-id"] const modelName = headers["dg-model-name"] return SpeechStream.finish(ADAPTER, state, { - ...(query.container === "none" && encoding !== undefined + ...(format.container === "none" && encoding !== undefined ? SpeechStream.pcm(encoding, SpeechStream.sampleRate(mediaType), mediaType) : // Deepgram's default encoding is MP3; WAV is a container around any encoding. - { mediaType, info: { format: query.container === "wav" ? "wav" : (query.encoding ?? "mp3") } }), + { mediaType, info: { format: format.container === "wav" ? "wav" : (format.encoding ?? "mp3") } }), usage: SpeechStream.headerUsage("characters", headers["dg-char-count"]), providerMetadata: requestID === undefined && modelName === undefined diff --git a/packages/ai/src/protocols/deepgram-transcription.ts b/packages/ai/src/protocols/deepgram-transcription.ts new file mode 100644 index 00000000000..f070584c48a --- /dev/null +++ b/packages/ai/src/protocols/deepgram-transcription.ts @@ -0,0 +1,191 @@ +import { Effect, Schema } from "effect" +import type { HttpClientResponse } from "effect/unstable/http" +import { MediaProtocol } from "../route/media-protocol.js" +import { MediaRoute } from "../route/media.js" +import { ProviderID, mergeJsonRecords } from "../schema/index.js" +import { TranscriptionModel, TranscriptionResponse, type TranscriptionRequestFor } from "../transcription.js" +import { ProviderShared } from "./shared.js" +import { MediaInput } from "./utils/media-input.js" + +const ADAPTER = "deepgram-transcription" +const NAME = "Deepgram" +const PROVIDER = ProviderID.make("deepgram") +export const DEFAULT_BASE_URL = "https://api.deepgram.com" +export const PATH = "/v1/listen" + +// --------------------------------------------------------------------------- +// 1. Public model input +// --------------------------------------------------------------------------- + +export type DeepgramTranscriptionOptions = { + readonly smart_format?: boolean + readonly punctuate?: boolean + readonly paragraphs?: boolean + readonly utterances?: boolean + readonly detect_language?: boolean | ReadonlyArray + readonly keyterm?: ReadonlyArray + readonly diarize_model?: "latest" | "v1" | "v2" | (string & {}) + readonly filler_words?: boolean + readonly numerals?: boolean + readonly mip_opt_out?: boolean + readonly tag?: string | ReadonlyArray +} & Record + +export type Request = TranscriptionRequestFor + +// --------------------------------------------------------------------------- +// 2. Response schema +// --------------------------------------------------------------------------- + +const Word = Schema.Struct({ + word: Schema.String, + start: Schema.Number, + end: Schema.Number, + confidence: Schema.optional(Schema.Number), + speaker: Schema.optional(Schema.Number), + punctuated_word: Schema.optional(Schema.String), +}) + +const ListenResponse = Schema.Struct({ + metadata: Schema.optional( + Schema.Struct({ request_id: Schema.optional(Schema.String), duration: Schema.optional(Schema.Number) }), + ), + results: Schema.Struct({ + channels: Schema.Array( + Schema.Struct({ + alternatives: Schema.optional( + Schema.Array(Schema.Struct({ transcript: Schema.String, words: Schema.optional(Schema.Array(Word)) })), + ), + detected_language: Schema.optional(Schema.String), + }), + ), + utterances: Schema.optional( + Schema.Array( + Schema.Struct({ + start: Schema.Number, + end: Schema.Number, + transcript: Schema.String, + speaker: Schema.optional(Schema.Number), + words: Schema.optional(Schema.Array(Word)), + }), + ), + ), + }), +}) + +// --------------------------------------------------------------------------- +// 5. Request body construction +// --------------------------------------------------------------------------- + +const query = (request: Request) => + MediaInput.query( + ADAPTER, + mergeJsonRecords( + { + model: request.model.id, + smart_format: true, + language: request.language, + // Deepgram assumes English unless asked to detect, unlike the other routes' auto-detection. + detect_language: request.language === undefined ? true : undefined, + // `diarize=true` is deprecated in favor of choosing a diarization model. + diarize_model: request.diarize === true ? "latest" : undefined, + utterances: request.diarize === true || request.timestamps === "segment" ? true : undefined, + }, + request.providerOptions, + ) ?? {}, + ) + +const fromRequest = Effect.fn("DeepgramTranscription.fromRequest")(function* (request: Request) { + const url = ProviderShared.mediaUrl(request.audio) + if (url !== undefined) + return MediaProtocol.json(mergeJsonRecords({ url }, request.http?.body) ?? {}, yield* query(request)) + if (request.http?.body !== undefined) + return yield* ProviderShared.invalidRequest(`${NAME} sends inline audio as the raw body, so http.body cannot apply`) + const audio = yield* MediaInput.inlineBytes(ADAPTER, request.audio) + return MediaProtocol.binary(audio, request.audio.mediaType, yield* query(request)) +}) + +// --------------------------------------------------------------------------- +// 6. Response decoding +// --------------------------------------------------------------------------- + +const decodeListen = MediaProtocol.decodeJson(ADAPTER, NAME, ListenResponse) + +const speaker = (value: number | undefined) => (value === undefined ? undefined : String(value)) + +const wordText = (word: typeof Word.Type) => word.punctuated_word ?? word.word + +// Utterances split on pauses, not speakers: the v2 diarizer labels a whole utterance with one speaker even when its +// words change speaker, so segments split each utterance at speaker changes. +const speakerTurns = (words: ReadonlyArray) => + words.reduce>>((turns, word) => { + const last = turns.at(-1) + if (last === undefined || last[0].speaker !== word.speaker) return [...turns, [word]] + last.push(word) + return turns + }, []) + +const decodeResponse = Effect.fn("DeepgramTranscription.decodeResponse")(function* ( + response: HttpClientResponse.HttpClientResponse, +) { + const output = yield* decodeListen(response) + const channel = output.value.results.channels[0] + const alternative = channel?.alternatives?.[0] + if (alternative === undefined) return yield* output.invalid(`${NAME} returned no transcript`) + const duration = output.value.metadata?.duration + const requestID = output.value.metadata?.request_id + return new TranscriptionResponse({ + text: alternative.transcript, + segments: output.value.results.utterances?.flatMap((utterance) => + utterance.words === undefined || utterance.words.length === 0 + ? [ + { + text: utterance.transcript, + startSeconds: utterance.start, + endSeconds: utterance.end, + speaker: speaker(utterance.speaker), + }, + ] + : speakerTurns(utterance.words).map((turn) => ({ + text: turn.map(wordText).join(" "), + startSeconds: turn[0].start, + endSeconds: turn[turn.length - 1].end, + speaker: speaker(turn[0].speaker), + })), + ), + words: alternative.words?.map((word) => ({ + text: wordText(word), + startSeconds: word.start, + endSeconds: word.end, + speaker: speaker(word.speaker), + confidence: word.confidence, + })), + language: channel?.detected_language?.toLowerCase(), + durationSeconds: duration, + usage: duration === undefined ? undefined : { type: "seconds", seconds: duration }, + providerMetadata: requestID === undefined ? undefined : { deepgram: { requestId: requestID } }, + }) +}) + +// --------------------------------------------------------------------------- +// 7. Protocol and route +// --------------------------------------------------------------------------- + +export const protocol = MediaProtocol.inline({ + id: ADAPTER, + name: NAME, + unsupported: ["prompt", "speakers"], + body: { from: fromRequest }, + response: { decode: decodeResponse }, +}) + +export const model = (input: MediaRoute.ModelInput) => + TranscriptionModel.fromRoute( + { id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: PATH }, + input, + ) + +export const DeepgramTranscription = { + protocol, + model, +} as const diff --git a/packages/ai/src/protocols/gemini.ts b/packages/ai/src/protocols/gemini.ts index d1cdb308fea..46b7210ffa7 100644 --- a/packages/ai/src/protocols/gemini.ts +++ b/packages/ai/src/protocols/gemini.ts @@ -22,6 +22,7 @@ import { import { classifyProviderFailure } from "../provider-error.js" import { Media } from "../media.js" import { JsonObject, knownString, lenient, optionalArray, optionalNull, ProviderShared } from "./shared.js" +import { GeminiGenerateContent } from "./utils/gemini-generate-content.js" import { GeminiToolSchema } from "./utils/gemini-tool-schema.js" import { Lifecycle } from "./utils/lifecycle.js" import { ToolSchemaProjection } from "./utils/tool-schema.js" @@ -305,14 +306,9 @@ const lowerToolConfig = (toolChoice: NonNullable) => tool: (name) => ({ functionCallingConfig: { mode: "ANY" as const, allowedFunctionNames: [name] } }), }) -// Gemini does not fetch public URLs; inline payloads and Gemini Files references are the accepted inputs. const lowerContentPart = Effect.fn("Gemini.lowerContentPart")(function* (part: TextPart | MediaPart) { if (part.type === "text") return { text: part.text } - const source = part.media.source - if (source.type === "ref" && source.provider === "google") - return { fileData: { mimeType: part.media.mediaType, fileUri: source.id } } - const media = yield* ProviderShared.requireInlineMedia("Gemini", part.media) - return { inlineData: { mimeType: media.mime, data: media.base64 } } + return yield* GeminiGenerateContent.mediaPart("Gemini", part.media) }) const providerMetadata = (key: string, metadata: Record): ProviderMetadata => ({ [key]: metadata }) diff --git a/packages/ai/src/protocols/google-images.ts b/packages/ai/src/protocols/google-images.ts index d2a6885f1c6..5dd13fe2916 100644 --- a/packages/ai/src/protocols/google-images.ts +++ b/packages/ai/src/protocols/google-images.ts @@ -1,11 +1,11 @@ import { Effect, Schema } from "effect" import type { HttpClientResponse } from "effect/unstable/http" import { ImageModel, ImageResponse, type ImageRequestFor } from "../image.js" -import { Media } from "../media.js" import { MediaProtocol } from "../route/media-protocol.js" import { MediaRoute } from "../route/media.js" -import { ProviderID, mergeJsonRecords, type AIError } from "../schema/index.js" +import { ProviderID, mergeJsonRecords } from "../schema/index.js" import { ProviderShared } from "./shared.js" +import { GeminiGenerateContent } from "./utils/gemini-generate-content.js" import { MediaInput } from "./utils/media-input.js" const ADAPTER = "google-images" @@ -102,25 +102,6 @@ const generationConfig = (request: Request) => { ) } -// Gemini does not fetch public URLs; inline payloads or Gemini Files references are the only accepted inputs. -const imagePart = (asset: Media.Asset): Effect.Effect, AIError> => { - const inline = asset.inline() - if (inline) return Effect.succeed({ inlineData: { mimeType: inline.mime, data: inline.base64 } }) - const id = MediaInput.refID(asset, PROVIDER) - if (id) return Effect.succeed({ fileData: { mimeType: asset.mediaType, fileUri: id } }) - if (asset.source.type === "ref") - return Effect.fail( - ProviderShared.invalidRequest( - "Google generateContent requires Gemini file references rather than other providers' file IDs", - ), - ) - return Effect.fail( - ProviderShared.invalidRequest( - "Google generateContent does not fetch public image URLs; use bytes, a data URL, or a Gemini file reference", - ), - ) -} - const fromRequest = Effect.fn("GoogleImages.fromRequest")(function* (request: Request) { if (request.n !== undefined && request.n > 1) return yield* ProviderShared.unsupportedOperation({ @@ -129,7 +110,7 @@ const fromRequest = Effect.fn("GoogleImages.fromRequest")(function* (request: Re route: ADAPTER, message: `${NAME} generates one image per request; call it once per image instead of n=${request.n}`, }) - const parts = yield* Effect.forEach(request.images ?? [], imagePart) + const parts = yield* Effect.forEach(request.images ?? [], (image) => GeminiGenerateContent.mediaPart(NAME, image)) return MediaProtocol.json( mergeJsonRecords( { diff --git a/packages/ai/src/protocols/google-speech.ts b/packages/ai/src/protocols/google-speech.ts index c76f736b786..7f09167bca3 100644 --- a/packages/ai/src/protocols/google-speech.ts +++ b/packages/ai/src/protocols/google-speech.ts @@ -1,10 +1,9 @@ import { Effect, Schema } from "effect" -import { Framing } from "../route/framing.js" import { MediaProtocol } from "../route/media-protocol.js" import { MediaRoute } from "../route/media.js" -import { AIError, ContentPolicyError, ProviderID, mergeJsonRecords } from "../schema/index.js" +import { ProviderID, mergeJsonRecords } from "../schema/index.js" import { SpeechModel, type SpeechEvent, type SpeechRequestFor } from "../speech.js" -import { ProviderShared } from "./shared.js" +import { GeminiGenerateContent } from "./utils/gemini-generate-content.js" import { SpeechStream } from "./utils/speech-stream.js" const ADAPTER = "google-speech" @@ -37,42 +36,12 @@ export type Request = SpeechRequestFor // 3. Streaming event schema // --------------------------------------------------------------------------- -const GenerateContentChunk = Schema.Struct({ - candidates: Schema.optional( - Schema.Array( - Schema.Struct({ - content: Schema.optional( - Schema.Struct({ - parts: Schema.optional( - Schema.Array( - Schema.Struct({ - text: Schema.optional(Schema.String), - inlineData: Schema.optional( - Schema.Struct({ mimeType: Schema.String, data: Schema.Uint8ArrayFromBase64 }), - ), - }), - ), - ), - }), - ), - finishReason: Schema.optional(Schema.String), - }), - ), - ), - promptFeedback: Schema.optional( - Schema.Struct({ blockReason: Schema.optional(Schema.String), blockReasonMessage: Schema.optional(Schema.String) }), - ), - usageMetadata: Schema.optional( - Schema.Struct({ - promptTokenCount: Schema.optional(Schema.Number), - candidatesTokenCount: Schema.optional(Schema.Number), - totalTokenCount: Schema.optional(Schema.Number), - }), - ), - modelVersion: Schema.optional(Schema.String), - responseId: Schema.optional(Schema.String), -}) -type GenerateContentChunk = Schema.Schema.Type +const GenerateContentChunk = GeminiGenerateContent.chunk( + Schema.Struct({ + text: Schema.optional(Schema.String), + inlineData: Schema.optional(Schema.Struct({ mimeType: Schema.String, data: Schema.Uint8ArrayFromBase64 })), + }), +) const decodeChunk = MediaProtocol.decodeFrame(ADAPTER, NAME, GenerateContentChunk) @@ -80,12 +49,8 @@ const decodeChunk = MediaProtocol.decodeFrame(ADAPTER, NAME, GenerateContentChun // 4. Parser state // --------------------------------------------------------------------------- -interface State extends SpeechStream.Audio { +interface State extends SpeechStream.Audio, GeminiGenerateContent.Metadata { readonly mimeType?: string - readonly usage?: GenerateContentChunk["usageMetadata"] - readonly finishReason?: string - readonly modelVersion?: string - readonly responseId?: string } // --------------------------------------------------------------------------- @@ -120,65 +85,27 @@ const fromRequest = Effect.fn("GoogleSpeech.fromRequest")(function* (request: Me ) }) -// Only `gemini-3.1-flash-tts-preview` and later stream; earlier TTS models reject `streamGenerateContent`. -const path = (request: MediaProtocol.Addressed) => - request.mode === "stream" - ? `/models/${request.model.id}:streamGenerateContent?alt=sse` - : `/models/${request.model.id}:generateContent` - // --------------------------------------------------------------------------- // 6. Stream parsing // --------------------------------------------------------------------------- const step = Effect.fn("GoogleSpeech.step")(function* (state: State, frame: string) { const chunk = yield* decodeChunk(frame) - const blockReason = chunk.promptFeedback?.blockReason - if (blockReason !== undefined) - return yield* new AIError({ - reason: new ContentPolicyError({ - message: `${NAME} blocked the prompt (${blockReason})${ - chunk.promptFeedback?.blockReasonMessage === undefined ? "" : `: ${chunk.promptFeedback.blockReasonMessage}` - }`, - body: frame, - }), - }) - const candidate = chunk.candidates?.[0] - const audio = (candidate?.content?.parts ?? []).flatMap((part) => + const blocked = GeminiGenerateContent.blocked(NAME, chunk, frame) + if (blocked !== undefined) return yield* blocked + const audio = (chunk.candidates?.[0]?.content?.parts ?? []).flatMap((part) => part.inlineData === undefined ? [] : [part.inlineData], ) - const next: State = { - ...state, - mimeType: state.mimeType ?? audio[0]?.mimeType, - usage: chunk.usageMetadata ?? state.usage, - finishReason: candidate?.finishReason ?? state.finishReason, - modelVersion: chunk.modelVersion ?? state.modelVersion, - responseId: chunk.responseId ?? state.responseId, - } + const next: State = { ...GeminiGenerateContent.track(state, chunk), mimeType: state.mimeType ?? audio[0]?.mimeType } return [next, audio.flatMap((part) => SpeechStream.delta(next, part.data)[1])] as const }) const finish = (state: State) => { - const usage = state.usage const sampleRate = SpeechStream.sampleRate(state.mimeType) ?? DEFAULT_SAMPLE_RATE return SpeechStream.finish(ADAPTER, state, { ...SpeechStream.pcm("pcm_s16le", sampleRate, state.mimeType ?? `audio/L16;codec=pcm;rate=${sampleRate}`), - usage: - usage === undefined - ? undefined - : { - type: "tokens", - input: usage.promptTokenCount, - output: usage.candidatesTokenCount, - total: ProviderShared.totalTokens( - usage.promptTokenCount, - usage.candidatesTokenCount, - usage.totalTokenCount, - ), - details: { google: usage }, - }, - providerMetadata: { - google: { finishReason: state.finishReason, modelVersion: state.modelVersion, responseId: state.responseId }, - }, + usage: GeminiGenerateContent.usage(state.usage), + providerMetadata: GeminiGenerateContent.providerMetadata(state), detail: state.finishReason === undefined ? undefined : `finish reason: ${state.finishReason}`, }) } @@ -192,9 +119,7 @@ export const protocol = MediaProtocol.stream - context.request.mode === "stream" ? Framing.sse.frame(bytes) : Framing.document.frame(bytes), + frames: (bytes, context) => GeminiGenerateContent.frames(bytes, context.request.mode), initial: () => ({ chunks: [] }), step, finish, @@ -202,7 +127,14 @@ export const protocol = MediaProtocol.stream SpeechModel.fromRoute( - { id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: ({ request }) => path(request) }, + { + id: ADAPTER, + provider: PROVIDER, + protocol, + baseURL: DEFAULT_BASE_URL, + // Only `gemini-3.1-flash-tts-preview` and later stream; earlier TTS models reject `streamGenerateContent`. + path: ({ request }) => GeminiGenerateContent.path(request.model.id, request.mode), + }, input, ) diff --git a/packages/ai/src/protocols/google-transcription.ts b/packages/ai/src/protocols/google-transcription.ts new file mode 100644 index 00000000000..a7470786169 --- /dev/null +++ b/packages/ai/src/protocols/google-transcription.ts @@ -0,0 +1,214 @@ +import { Effect, Schema, SchemaGetter } from "effect" +import { MediaProtocol } from "../route/media-protocol.js" +import { MediaRoute } from "../route/media.js" +import { ProviderID, mergeJsonRecords } from "../schema/index.js" +import { + TranscriptionFinishEvent, + TranscriptionModel, + TranscriptionSegmentEvent, + TranscriptionTextDeltaEvent, + type TranscriptionRequestFor, + type TranscriptionSegment, + type TranscriptionWord, + type TranscriptionEvent, +} from "../transcription.js" +import { ProviderShared } from "./shared.js" +import { GeminiGenerateContent } from "./utils/gemini-generate-content.js" + +const ADAPTER = "google-transcription" +const NAME = "Google Transcription" +const PROVIDER = ProviderID.make("google") +export const DEFAULT_BASE_URL = "https://generativelanguage.googleapis.com/v1beta" + +// --------------------------------------------------------------------------- +// 1. Public model input +// --------------------------------------------------------------------------- + +/** + * Merged into `generationConfig`. The API rejects `customVocabulary` and `mode: "SMART"` alongside diarization or word + * timestamps. + */ +export type GoogleTranscriptionOptions = { + readonly audioTranscriptionConfig?: { + readonly mode?: "VERBATIM" | "SMART" | (string & {}) + readonly customVocabulary?: ReadonlyArray + readonly languageCodes?: ReadonlyArray + } +} & Record + +export type Request = TranscriptionRequestFor + +// --------------------------------------------------------------------------- +// 3. Streaming event schema +// --------------------------------------------------------------------------- + +const Seconds = Schema.String.check(Schema.isPattern(/^\d+(\.\d+)?s$/)).pipe( + Schema.decodeTo(Schema.Number, { + decode: SchemaGetter.transform((value) => Number.parseFloat(value)), + encode: SchemaGetter.transform((value) => `${value}s`), + }), +) + +const AudioTranscription = Schema.Struct({ + text: Schema.String, + speakerLabel: Schema.optional(Schema.String), + words: Schema.optional( + Schema.Array( + Schema.Struct({ + word: Schema.String, + startOffset: Seconds, + endOffset: Seconds, + speakerLabel: Schema.optional(Schema.String), + }), + ), + ), +}) + +const decodeChunk = MediaProtocol.decodeFrame( + ADAPTER, + NAME, + GeminiGenerateContent.chunk(Schema.Struct({ audioTranscription: Schema.optional(AudioTranscription) })), +) + +// --------------------------------------------------------------------------- +// 4. Parser state +// --------------------------------------------------------------------------- + +interface State extends GeminiGenerateContent.Metadata { + readonly text: string + readonly segments: Array + readonly words: Array +} + +// --------------------------------------------------------------------------- +// 5. Request body construction +// --------------------------------------------------------------------------- + +const fromRequest = Effect.fn("GoogleTranscription.fromRequest")(function* (request: MediaProtocol.Addressed) { + // General Gemini models ignore `audioTranscriptionConfig` and answer the audio conversationally. + if (!request.model.id.includes("transcribe")) + return yield* ProviderShared.unsupportedOperation({ + operation: "transcription.model", + provider: PROVIDER, + route: ADAPTER, + message: `${request.model.id} is not a transcription model; use a transcribe model such as gemini-3.5-transcribe`, + }) + return MediaProtocol.json( + mergeJsonRecords( + { + contents: [{ role: "user", parts: [yield* GeminiGenerateContent.mediaPart(ADAPTER, request.audio)] }], + generationConfig: mergeJsonRecords( + { + audioTranscriptionConfig: { + languageCodes: request.language === undefined ? undefined : [request.language], + // Parts carry no offsets of their own, so segment times come from word offsets. + wordTimestamp: + request.diarize === true || request.timestamps === "word" || request.timestamps === "segment" + ? true + : undefined, + diarization: request.diarize === true ? true : undefined, + }, + }, + request.providerOptions, + ), + }, + request.http?.body, + ) ?? {}, + ) +}) + +// --------------------------------------------------------------------------- +// 6. Stream parsing +// --------------------------------------------------------------------------- + +const turn = (part: Schema.Schema.Type) => { + const words = (part.words ?? []).map((word) => ({ + text: word.word, + startSeconds: word.startOffset, + endSeconds: word.endOffset, + speaker: word.speakerLabel ?? part.speakerLabel, + })) + const first = words[0] + const last = words.at(-1) + return { + text: part.text, + words, + segment: + first === undefined || last === undefined + ? undefined + : { + text: part.text, + startSeconds: first.startSeconds, + endSeconds: last.endSeconds, + speaker: part.speakerLabel, + }, + } +} + +const step = Effect.fn("GoogleTranscription.step")(function* (state: State, frame: string) { + const chunk = yield* decodeChunk(frame) + const blocked = GeminiGenerateContent.blocked(NAME, chunk, frame) + if (blocked !== undefined) return yield* blocked + const turns = (chunk.candidates?.[0]?.content?.parts ?? []).flatMap((part) => + part.audioTranscription === undefined ? [] : [turn(part.audioTranscription)], + ) + const segments = turns.flatMap((item) => (item.segment === undefined ? [] : [item.segment])) + state.words.push(...turns.flatMap((item) => item.words)) + state.segments.push(...segments) + // Each part is one whole speaker turn without surrounding whitespace, so turns join with a space. + const text = turns + .map((item) => item.text) + .filter((item) => item.length > 0) + .join(" ") + const delta = text.length === 0 || state.text.length === 0 ? text : ` ${text}` + const events: ReadonlyArray = [ + ...(delta.length === 0 ? [] : [TranscriptionTextDeltaEvent.make({ delta })]), + ...segments.map((segment) => TranscriptionSegmentEvent.make({ segment })), + ] + return [{ ...GeminiGenerateContent.track(state, chunk), text: state.text + delta }, events] as const +}) + +const finish = (state: State) => { + if (state.finishReason === undefined) return Effect.fail(MediaProtocol.incomplete(ADAPTER)) + return Effect.succeed([ + TranscriptionFinishEvent.make({ + text: state.text, + segments: state.segments.length === 0 ? undefined : state.segments, + words: state.words.length === 0 ? undefined : state.words, + usage: GeminiGenerateContent.usage(state.usage), + providerMetadata: GeminiGenerateContent.providerMetadata(state), + }), + ]) +} + +// --------------------------------------------------------------------------- +// 7. Protocol and route +// --------------------------------------------------------------------------- + +export const protocol = MediaProtocol.stream({ + id: ADAPTER, + name: NAME, + unsupported: ["prompt", "speakers"], + body: { from: fromRequest }, + frames: (bytes, context) => GeminiGenerateContent.frames(bytes, context.request.mode), + initial: () => ({ text: "", segments: [], words: [] }), + step, + finish, +}) + +export const model = (input: MediaRoute.ModelInput) => + TranscriptionModel.fromRoute( + { + id: ADAPTER, + provider: PROVIDER, + protocol, + baseURL: DEFAULT_BASE_URL, + path: ({ request }) => GeminiGenerateContent.path(request.model.id, request.mode), + }, + input, + ) + +export const GoogleTranscription = { + protocol, + model, +} as const diff --git a/packages/ai/src/protocols/openai-images.ts b/packages/ai/src/protocols/openai-images.ts index 34a581b4179..0eeeb1dd835 100644 --- a/packages/ai/src/protocols/openai-images.ts +++ b/packages/ai/src/protocols/openai-images.ts @@ -72,12 +72,6 @@ const isEdit = (request: Request) => (request.images?.length ?? 0) > 0 const isInline = (asset: Media.Asset) => asset.inline() !== undefined -const blob = (data: Uint8Array, mediaType: string) => { - const buffer = new ArrayBuffer(data.byteLength) - new Uint8Array(buffer).set(data) - return new Blob([buffer], { type: mediaType }) -} - const reference = (asset: Media.Asset) => ProviderShared.mediaReference(asset, PROVIDER, NAME).pipe( Effect.map((item) => (item.type === "ref" ? { file_id: item.value } : { image_url: item.value })), @@ -104,9 +98,11 @@ const fromRequest = Effect.fn("OpenAIImages.fromRequest")(function* (request: Re form.append(key, typeof value === "string" ? value : ProviderShared.encodeJson(value)) }) const uploads = yield* Effect.forEach(images, (image) => MediaInput.inlineBytes(ADAPTER, image)) - uploads.forEach((data, index) => form.append("image[]", blob(data, images[index].mediaType), `image-${index}`)) + uploads.forEach((data, index) => + form.append("image[]", MediaInput.blob(data, images[index].mediaType), `image-${index}`), + ) if (mask !== undefined) - form.append("mask", blob(yield* MediaInput.inlineBytes(ADAPTER, mask), mask.mediaType), "mask") + form.append("mask", MediaInput.blob(yield* MediaInput.inlineBytes(ADAPTER, mask), mask.mediaType), "mask") return MediaProtocol.multipart(form) } @@ -130,6 +126,7 @@ const fromRequest = Effect.fn("OpenAIImages.fromRequest")(function* (request: Re // --------------------------------------------------------------------------- const requestedFormat = (body: MediaProtocol.Body) => { + if (body.type === "binary") return undefined const value = body.type === "json" ? body.value.output_format : body.value.get("output_format") return typeof value === "string" ? value : undefined } diff --git a/packages/ai/src/protocols/openai-transcription.ts b/packages/ai/src/protocols/openai-transcription.ts new file mode 100644 index 00000000000..a6d0ae98720 --- /dev/null +++ b/packages/ai/src/protocols/openai-transcription.ts @@ -0,0 +1,277 @@ +import { Effect, Schema, Stream } from "effect" +import { Framing } from "../route/framing.js" +import { MediaProtocol } from "../route/media-protocol.js" +import { MediaRoute } from "../route/media.js" +import { ProviderID, mergeJsonRecords, type MediaUsage } from "../schema/index.js" +import { + TranscriptionFinishEvent, + TranscriptionModel, + TranscriptionSegmentEvent, + TranscriptionTextDeltaEvent, + type TranscriptionEvent, + type TranscriptionRequestFor, + type TranscriptionSegment, +} from "../transcription.js" +import { mediaTypeExtension } from "../utils/media-type.js" +import { ProviderShared } from "./shared.js" +import { MediaInput } from "./utils/media-input.js" + +const ADAPTER = "openai-transcription" +const NAME = "OpenAI Transcription" +const PROVIDER = ProviderID.make("openai") +export const DEFAULT_BASE_URL = "https://api.openai.com/v1" +export const PATH = "/audio/transcriptions" + +// --------------------------------------------------------------------------- +// 1. Public model input +// --------------------------------------------------------------------------- + +export type OpenAITranscriptionOptions = { + readonly chunking_strategy?: + | "auto" + | { + readonly type: "server_vad" + readonly prefix_padding_ms?: number + readonly silence_duration_ms?: number + readonly threshold?: number + } + readonly include?: ReadonlyArray<"logprobs"> + readonly keywords?: ReadonlyArray + readonly known_speaker_names?: ReadonlyArray + readonly known_speaker_references?: ReadonlyArray + readonly temperature?: number +} & Record + +export type Request = TranscriptionRequestFor + +// --------------------------------------------------------------------------- +// 3. Streaming event schema +// --------------------------------------------------------------------------- + +const Segment = Schema.Struct({ + text: Schema.String, + start: Schema.Number, + end: Schema.Number, + speaker: Schema.optional(Schema.String), +}) + +const Usage = Schema.Union([ + Schema.Struct({ + type: Schema.Literal("tokens"), + input_tokens: Schema.optional(Schema.Number), + output_tokens: Schema.optional(Schema.Number), + total_tokens: Schema.optional(Schema.Number), + }), + Schema.Struct({ type: Schema.Literal("duration"), seconds: Schema.Number }), +]) + +const transcriptFields = { + text: Schema.String, + language: Schema.optional(Schema.String), + languages: Schema.optional(Schema.Array(Schema.Struct({ code: Schema.String }))), + duration: Schema.optional(Schema.Number), + segments: Schema.optional(Schema.Array(Segment)), + words: Schema.optional( + Schema.Array(Schema.Struct({ word: Schema.String, start: Schema.Number, end: Schema.Number })), + ), + usage: Schema.optional(Usage), +} + +const Event = Schema.Union([ + Schema.Struct({ type: Schema.Literal("transcript.text.delta"), delta: Schema.String }), + Schema.Struct({ type: Schema.Literal("transcript.text.segment"), ...Segment.fields }), + Schema.Struct({ type: Schema.Literal("transcript.text.done"), ...transcriptFields }), +]) +const Transcript = Schema.Struct(transcriptFields) +type Transcript = Schema.Schema.Type + +const decodeEvent = MediaProtocol.decodeFrame(ADAPTER, NAME, Event) +const decodeTranscript = MediaProtocol.decodeFrame(ADAPTER, NAME, Transcript) + +type Frame = string | { readonly document: string } + +// --------------------------------------------------------------------------- +// 4. Parser state +// --------------------------------------------------------------------------- + +interface State { + readonly segments: Array + readonly transcript?: Transcript +} + +// --------------------------------------------------------------------------- +// 5. Request body construction +// --------------------------------------------------------------------------- + +interface Capabilities { + readonly stream: boolean + readonly timestamps: ReadonlyArray<"segment" | "word"> + readonly diarize: boolean + readonly languageField: "language" | "languages" +} + +const TRANSCRIBE: Capabilities = { stream: true, timestamps: [], diarize: false, languageField: "language" } + +const capabilities = (model: string): Capabilities => { + if (model.startsWith("whisper")) return { ...TRANSCRIBE, stream: false, timestamps: ["segment", "word"] } + if (model.includes("diarize")) return { ...TRANSCRIBE, timestamps: ["segment"], diarize: true } + // `gpt-transcribe` replaces `language` with `languages[]` and rejects both together. + if (model.startsWith("gpt-transcribe")) return { ...TRANSCRIBE, languageField: "languages" } + return TRANSCRIBE +} + +const unsupported = (operation: string, message: string) => + Effect.fail(ProviderShared.unsupportedOperation({ operation, provider: PROVIDER, route: ADAPTER, message })) + +const validate = (request: MediaProtocol.Addressed, model: Capabilities) => { + const id = request.model.id + if (request.mode === "stream" && !model.stream) + return unsupported("media.stream", `${id} does not stream; use Transcription.generate`) + if (request.diarize === true && !model.diarize) + return unsupported("media.diarize", `${id} does not diarize; use gpt-4o-transcribe-diarize`) + if (request.prompt !== undefined && model.diarize) + return unsupported("media.prompt", `${id} does not accept a prompt`) + if ( + request.timestamps === undefined || + request.timestamps === "none" || + model.timestamps.includes(request.timestamps) + ) + return Effect.void + return unsupported("media.timestamps", `${id} does not return ${request.timestamps} timestamps`) +} + +const RESERVED_FORM_FIELDS = new Set([ + "file", + "model", + "prompt", + "language", + "response_format", + "timestamp_granularities", + "stream", +]) + +const appendField = (form: FormData, key: string, value: unknown) => { + if (Array.isArray(value)) return value.forEach((item) => form.append(`${key}[]`, String(item))) + form.append(key, typeof value === "object" && value !== null ? ProviderShared.encodeJson(value) : String(value)) +} + +const fromRequest = Effect.fn("OpenAITranscription.fromRequest")(function* (request: MediaProtocol.Addressed) { + const model = capabilities(request.model.id) + yield* validate(request, model) + // The API detects the audio format from the upload's filename extension. + const extension = mediaTypeExtension(request.audio.mediaType) + if (extension === undefined) + return yield* ProviderShared.invalidRequest( + `${NAME} cannot name a ${request.audio.mediaType} upload; send mp3, mp4, m4a, wav, webm, ogg, or flac audio`, + ) + const audio = yield* MediaInput.inlineBytes(ADAPTER, request.audio) + const responseFormat = model.diarize + ? "diarized_json" + : request.timestamps === undefined || request.timestamps === "none" + ? undefined + : "verbose_json" + const native = Object.entries(mergeJsonRecords(request.providerOptions, request.http?.body) ?? {}).filter( + ([key]) => !RESERVED_FORM_FIELDS.has(key), + ) + const fields = mergeJsonRecords( + { + model: request.model.id, + language: model.languageField === "language" ? request.language : undefined, + languages: model.languageField === "languages" && request.language !== undefined ? [request.language] : undefined, + prompt: request.prompt, + response_format: responseFormat, + timestamp_granularities: responseFormat === "verbose_json" ? [request.timestamps] : undefined, + // Diarizing audio longer than 30 seconds requires a chunking strategy. + chunking_strategy: model.diarize ? "auto" : undefined, + stream: request.mode === "stream" ? true : undefined, + }, + Object.fromEntries(native), + ) + const form = new FormData() + form.append("file", MediaInput.blob(audio, request.audio.mediaType), `audio.${extension}`) + Object.entries(fields ?? {}).forEach(([key, value]) => appendField(form, key, value)) + return MediaProtocol.multipart(form) +}) + +// --------------------------------------------------------------------------- +// 6. Stream parsing +// --------------------------------------------------------------------------- + +const segment = (value: Schema.Schema.Type): TranscriptionSegment => ({ + text: value.text.trim(), + startSeconds: value.start, + endSeconds: value.end, + speaker: value.speaker, +}) + +const onEvent = Effect.fn("OpenAITranscription.onEvent")(function* (state: State, frame: string) { + const event = yield* decodeEvent(frame) + if (event.type === "transcript.text.done") return [{ ...state, transcript: event }, []] as const + if (event.type === "transcript.text.delta") + return [state, event.delta.length === 0 ? [] : [TranscriptionTextDeltaEvent.make({ delta: event.delta })]] as const + const next = segment(event) + state.segments.push(next) + return [state, [TranscriptionSegmentEvent.make({ segment: next })]] as const +}) + +const step = (state: State, frame: Frame) => + typeof frame === "string" + ? onEvent(state, frame) + : decodeTranscript(frame.document).pipe(Effect.map((transcript) => [{ ...state, transcript }, []] as const)) + +const usage = (value: Transcript["usage"]): MediaUsage | undefined => { + if (value === undefined) return undefined + if (value.type === "duration") return { type: "seconds", seconds: value.seconds } + return { + type: "tokens", + input: value.input_tokens, + output: value.output_tokens, + total: value.total_tokens, + details: { openai: value }, + } +} + +const finish = (state: State) => { + const transcript = state.transcript + if (transcript === undefined) return Effect.fail(MediaProtocol.incomplete(ADAPTER)) + const segments = transcript.segments?.map(segment) ?? state.segments + return Effect.succeed([ + TranscriptionFinishEvent.make({ + text: transcript.text, + segments: segments.length === 0 ? undefined : segments, + words: transcript.words?.map((word) => ({ text: word.word, startSeconds: word.start, endSeconds: word.end })), + language: (transcript.language ?? transcript.languages?.[0]?.code)?.toLowerCase(), + durationSeconds: transcript.duration, + usage: usage(transcript.usage), + }), + ]) +} + +// --------------------------------------------------------------------------- +// 7. Protocol and route +// --------------------------------------------------------------------------- + +export const protocol = MediaProtocol.stream({ + id: ADAPTER, + name: NAME, + unsupported: ["speakers"], + body: { from: fromRequest }, + frames: (bytes, context) => + context.request.mode === "stream" + ? Framing.sse.frame(bytes) + : Framing.document.frame(bytes).pipe(Stream.map((document) => ({ document }))), + initial: () => ({ segments: [] }), + step, + finish, +}) + +export const model = (input: MediaRoute.ModelInput) => + TranscriptionModel.fromRoute( + { id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: PATH }, + input, + ) + +export const OpenAITranscription = { + protocol, + model, +} as const diff --git a/packages/ai/src/protocols/utils/gemini-generate-content.ts b/packages/ai/src/protocols/utils/gemini-generate-content.ts new file mode 100644 index 00000000000..9ccd02d5362 --- /dev/null +++ b/packages/ai/src/protocols/utils/gemini-generate-content.ts @@ -0,0 +1,113 @@ +import { Effect, Schema, type Stream } from "effect" +import type { Media } from "../../media.js" +import { Framing } from "../../route/framing.js" +import type { MediaProtocol } from "../../route/media-protocol.js" +import { AIError, ContentPolicyError, ProviderID, type MediaUsage, type ProviderMetadata } from "../../schema/index.js" +import { ProviderShared } from "../shared.js" +import { MediaInput } from "./media-input.js" + +const PROVIDER = ProviderID.make("google") + +const UsageMetadata = Schema.Struct({ + promptTokenCount: Schema.optional(Schema.Number), + candidatesTokenCount: Schema.optional(Schema.Number), + totalTokenCount: Schema.optional(Schema.Number), +}) +type UsageMetadata = Schema.Schema.Type + +export const chunk = (part: Part) => + Schema.Struct({ + candidates: Schema.optional( + Schema.Array( + Schema.Struct({ + content: Schema.optional(Schema.Struct({ parts: Schema.optional(Schema.Array(part)) })), + finishReason: Schema.optional(Schema.String), + }), + ), + ), + promptFeedback: Schema.optional( + Schema.Struct({ + blockReason: Schema.optional(Schema.String), + blockReasonMessage: Schema.optional(Schema.String), + }), + ), + usageMetadata: Schema.optional(UsageMetadata), + modelVersion: Schema.optional(Schema.String), + responseId: Schema.optional(Schema.String), + }) + +interface Chunk { + readonly candidates?: ReadonlyArray<{ readonly finishReason?: string }> + readonly promptFeedback?: { readonly blockReason?: string; readonly blockReasonMessage?: string } + readonly usageMetadata?: UsageMetadata + readonly modelVersion?: string + readonly responseId?: string +} + +export interface Metadata { + readonly usage?: UsageMetadata + readonly finishReason?: string + readonly modelVersion?: string + readonly responseId?: string +} + +export const track = (state: State, chunk: Chunk): State => ({ + ...state, + usage: chunk.usageMetadata ?? state.usage, + finishReason: chunk.candidates?.[0]?.finishReason ?? state.finishReason, + modelVersion: chunk.modelVersion ?? state.modelVersion, + responseId: chunk.responseId ?? state.responseId, +}) + +export const blocked = (name: string, chunk: Chunk, frame: string) => { + const feedback = chunk.promptFeedback + if (feedback?.blockReason === undefined) return undefined + return new AIError({ + reason: new ContentPolicyError({ + message: `${name} blocked the request (${feedback.blockReason})${ + feedback.blockReasonMessage === undefined ? "" : `: ${feedback.blockReasonMessage}` + }`, + body: frame, + }), + }) +} + +export const usage = (usage: UsageMetadata | undefined): MediaUsage | undefined => + usage === undefined + ? undefined + : { + type: "tokens", + input: usage.promptTokenCount, + output: usage.candidatesTokenCount, + total: ProviderShared.totalTokens(usage.promptTokenCount, usage.candidatesTokenCount, usage.totalTokenCount), + details: { google: usage }, + } + +export const providerMetadata = (state: Metadata): ProviderMetadata => ({ + google: { finishReason: state.finishReason, modelVersion: state.modelVersion, responseId: state.responseId }, +}) + +export const path = (model: string, mode: MediaProtocol.Mode) => + mode === "stream" ? `/models/${model}:streamGenerateContent?alt=sse` : `/models/${model}:generateContent` + +// `generateContent` answers with one document shaped exactly like a streamed chunk, so it is a single frame. +export const frames = (bytes: Stream.Stream, mode: MediaProtocol.Mode) => + mode === "stream" ? Framing.sse.frame(bytes) : Framing.document.frame(bytes) + +// Gemini does not fetch public URLs; inline payloads and Gemini Files references are the accepted inputs. +export const mediaPart = ( + route: string, + asset: Media.Asset, +): Effect.Effect< + | { readonly fileData: { readonly mimeType: string; readonly fileUri: string } } + | { readonly inlineData: { readonly mimeType: string; readonly data: string } }, + AIError +> => { + const fileUri = MediaInput.refID(asset, PROVIDER) + if (fileUri !== undefined) return Effect.succeed({ fileData: { mimeType: asset.mediaType, fileUri } }) + return ProviderShared.requireInlineMedia(route, asset).pipe( + Effect.map((media) => ({ inlineData: { mimeType: media.mime, data: media.base64 } })), + ) +} + +export * as GeminiGenerateContent from "./gemini-generate-content.js" diff --git a/packages/ai/src/protocols/utils/media-input.ts b/packages/ai/src/protocols/utils/media-input.ts index 9a62e8e9b8d..2650ae5e669 100644 --- a/packages/ai/src/protocols/utils/media-input.ts +++ b/packages/ai/src/protocols/utils/media-input.ts @@ -1,5 +1,6 @@ import { Effect, Encoding } from "effect" import { Media } from "../../media.js" +import type { MediaProtocol } from "../../route/media-protocol.js" import type { AIError, ProviderID } from "../../schema/index.js" import { ProviderShared } from "../shared.js" @@ -13,6 +14,26 @@ export const inlineBytes = (route: string, asset: Media.Asset): Effect.Effect { + const buffer = new ArrayBuffer(data.byteLength) + new Uint8Array(buffer).set(data) + return new Blob([buffer], { type: mediaType }) +} + +const isScalar = (value: unknown): value is string | number | boolean => + typeof value === "string" || typeof value === "number" || typeof value === "boolean" + +export const query = (route: string, values: Record): Effect.Effect => { + const entries = Object.entries(values).filter(([, value]) => value !== undefined) + const invalid = entries.find(([, value]) => !isScalar(value) && !(Array.isArray(value) && value.every(isScalar))) + if (invalid !== undefined) + return Effect.fail(ProviderShared.invalidRequest(`${route} cannot send "${invalid[0]}" as a query parameter`)) + return Effect.succeed( + Object.fromEntries(entries.map(([key, value]) => [key, Array.isArray(value) ? value.map(String) : String(value)])), + ) +} + /** Provider file handle when the ref belongs to this provider; refs from other providers are never forwarded. */ export const refID = (asset: Media.Asset, provider: ProviderID) => asset.source.type === "ref" && asset.source.provider === provider ? asset.source.id : undefined diff --git a/packages/ai/src/providers/assemblyai.ts b/packages/ai/src/providers/assemblyai.ts new file mode 100644 index 00000000000..896984a1f65 --- /dev/null +++ b/packages/ai/src/providers/assemblyai.ts @@ -0,0 +1,43 @@ +import { Auth } from "../route/auth.js" +import type { ProviderAuthOption } from "../route/auth-options.js" +import { HttpOptions, ProviderID, type ModelID } from "../schema/index.js" +import { AssemblyAITranscription, DEFAULT_BASE_URL } from "../protocols/assemblyai-transcription.js" + +export type { AssemblyAITranscriptionOptions } from "../protocols/assemblyai-transcription.js" + +export const id = ProviderID.make("assemblyai") +const baseURL = DEFAULT_BASE_URL + +export type Config = ProviderAuthOption<"optional"> & { + /** `https://api.eu.assemblyai.com` for the EU region. */ + readonly baseURL?: string + readonly headers?: Record + readonly http?: HttpOptions.Input +} + +// The key is the whole `authorization` value, without a scheme. +const auth = (options: ProviderAuthOption<"optional">) => { + if ("auth" in options && options.auth) return options.auth + return Auth.optional("apiKey" in options ? options.apiKey : undefined, "apiKey") + .orElse(Auth.config("ASSEMBLYAI_API_KEY")) + .pipe(Auth.header("authorization")) +} + +export const configure = (input: Config = {}) => { + const transcription = (modelID: string | ModelID) => + AssemblyAITranscription.model({ + id: modelID, + auth: auth(input), + baseURL: input.baseURL ?? baseURL, + headers: input.headers, + http: input.http === undefined ? undefined : HttpOptions.make(input.http), + }) + return { + id, + transcription, + configure, + } +} + +export const provider = configure() +export const transcription = provider.transcription diff --git a/packages/ai/src/providers/deepgram.ts b/packages/ai/src/providers/deepgram.ts index c6553c25fe4..cfdc35b39c1 100644 --- a/packages/ai/src/providers/deepgram.ts +++ b/packages/ai/src/providers/deepgram.ts @@ -2,8 +2,10 @@ import { Auth } from "../route/auth.js" import type { ProviderAuthOption } from "../route/auth-options.js" import { HttpOptions, ProviderID, type ModelID } from "../schema/index.js" import { DEFAULT_BASE_URL, DeepgramSpeech } from "../protocols/deepgram-speech.js" +import { DeepgramTranscription } from "../protocols/deepgram-transcription.js" export type { DeepgramEncoding, DeepgramSpeechOptions } from "../protocols/deepgram-speech.js" +export type { DeepgramTranscriptionOptions } from "../protocols/deepgram-transcription.js" export const id = ProviderID.make("deepgram") const baseURL = DEFAULT_BASE_URL @@ -22,20 +24,21 @@ const auth = (options: ProviderAuthOption<"optional">) => { } export const configure = (input: Config = {}) => { - const speech = (modelID: string | ModelID) => - DeepgramSpeech.model({ - id: modelID, - auth: auth(input), - baseURL: input.baseURL ?? baseURL, - headers: input.headers, - http: input.http === undefined ? undefined : HttpOptions.make(input.http), - }) + const media = (modelID: string | ModelID) => ({ + id: modelID, + auth: auth(input), + baseURL: input.baseURL ?? baseURL, + headers: input.headers, + http: input.http === undefined ? undefined : HttpOptions.make(input.http), + }) return { id, - speech, + speech: (modelID: string | ModelID) => DeepgramSpeech.model(media(modelID)), + transcription: (modelID: string | ModelID) => DeepgramTranscription.model(media(modelID)), configure, } } export const provider = configure() export const speech = provider.speech +export const transcription = provider.transcription diff --git a/packages/ai/src/providers/google.ts b/packages/ai/src/providers/google.ts index 8fe99c58d6d..aba6636c155 100644 --- a/packages/ai/src/providers/google.ts +++ b/packages/ai/src/providers/google.ts @@ -6,10 +6,12 @@ import { HttpOptions, ProviderID, mergeHttpOptions, type ModelID } from "../sche import { Gemini } from "../protocols/gemini.js" import { GoogleImages } from "../protocols/google-images.js" import { GoogleSpeech } from "../protocols/google-speech.js" +import { GoogleTranscription } from "../protocols/google-transcription.js" import { GoogleVideo } from "../protocols/google-video.js" export type { GoogleImageOptions } from "../protocols/google-images.js" export type { GoogleSpeechOptions } from "../protocols/google-speech.js" +export type { GoogleTranscriptionOptions } from "../protocols/google-transcription.js" export type { GoogleVideoOptions } from "../protocols/google-video.js" export type GeminiOptionsInput = Gemini.OptionsInput export type GeminiProviderOptionsInput = Gemini.ProviderOptionsInput @@ -57,6 +59,7 @@ export const configure = (input: Config = {}) => { image: (modelID: string | ModelID) => GoogleImages.model(media(modelID)), video: (modelID: string | ModelID) => GoogleVideo.model(media(modelID)), speech: (modelID: string | ModelID) => GoogleSpeech.model(media(modelID)), + transcription: (modelID: string | ModelID) => GoogleTranscription.model(media(modelID)), configure, } } @@ -77,3 +80,4 @@ export const model: ProviderPackage.Definition { }) const image = (modelID: string | ModelID) => OpenAIImages.model(media(modelID)) const speech = (modelID: string | ModelID) => OpenAISpeech.model(media(modelID)) + const transcription = (modelID: string | ModelID) => OpenAITranscription.model(media(modelID)) return { id, @@ -117,6 +120,7 @@ export const configure = (input: Config = {}) => { chat, image, speech, + transcription, configure, } } @@ -164,3 +168,4 @@ export const responses = provider.responses export const chat = provider.chat export const image = provider.image export const speech = provider.speech +export const transcription = provider.transcription diff --git a/packages/ai/src/route/media-protocol.ts b/packages/ai/src/route/media-protocol.ts index f68d98c0dae..c898b4115f4 100644 --- a/packages/ai/src/route/media-protocol.ts +++ b/packages/ai/src/route/media-protocol.ts @@ -16,17 +16,37 @@ import { // Bodies // --------------------------------------------------------------------------- -/** JSON `query` is appended to the endpoint URL before the route and caller `http.query` overlays. */ -export type Body = - | { readonly type: "json"; readonly value: Record; readonly query?: Record } - | { readonly type: "multipart"; readonly value: FormData } +/** Array values become repeated parameters (`keyterm=a&keyterm=b`). */ +export type Query = Readonly>> -export const json = (value: Record, query?: Record): Body => ({ +/** `query` is appended to the endpoint URL before the route and caller `http.query` overlays. */ +export type Body = + | { readonly type: "json"; readonly value: Record; readonly query?: Query } + | { readonly type: "multipart"; readonly value: FormData } + | { + readonly type: "binary" + readonly value: Uint8Array + readonly contentType: string + readonly query?: Query + } + +export const json = (value: Record, query?: Query): Body => ({ type: "json", value, query, }) export const multipart = (value: FormData): Body => ({ type: "multipart", value }) +export const binary = (value: Uint8Array, contentType: string, query?: Query): Body => ({ + type: "binary", + value, + contentType, + query, +}) + +export type Send = (path: string, body: Body) => Effect.Effect + +/** Runs after unsupported-field rejection and before `body.from`, for providers that need an upload first. */ +export type Prepare = (request: Request, send: Send) => Effect.Effect // --------------------------------------------------------------------------- // Protocol kinds @@ -91,6 +111,7 @@ export interface Queued { /** Serializable handle. `Generation.token` carries the encoded form so it can be persisted and resumed elsewhere. */ readonly token: Schema.Codec readonly start: { + readonly prepare?: Prepare readonly body: { readonly from: (request: Request) => Effect.Effect } readonly decode: ( response: HttpClientResponse.HttpClientResponse, diff --git a/packages/ai/src/route/media.ts b/packages/ai/src/route/media.ts index d10faa790d1..1b66b723e9d 100644 --- a/packages/ai/src/route/media.ts +++ b/packages/ai/src/route/media.ts @@ -40,6 +40,7 @@ export interface ModelInput { /** One request, one response. */ export interface Route { + readonly kind: "inline" readonly id: string readonly provider: ProviderID readonly protocol: string @@ -48,6 +49,7 @@ export interface Route { /** Submit, then poll through the returned `Generation`. */ export interface QueuedRoute { + readonly kind: "queued" readonly id: string readonly provider: ProviderID readonly protocol: string @@ -62,6 +64,7 @@ export interface QueuedRoute { /** One request whose response parses into events; `generate` runs the same stream and collects it. */ export interface StreamRoute { + readonly kind: "stream" readonly id: string readonly provider: ProviderID readonly protocol: string @@ -95,14 +98,15 @@ export interface StreamInput( input: InlineInput, ): Route => { const transport = makeTransport(input) return { + kind: "inline", id: input.id, provider: transport.provider, protocol: input.protocol.id, @@ -157,7 +161,7 @@ export const queued = ( const start = Effect.fn("MediaRoute.start")(function* (request: Request, execute: Execute) { const submitted = yield* transport.submit( request, - { unsupported: protocol.unsupported, from: protocol.start.body.from }, + { unsupported: protocol.unsupported, prepare: protocol.start.prepare, from: protocol.start.body.from }, execute, ) const started = yield* protocol.start.decode(submitted.response, submitted.context) @@ -185,7 +189,7 @@ export const queued = ( return new Generation(route, encodeToken(token), yield* route.status) }) - return { id: input.id, provider: transport.provider, protocol: protocol.id, start, resume } + return { kind: "queued", id: input.id, provider: transport.provider, protocol: protocol.id, start, resume } } /** Compose a streaming media protocol; `generate` runs the same stream in `generate` mode and folds it with `collect`. */ @@ -236,6 +240,7 @@ export const stream = (input: Composition const provider = ProviderID.make(input.provider) const routeHttp = input.headers === undefined ? undefined : new HttpOptions({ headers: input.headers }) const authorize = Auth.toEffect(input.auth) - const withQuery = (url: URL, query: Record | undefined) => { - for (const [key, value] of Object.entries(query ?? {})) url.searchParams.set(key, value) - return url - } + const baseURL = (path: string) => new URL(`${ProviderShared.trimBaseUrl(input.endpoint.baseURL ?? "")}${path}`) + /** `auth` is only what `Auth` added, never deployment headers. */ + const send = Effect.fn("MediaRoute.send")(function* ( + call: { + readonly method: AuthInput["method"] + readonly url: URL + readonly headers: Headers.Headers + readonly request: AuthInput["request"] + readonly body?: MediaProtocol.Body + }, + execute: Execute, + ) { + const encoded = encode(call.body, call.headers) + const url = call.url.toString() + const headers = yield* authorize({ + request: call.request, + method: call.method, + url, + body: encoded.text, + headers: encoded.headers, + }) + const response = yield* execute( + encoded.apply(HttpClientRequest.make(call.method)(url).pipe(HttpClientRequest.setHeaders(headers))), + ) + return { response, auth: Object.fromEntries(Object.entries(headers).filter(([key]) => !(key in call.headers))) } + }) return { provider, /** Route and model overlays; `start` additionally merges the request's own `http`. */ @@ -266,67 +293,70 @@ const makeTransport = (input: Composition request: Request, protocol: { readonly unsupported?: ReadonlyArray + readonly prepare?: MediaProtocol.Prepare readonly from: (request: Request) => Effect.Effect }, execute: Execute, ) { yield* rejectUnsupported(input.id, provider, request, protocol.unsupported) const http = mergeHttpOptions(routeHttp, request.model.http, request.http) + const headers = Headers.fromInput(http?.headers) + const prepared = + protocol.prepare === undefined + ? request + : yield* protocol.prepare(request, (path, body) => + send({ method: "POST", url: baseURL(path), headers, request, body }, execute).pipe( + Effect.map((sent) => sent.response), + ), + ) // Sanitize after merging so model-level overlays are covered; the model value is restored, not sanitized. - const resolved: Request = { ...sanitizeSurrogates({ ...request, http }), model: request.model } + const resolved: Request = { ...sanitizeSurrogates({ ...prepared, http }), model: request.model } const body = yield* protocol.from(resolved) const url = withQuery( withQuery( Endpoint.render(input.endpoint, { request: resolved, body }), - body.type === "json" ? body.query : undefined, + body.type === "multipart" ? undefined : body.query, ), http?.query, ) - const encoded = body.type === "json" ? ProviderShared.encodeJson(body.value) : "[multipart/form-data]" - const baseHeaders = Headers.fromInput(http?.headers) - const headers = yield* authorize({ - request: resolved, - method: "POST", - url: url.toString(), - body: encoded, - // The HTTP client sets the multipart boundary; a caller-supplied content-type would corrupt it. - headers: body.type === "multipart" ? Headers.remove(baseHeaders, "content-type") : baseHeaders, - }) - const transport = HttpClientRequest.post(url.toString()).pipe( - HttpClientRequest.setHeaders(headers), - body.type === "json" - ? HttpClientRequest.bodyText(encoded, "application/json") - : HttpClientRequest.bodyFormData(body.value), - ) - const response = yield* execute(transport) - return { response, context: { request: resolved, body } } + const sent = yield* send({ method: "POST", url, headers, request: resolved, body }, execute) + return { response: sent.response, context: { request: resolved, body } } }), - /** - * Bodiless follow-up call (status, result, cancel) with the same auth and headers as `submit`. `auth` is only - * what `Auth` added, so protocols can hand download credentials to output assets without deployment headers. - */ - call: Effect.fn("MediaRoute.call")(function* ( - method: AuthInput["method"], - path: string, - http: HttpOptions | undefined, - execute: Execute, - ) { + /** Bodiless follow-up call (status, result, cancel) with the same auth and headers as `submit`. */ + call: (method: AuthInput["method"], path: string, http: HttpOptions | undefined, execute: Execute) => { // Provider-issued absolute URLs (fal `status_url`) are used as-is; everything else resolves against the base. - const url = withQuery( - /^https?:\/\//.test(path) - ? new URL(path) - : new URL(`${ProviderShared.trimBaseUrl(input.endpoint.baseURL ?? "")}${path}`), - http?.query, - ) + const url = withQuery(/^https?:\/\//.test(path) ? new URL(path) : baseURL(path), http?.query) for (const [key, value] of Object.entries(input.endpoint.query ?? {})) url.searchParams.set(key, value) - const base = Headers.fromInput(http?.headers) - const headers = yield* authorize({ request: { http }, method, url: url.toString(), body: "", headers: base }) - const response = yield* execute( - HttpClientRequest.make(method)(url.toString()).pipe(HttpClientRequest.setHeaders(headers)), - ) - const auth = Object.fromEntries(Object.entries(headers).filter(([key]) => !(key in base))) - return { response, auth } - }), + return send({ method, url, headers: Headers.fromInput(http?.headers), request: { http } }, execute) + }, + } +} + +const withQuery = (url: URL, query: MediaProtocol.Query | undefined) => { + for (const [key, value] of Object.entries(query ?? {})) { + url.searchParams.delete(key) + for (const item of typeof value === "string" ? [value] : value) url.searchParams.append(key, item) + } + return url +} + +const encode = (body: MediaProtocol.Body | undefined, headers: Headers.Headers) => { + if (body === undefined) return { text: "", headers, apply: (request: HttpClientRequest.HttpClientRequest) => request } + if (body.type === "json") { + const text = ProviderShared.encodeJson(body.value) + return { text, headers, apply: HttpClientRequest.bodyText(text, "application/json") } + } + if (body.type === "binary") + return { + text: `[${body.contentType}]`, + headers, + apply: HttpClientRequest.bodyUint8Array(body.value, body.contentType), + } + return { + text: "[multipart/form-data]", + // The HTTP client sets the multipart boundary; a caller-supplied content-type would corrupt it. + headers: Headers.remove(headers, "content-type"), + apply: HttpClientRequest.bodyFormData(body.value), } } diff --git a/packages/ai/src/transcription-client.ts b/packages/ai/src/transcription-client.ts new file mode 100644 index 00000000000..786540392c4 --- /dev/null +++ b/packages/ai/src/transcription-client.ts @@ -0,0 +1,122 @@ +import { Context, Effect, Layer, Stream } from "effect" +import { resultEvents, type AwaitOptions, type Generation } from "./generation.js" +import { ProviderShared } from "./protocols/shared.js" +import { RequestExecutor } from "./route/executor.js" +import type { AIError } from "./schema/index.js" +import { + responseEvents, + type TranscriptionEvent, + type TranscriptionModel, + type TranscriptionOptions, + type TranscriptionRequestFor, + type TranscriptionResponse, + type TranscriptionRoute, +} from "./transcription.js" + +export interface Interface { + readonly generate: ( + request: TranscriptionRequestFor, + options?: AwaitOptions, + ) => Effect.Effect + readonly stream: ( + request: TranscriptionRequestFor, + options?: AwaitOptions, + ) => Stream.Stream + readonly start: ( + request: TranscriptionRequestFor, + ) => Effect.Effect, AIError> + readonly resume: ( + model: TranscriptionModel, + token: unknown, + ) => Effect.Effect, AIError> +} + +export class Service extends Context.Service()("@opencode/TranscriptionClient") {} + +export const generate = ( + request: TranscriptionRequestFor, + options?: AwaitOptions, +): Effect.Effect => + Effect.gen(function* () { + const client = yield* Service + return yield* client.generate(request, options) + }) + +export const stream = ( + request: TranscriptionRequestFor, + options?: AwaitOptions, +): Stream.Stream => + Stream.unwrap( + Effect.gen(function* () { + const client = yield* Service + return client.stream(request, options) + }), + ) + +export const start = ( + request: TranscriptionRequestFor, +): Effect.Effect, AIError, Service> => + Effect.gen(function* () { + const client = yield* Service + return yield* client.start(request) + }) + +export const resume = ( + model: TranscriptionModel, + token: unknown, +): Effect.Effect, AIError, Service> => + Effect.gen(function* () { + const client = yield* Service + return yield* client.resume(model, token) + }) + +const notQueued = (route: TranscriptionRoute, operation: string) => + ProviderShared.unsupportedOperation({ + operation: `transcription.${operation}`, + provider: route.provider, + route: route.id, + message: `${route.provider}/${route.id} is not a queued route; use Transcription.generate or Transcription.stream`, + }) + +export const layer: Layer.Layer = Layer.effect( + Service, + Effect.gen(function* () { + const executor = yield* RequestExecutor.Service + const start = (request: TranscriptionRequestFor) => { + const route = request.model.route + if (route.kind !== "queued") return Effect.fail(notQueued(route, "start")) + return route.start(request, executor.execute) + } + return Service.of({ + start, + resume: (model, token) => { + const route = model.route + if (route.kind !== "queued") return Effect.fail(notQueued(route, "resume")) + return route.resume(model, token, executor.execute) + }, + generate: (request, options) => { + const route = request.model.route + if (route.kind !== "queued") return route.generate(request, executor.execute) + return start(request).pipe(Effect.flatMap((generation) => generation.await(options))) + }, + stream: (request, options) => { + const route = request.model.route + if (route.kind === "stream") return route.stream(request, executor.execute) + if (route.kind === "queued") + return Stream.unwrap( + start(request).pipe(Effect.map((generation) => resultEvents(generation, responseEvents, options))), + ) + return Stream.fromIterableEffect(Effect.map(route.generate(request, executor.execute), responseEvents)) + }, + }) + }), +) + +export const TranscriptionClient = { + Service, + layer, + generate, + stream, + start, + resume, +} as const diff --git a/packages/ai/src/transcription.ts b/packages/ai/src/transcription.ts new file mode 100644 index 00000000000..43ac82417f4 --- /dev/null +++ b/packages/ai/src/transcription.ts @@ -0,0 +1,294 @@ +import { Effect, Schema, Stream } from "effect" +import { Generation, ProgressEvent, QueuedEvent, type AwaitOptions } from "./generation.js" +import { Media } from "./media.js" +import { MediaModel, composeRoute, tryRequest } from "./media-model.js" +import { MediaRoute } from "./route/media.js" +import type { MediaProtocol } from "./route/media-protocol.js" +import { AIError, HttpOptions, MediaUsage, ProviderMetadata } from "./schema/index.js" +import { TranscriptionClient, Service } from "./transcription-client.js" + +// --------------------------------------------------------------------------- +// Model +// --------------------------------------------------------------------------- + +export type TranscriptionOptions = Record + +export type TranscriptionRoute = + | MediaRoute.Route, TranscriptionResponse> + | MediaRoute.StreamRoute, TranscriptionEvent, TranscriptionResponse> + | MediaRoute.QueuedRoute, TranscriptionResponse> + +export class TranscriptionModel extends MediaModel< + TranscriptionRoute, + Options +> { + declare protected readonly _TranscriptionModel: void + + static make( + input: MediaModel.Input>, + ) { + return new TranscriptionModel(input) + } + + /** The number of type arguments selects the kind: ``, ``, or ``. */ + static fromRoute( + route: TranscriptionModel.InlineRouteInput, + input: MediaRoute.ModelInput, + ): TranscriptionModel + static fromRoute( + route: TranscriptionModel.StreamRouteInput, + input: MediaRoute.ModelInput, + ): TranscriptionModel + static fromRoute( + route: TranscriptionModel.QueuedRouteInput, + input: MediaRoute.ModelInput, + ): TranscriptionModel + static fromRoute( + route: TranscriptionModel.RouteInput, + input: MediaRoute.ModelInput, + ) { + const composed: TranscriptionRoute = isStreamInput(route) + ? composeRoute((composition) => MediaRoute.stream({ ...composition, collect: collectResponse }), route, input) + : isQueuedInput(route) + ? composeRoute(MediaRoute.queued, route, input) + : composeRoute(MediaRoute.inline, route, input) + return new TranscriptionModel({ + id: input.id, + provider: route.provider, + http: input.http, + route: composed, + }) + } +} + +export namespace TranscriptionModel { + export type InlineRouteInput = MediaModel.RouteInput< + TranscriptionRequestFor, + MediaProtocol.Inline, TranscriptionResponse> + > + + export type StreamRouteInput< + Options extends TranscriptionOptions = TranscriptionOptions, + Frame = unknown, + State = unknown, + > = MediaModel.RouteInput< + MediaProtocol.Addressed>, + MediaProtocol.Streamed, TranscriptionEvent, Frame, State> + > + + export type QueuedRouteInput< + Options extends TranscriptionOptions = TranscriptionOptions, + Token = unknown, + > = MediaModel.RouteInput< + TranscriptionRequestFor, + MediaProtocol.Queued, TranscriptionResponse, Token> + > + + export type RouteInput< + Options extends TranscriptionOptions = TranscriptionOptions, + Frame = unknown, + State = unknown, + Token = unknown, + > = InlineRouteInput | StreamRouteInput | QueuedRouteInput +} + +const isStreamInput = ( + route: TranscriptionModel.RouteInput, +): route is TranscriptionModel.StreamRouteInput => route.protocol.kind === "stream" + +const isQueuedInput = ( + route: TranscriptionModel.RouteInput, +): route is TranscriptionModel.QueuedRouteInput => route.protocol.kind === "queued" + +export const TranscriptionModelSchema = Schema.declare( + (value): value is TranscriptionModel => value instanceof TranscriptionModel, + { expected: "Transcription.Model" }, +) + +// --------------------------------------------------------------------------- +// Request +// --------------------------------------------------------------------------- + +export const TranscriptionTimestamps = Schema.Literals(["none", "segment", "word"]) +export type TranscriptionTimestamps = Schema.Schema.Type + +export class TranscriptionRequest extends Schema.Class("Transcription.Request")({ + model: TranscriptionModelSchema, + audio: Media.AssetSchema, + language: Schema.optional(Schema.String), + prompt: Schema.optional(Schema.String), + /** Routes that cannot produce the requested granularity fail typed; routes may return more than asked. */ + timestamps: Schema.optional(TranscriptionTimestamps), + diarize: Schema.optional(Schema.Boolean), + speakers: Schema.optional(Schema.Int), + providerOptions: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)), + http: Schema.optional(HttpOptions), +}) { + declare protected readonly _TranscriptionRequest: void +} + +export type TranscriptionRequestFor = Omit< + TranscriptionRequest, + "model" | "providerOptions" +> & { + readonly model: TranscriptionModel + readonly providerOptions?: Options +} + +export type TranscriptionModelOptions = Model extends TranscriptionModel ? Options : never + +export type TranscriptionRequestInput = Omit< + ConstructorParameters[0], + "model" | "providerOptions" | "http" +> & { + readonly model: Model + readonly providerOptions?: NoInfer> + readonly http?: HttpOptions.Input +} + +// --------------------------------------------------------------------------- +// Response and events +// --------------------------------------------------------------------------- + +/** Speaker labels are provider-native (`A`, `0`, `spk:0`, or a known speaker name). */ +export const TranscriptionSegment = Schema.Struct({ + text: Schema.String, + startSeconds: Schema.Number, + endSeconds: Schema.Number, + speaker: Schema.optional(Schema.String), +}).annotate({ identifier: "Transcription.Segment" }) +export type TranscriptionSegment = Schema.Schema.Type + +export const TranscriptionWord = Schema.Struct({ + text: Schema.String, + startSeconds: Schema.Number, + endSeconds: Schema.Number, + speaker: Schema.optional(Schema.String), + confidence: Schema.optional(Schema.Number), +}).annotate({ identifier: "Transcription.Word" }) +export type TranscriptionWord = Schema.Schema.Type + +const transcriptFields = { + text: Schema.String, + segments: Schema.optional(Schema.Array(TranscriptionSegment)), + words: Schema.optional(Schema.Array(TranscriptionWord)), + /** Provider-native language as detected or echoed (`en`, `english`, `en_us`), lowercased but not normalized. */ + language: Schema.optional(Schema.String), + durationSeconds: Schema.optional(Schema.Number), + usage: Schema.optional(MediaUsage), + notices: Schema.optional(Schema.Array(Media.Notice)), + providerMetadata: Schema.optional(ProviderMetadata), +} + +export class TranscriptionResponse extends Schema.Class("Transcription.Response")( + transcriptFields, +) {} + +export const TranscriptionTextDeltaEvent = Schema.Struct({ + type: Schema.tag("text-delta"), + delta: Schema.String, +}).annotate({ identifier: "Transcription.Event.TextDelta" }) + +export const TranscriptionSegmentEvent = Schema.Struct({ + type: Schema.tag("segment"), + segment: TranscriptionSegment, +}).annotate({ identifier: "Transcription.Event.Segment" }) + +export const TranscriptionFinishEvent = Schema.Struct({ + type: Schema.tag("finish"), + ...transcriptFields, +}).annotate({ identifier: "Transcription.Event.Finish" }) + +const transcriptionEventTagged = Schema.Union([ + QueuedEvent, + ProgressEvent, + TranscriptionTextDeltaEvent, + TranscriptionSegmentEvent, + TranscriptionFinishEvent, +]).pipe(Schema.toTaggedUnion("type")) +export const TranscriptionEvent = Object.assign(transcriptionEventTagged, { + is: { + generationQueued: transcriptionEventTagged.guards["generation-queued"], + generationProgress: transcriptionEventTagged.guards["generation-progress"], + textDelta: transcriptionEventTagged.guards["text-delta"], + segment: transcriptionEventTagged.guards.segment, + finish: transcriptionEventTagged.guards.finish, + }, +}) +export type TranscriptionEvent = Schema.Schema.Type + +export const responseEvents = (response: TranscriptionResponse): ReadonlyArray => [ + TranscriptionFinishEvent.make({ ...response }), +] + +const collectResponse = (events: ReadonlyArray): Effect.Effect => { + const finish = events.find(TranscriptionEvent.is.finish) + // Every transcription protocol's `finish` emits the terminal event or fails, so a completed stream always has one. + if (finish === undefined) return Effect.die(new Error("The transcription stream completed without a finish event")) + const { type: _type, ...transcript } = finish + return Effect.succeed(new TranscriptionResponse(transcript)) +} + +// --------------------------------------------------------------------------- +// Request-shaped call API +// --------------------------------------------------------------------------- + +export function request( + input: TranscriptionRequestInput, +): TranscriptionRequestFor> +export function request(input: TranscriptionRequest): TranscriptionRequest +export function request(input: TranscriptionRequest | TranscriptionRequestInput) { + if (input instanceof TranscriptionRequest) return input + return new TranscriptionRequest({ + ...input, + http: input.http === undefined ? undefined : HttpOptions.make(input.http), + }) +} + +const requestEffect = (input: TranscriptionRequest | TranscriptionRequestInput) => tryRequest(() => request(input)) + +export function generate( + input: TranscriptionRequestInput, + options?: AwaitOptions, +): Effect.Effect +export function generate( + input: TranscriptionRequest, + options?: AwaitOptions, +): Effect.Effect +export function generate(input: TranscriptionRequest | TranscriptionRequestInput, options?: AwaitOptions) { + return requestEffect(input).pipe(Effect.flatMap((request) => TranscriptionClient.generate(request, options))) +} + +export function stream( + input: TranscriptionRequestInput, + options?: AwaitOptions, +): Stream.Stream +export function stream( + input: TranscriptionRequest, + options?: AwaitOptions, +): Stream.Stream +export function stream(input: TranscriptionRequest | TranscriptionRequestInput, options?: AwaitOptions) { + return Stream.unwrap(requestEffect(input).pipe(Effect.map((request) => TranscriptionClient.stream(request, options)))) +} + +/** Inline and streaming routes fail with `UnsupportedOperation`. */ +export function start( + input: TranscriptionRequestInput, +): Effect.Effect, AIError, Service> +export function start(input: TranscriptionRequest): Effect.Effect, AIError, Service> +export function start(input: TranscriptionRequest | TranscriptionRequestInput) { + return requestEffect(input).pipe(Effect.flatMap((request) => TranscriptionClient.start(request))) +} + +export const resume = ( + model: TranscriptionModel, + token: unknown, +): Effect.Effect, AIError, Service> => TranscriptionClient.resume(model, token) + +export const Transcription = { + request, + generate, + stream, + start, + resume, +} as const diff --git a/packages/ai/src/utils/media-type.ts b/packages/ai/src/utils/media-type.ts index 016d2179275..fb9d2c0a72e 100644 --- a/packages/ai/src/utils/media-type.ts +++ b/packages/ai/src/utils/media-type.ts @@ -38,6 +38,7 @@ const EXTENSIONS: Readonly> = { mp4: "video/mp4", webm: "video/webm", mp3: "audio/mpeg", + m4a: "audio/mp4", wav: "audio/wav", ogg: "audio/ogg", flac: "audio/flac", @@ -49,3 +50,18 @@ const EXTENSIONS: Readonly> = { export const extensionMediaType = (path: string): string | undefined => EXTENSIONS[path.slice(path.lastIndexOf(".") + 1).toLowerCase()] + +const EXTENSION_ALIASES: Readonly> = { + "audio/mp3": "mp3", + "audio/m4a": "m4a", + "audio/x-m4a": "m4a", + "audio/webm": "webm", + "audio/wave": "wav", + "audio/x-wav": "wav", + "audio/x-flac": "flac", +} + +export const mediaTypeExtension = (mediaType: string): string | undefined => { + const type = mediaType.split(";", 1)[0].trim().toLowerCase() + return EXTENSION_ALIASES[type] ?? Object.entries(EXTENSIONS).find(([, known]) => known === type)?.[0] +} diff --git a/packages/ai/src/video-client.ts b/packages/ai/src/video-client.ts index e63ebfa6e02..35dc276d4f0 100644 --- a/packages/ai/src/video-client.ts +++ b/packages/ai/src/video-client.ts @@ -1,9 +1,8 @@ import { Context, Effect, Layer, Stream } from "effect" -import type { AwaitOptions, Generation } from "./generation.js" +import { resultEvents, type AwaitOptions, type Generation } from "./generation.js" import { RequestExecutor } from "./route/executor.js" import type { AIError } from "./schema/index.js" import { - isObservation, responseEvents, type VideoEvent, type VideoModel, @@ -69,15 +68,6 @@ export const stream = ( }), ) -/** Status observations until the first terminal one, then the fetched result expanded into events. */ -const events = (generation: Generation, options: AwaitOptions | undefined) => - generation - .events(options) - .pipe( - Stream.filter(isObservation), - Stream.concat(Stream.fromIterableEffect(Effect.map(generation.result(), responseEvents))), - ) - export const layer: Layer.Layer = Layer.effect( Service, Effect.gen(function* () { @@ -89,7 +79,9 @@ export const layer: Layer.Layer = Layer resume: (model, token) => model.route.resume(model, token, executor.execute), generate: (request, options) => start(request).pipe(Effect.flatMap((generation) => generation.await(options))), stream: (request, options) => - Stream.unwrap(start(request).pipe(Effect.map((generation) => events(generation, options)))), + Stream.unwrap( + start(request).pipe(Effect.map((generation) => resultEvents(generation, responseEvents, options))), + ), }) }), ) diff --git a/packages/ai/src/video.ts b/packages/ai/src/video.ts index f3ae17670c4..9f87a082ff4 100644 --- a/packages/ai/src/video.ts +++ b/packages/ai/src/video.ts @@ -1,5 +1,5 @@ import { Effect, Schema, Stream } from "effect" -import { Generation, type AwaitOptions, type Event as GenerationEvent } from "./generation.js" +import { Generation, ProgressEvent, QueuedEvent, type AwaitOptions } from "./generation.js" import { Media } from "./media.js" import { MediaModel, composeRoute, tryRequest } from "./media-model.js" import { MediaRoute } from "./route/media.js" @@ -123,18 +123,6 @@ export class VideoResponse extends Schema.Class("Video.Response") } } -export const VideoQueuedEvent = Schema.Struct({ - type: Schema.tag("generation-queued"), - id: Schema.String, - position: Schema.optional(Schema.Number), -}).annotate({ identifier: "Video.Event.Queued" }) - -export const VideoProgressEvent = Schema.Struct({ - type: Schema.tag("generation-progress"), - id: Schema.String, - progress: Schema.optional(Schema.Number), -}).annotate({ identifier: "Video.Event.Progress" }) - export const VideoOutputEvent = Schema.Struct({ type: Schema.tag("video"), index: Schema.Number, @@ -148,7 +136,7 @@ export const VideoFinishEvent = Schema.Struct({ providerMetadata: Schema.optional(ProviderMetadata), }).annotate({ identifier: "Video.Event.Finish" }) -const videoEventTagged = Schema.Union([VideoQueuedEvent, VideoProgressEvent, VideoOutputEvent, VideoFinishEvent]).pipe( +const videoEventTagged = Schema.Union([QueuedEvent, ProgressEvent, VideoOutputEvent, VideoFinishEvent]).pipe( Schema.toTaggedUnion("type"), ) export const VideoEvent = Object.assign(videoEventTagged, { @@ -171,10 +159,6 @@ export const responseEvents = (response: VideoResponse): ReadonlyArray => - event.type !== "generation-finished" - // --------------------------------------------------------------------------- // Request-shaped call API // --------------------------------------------------------------------------- diff --git a/packages/ai/test/exports.test.ts b/packages/ai/test/exports.test.ts index 21280fcb7f8..e241df07857 100644 --- a/packages/ai/test/exports.test.ts +++ b/packages/ai/test/exports.test.ts @@ -17,6 +17,7 @@ import { import { Route, Protocol, WebSocketTransport } from "@opencode/ai/route" import { Provider as ProviderSubpath } from "@opencode/ai/provider" import { + AssemblyAI, Baseten, Cartesia, CloudflareAIGateway, @@ -122,6 +123,10 @@ describe("public exports", () => { expect(ElevenLabs.configure({ apiKey: "fixture" }).speech("eleven_flash_v2_5").route.id).toBe("elevenlabs-speech") expect(Cartesia.configure({ apiKey: "fixture" }).speech("sonic-3").route.id).toBe("cartesia-speech") expect(Deepgram.configure({ apiKey: "fixture" }).speech("aura-2-thalia-en").route.id).toBe("deepgram-speech") + expect(OpenAI.configure({ apiKey: "fixture" }).transcription("gpt-transcribe").route.kind).toBe("stream") + expect(Google.configure({ apiKey: "fixture" }).transcription("gemini-3.5-transcribe").route.kind).toBe("stream") + expect(Deepgram.configure({ apiKey: "fixture" }).transcription("nova-3").route.kind).toBe("inline") + expect(AssemblyAI.configure({ apiKey: "fixture" }).transcription("universal-3-5-pro").route.kind).toBe("queued") }) test("protocol barrels expose supported low-level routes", () => { diff --git a/packages/ai/test/fixtures/audio/dialog.mp3 b/packages/ai/test/fixtures/audio/dialog.mp3 new file mode 100644 index 00000000000..12ea29ed8cc Binary files /dev/null and b/packages/ai/test/fixtures/audio/dialog.mp3 differ diff --git a/packages/ai/test/fixtures/audio/hello.mp3 b/packages/ai/test/fixtures/audio/hello.mp3 new file mode 100644 index 00000000000..94bb4fc281d Binary files /dev/null and b/packages/ai/test/fixtures/audio/hello.mp3 differ diff --git a/packages/ai/test/fixtures/recordings/assemblyai-transcription/uploads-submits-and-polls-a-transcript.json b/packages/ai/test/fixtures/recordings/assemblyai-transcription/uploads-submits-and-polls-a-transcript.json new file mode 100644 index 00000000000..ca316c0f63f --- /dev/null +++ b/packages/ai/test/fixtures/recordings/assemblyai-transcription/uploads-submits-and-polls-a-transcript.json @@ -0,0 +1,98 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:assemblyai-transcription", + "provider:assemblyai", + "protocol:assemblyai-transcription" + ], + "name": "assemblyai-transcription/uploads-submits-and-polls-a-transcript", + "recordedAt": "2026-09-23T14:03:07.504Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.eu.assemblyai.com/v2/upload", + "headers": { + "content-type": "application/octet-stream" + }, + "body": "[audio]" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"upload_url\":\"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\"}\n" + } + }, + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.eu.assemblyai.com/v2/transcript", + "headers": { + "content-type": "application/json" + }, + "body": "{\"audio_url\":\"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\",\"speech_models\":[\"universal-3-5-pro\"],\"language_detection\":true,\"speaker_labels\":true}" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json; charset=UTF-8" + }, + "body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": null, \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"processing\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": null, \"words\": null, \"utterances\": null, \"confidence\": null, \"audio_duration\": null, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": null, \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {}, \"iab_categories_result\": {}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": null, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": null, \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}" + } + }, + { + "transport": "http", + "request": { + "method": "GET", + "url": "https://api.eu.assemblyai.com/v2/transcript/a1de21ad-a178-4c28-b8f0-4b0334b72126", + "headers": {}, + "body": "" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json; charset=UTF-8" + }, + "body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": null, \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"processing\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": null, \"words\": null, \"utterances\": null, \"confidence\": null, \"audio_duration\": null, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": [], \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {}, \"iab_categories_result\": {}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": null, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": null, \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}" + } + }, + { + "transport": "http", + "request": { + "method": "GET", + "url": "https://api.eu.assemblyai.com/v2/transcript/a1de21ad-a178-4c28-b8f0-4b0334b72126", + "headers": {}, + "body": "" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json; charset=UTF-8" + }, + "body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": \"en\", \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"completed\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": \"Hello from Open Code.\", \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}], \"utterances\": [{\"speaker\": \"A\", \"text\": \"Hello from Open Code.\", \"confidence\": 0.85656375, \"start\": 445, \"end\": 1455, \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}]}], \"confidence\": 0.85656375, \"audio_duration\": 2, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": [], \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"iab_categories_result\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": 0.9934, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": \"universal-3-5-pro\", \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}" + } + }, + { + "transport": "http", + "request": { + "method": "GET", + "url": "https://api.eu.assemblyai.com/v2/transcript/a1de21ad-a178-4c28-b8f0-4b0334b72126", + "headers": {}, + "body": "" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json; charset=UTF-8" + }, + "body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": \"en\", \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"completed\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": \"Hello from Open Code.\", \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}], \"utterances\": [{\"speaker\": \"A\", \"text\": \"Hello from Open Code.\", \"confidence\": 0.85656375, \"start\": 445, \"end\": 1455, \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}]}], \"confidence\": 0.85656375, \"audio_duration\": 2, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": [], \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"iab_categories_result\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": 0.9934, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": \"universal-3-5-pro\", \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/deepgram-transcription/splits-utterances-at-speaker-changes.json b/packages/ai/test/fixtures/recordings/deepgram-transcription/splits-utterances-at-speaker-changes.json new file mode 100644 index 00000000000..1132ead353c --- /dev/null +++ b/packages/ai/test/fixtures/recordings/deepgram-transcription/splits-utterances-at-speaker-changes.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:deepgram-transcription", + "provider:deepgram", + "protocol:deepgram-transcription" + ], + "name": "deepgram-transcription/splits-utterances-at-speaker-changes", + "recordedAt": "2026-09-23T13:46:35.886Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.deepgram.com/v1/listen?model=nova-3&smart_format=true&detect_language=true&diarize_model=latest&utterances=true", + "headers": { + "content-type": "audio/mpeg" + }, + "body": "[audio]" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"metadata\":{\"transaction_key\":\"deprecated\",\"request_id\":\"01a0ce84-a81d-7161-8505-c08d5e0c25bb\",\"sha256\":\"12e9ccb9d346c28b071c2fab7ce48556d0cc115af6850aacb338e7548807cb80\",\"created\":\"2026-09-23T13:46:35.566Z\",\"duration\":3.552625,\"channels\":1,\"models\":[\"2187e11a-3532-4498-b076-81fa530bdd49\"],\"model_info\":{\"2187e11a-3532-4498-b076-81fa530bdd49\":{\"name\":\"general-nova-3\",\"version\":\"2025-07-31.0\",\"arch\":\"nova-3\"}},\"diarize_info\":{\"model_uuid\":\"6946b038-8264-4505-acf7-9822c009fecb\",\"arch\":\"v2\"}},\"results\":{\"channels\":[{\"alternatives\":[{\"transcript\":\"Did the release ship? Yes. It shipped this morning.\",\"confidence\":1.0,\"words\":[{\"word\":\"did\",\"start\":0.08,\"end\":0.48,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Did\"},{\"word\":\"the\",\"start\":0.48,\"end\":0.64,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"the\"},{\"word\":\"release\",\"start\":0.64,\"end\":0.96,\"confidence\":0.99902344,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"release\"},{\"word\":\"ship\",\"start\":0.96,\"end\":1.52,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"ship?\"},{\"word\":\"yes\",\"start\":1.5999999,\"end\":2.32,\"confidence\":0.9980469,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"Yes.\"},{\"word\":\"it\",\"start\":2.32,\"end\":2.56,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"It\"},{\"word\":\"shipped\",\"start\":2.56,\"end\":2.8799999,\"confidence\":0.9707031,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"shipped\"},{\"word\":\"this\",\"start\":2.8799999,\"end\":3.1999998,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"this\"},{\"word\":\"morning\",\"start\":3.1999998,\"end\":3.52,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"morning.\"}],\"paragraphs\":{\"transcript\":\"\\nSpeaker 0: Did the release ship?\\n\\nSpeaker 1: Yes. It shipped this morning.\",\"paragraphs\":[{\"sentences\":[{\"text\":\"Did the release ship?\",\"start\":0.08,\"end\":1.52}],\"speaker\":0,\"num_words\":4,\"start\":0.08,\"end\":1.52},{\"sentences\":[{\"text\":\"Yes.\",\"start\":1.5999999,\"end\":2.32},{\"text\":\"It shipped this morning.\",\"start\":2.32,\"end\":3.52}],\"speaker\":1,\"num_words\":5,\"start\":1.5999999,\"end\":3.52}]}}],\"detected_language\":\"en\",\"language_confidence\":0.917855}],\"utterances\":[{\"start\":0.08,\"end\":3.52,\"confidence\":0.99641925,\"channel\":0,\"transcript\":\"Did the release ship? Yes. It shipped this morning.\",\"words\":[{\"word\":\"did\",\"start\":0.08,\"end\":0.48,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Did\"},{\"word\":\"the\",\"start\":0.48,\"end\":0.64,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"the\"},{\"word\":\"release\",\"start\":0.64,\"end\":0.96,\"confidence\":0.99902344,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"release\"},{\"word\":\"ship\",\"start\":0.96,\"end\":1.52,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"ship?\"},{\"word\":\"yes\",\"start\":1.5999999,\"end\":2.32,\"confidence\":0.9980469,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"Yes.\"},{\"word\":\"it\",\"start\":2.32,\"end\":2.56,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"It\"},{\"word\":\"shipped\",\"start\":2.56,\"end\":2.8799999,\"confidence\":0.9707031,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"shipped\"},{\"word\":\"this\",\"start\":2.8799999,\"end\":3.1999998,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"this\"},{\"word\":\"morning\",\"start\":3.1999998,\"end\":3.52,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"morning.\"}],\"speaker\":1,\"id\":\"67ba8055-e541-4089-abfa-0163558083cc\"}]}}" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/deepgram-transcription/transcribes-raw-audio-with-speakers-and-word-timestamps.json b/packages/ai/test/fixtures/recordings/deepgram-transcription/transcribes-raw-audio-with-speakers-and-word-timestamps.json new file mode 100644 index 00000000000..90e04abcafd --- /dev/null +++ b/packages/ai/test/fixtures/recordings/deepgram-transcription/transcribes-raw-audio-with-speakers-and-word-timestamps.json @@ -0,0 +1,50 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:deepgram-transcription", + "provider:deepgram", + "protocol:deepgram-transcription" + ], + "name": "deepgram-transcription/transcribes-raw-audio-with-speakers-and-word-timestamps", + "recordedAt": "2026-09-23T12:32:18.578Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.deepgram.com/v1/listen?model=nova-3&smart_format=true&detect_language=true&diarize_model=latest&utterances=true", + "headers": { + "content-type": "audio/mpeg" + }, + "body": "[audio]" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"metadata\":{\"transaction_key\":\"deprecated\",\"request_id\":\"01a0ce40-a44f-73b0-9d99-cdfc4f184268\",\"sha256\":\"95495edaebf6b10b2877de8c7b9eb7372de315c105fa8e73ed656ef2ec28ca02\",\"created\":\"2026-09-23T12:32:18.036Z\",\"duration\":1.5411875,\"channels\":1,\"models\":[\"2187e11a-3532-4498-b076-81fa530bdd49\"],\"model_info\":{\"2187e11a-3532-4498-b076-81fa530bdd49\":{\"name\":\"general-nova-3\",\"version\":\"2025-07-31.0\",\"arch\":\"nova-3\"}},\"diarize_info\":{\"model_uuid\":\"6946b038-8264-4505-acf7-9822c009fecb\",\"arch\":\"v2\"}},\"results\":{\"channels\":[{\"alternatives\":[{\"transcript\":\"Hello from OpenCode.\",\"confidence\":0.96972656,\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"paragraphs\":{\"transcript\":\"\\nSpeaker 0: Hello from OpenCode.\",\"paragraphs\":[{\"sentences\":[{\"text\":\"Hello from OpenCode.\",\"start\":0.16,\"end\":1.4399999}],\"speaker\":0,\"num_words\":3,\"start\":0.16,\"end\":1.4399999}]}}],\"detected_language\":\"en\",\"language_confidence\":0.9998003}],\"utterances\":[{\"start\":0.16,\"end\":1.4399999,\"confidence\":0.96875,\"channel\":0,\"transcript\":\"Hello from OpenCode.\",\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"speaker\":0,\"id\":\"43d717d5-ca00-48c1-a39c-381be0b1dad2\"}]}}" + } + }, + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.deepgram.com/v1/listen?model=nova-3&smart_format=true&detect_language=true&diarize_model=latest&utterances=true", + "headers": { + "content-type": "audio/mpeg" + }, + "body": "[audio]" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"metadata\":{\"transaction_key\":\"deprecated\",\"request_id\":\"01a0ce40-a609-78c2-8a9d-806e3ff8db0b\",\"sha256\":\"95495edaebf6b10b2877de8c7b9eb7372de315c105fa8e73ed656ef2ec28ca02\",\"created\":\"2026-09-23T12:32:18.478Z\",\"duration\":1.5411875,\"channels\":1,\"models\":[\"2187e11a-3532-4498-b076-81fa530bdd49\"],\"model_info\":{\"2187e11a-3532-4498-b076-81fa530bdd49\":{\"name\":\"general-nova-3\",\"version\":\"2025-07-31.0\",\"arch\":\"nova-3\"}},\"diarize_info\":{\"model_uuid\":\"6946b038-8264-4505-acf7-9822c009fecb\",\"arch\":\"v2\"}},\"results\":{\"channels\":[{\"alternatives\":[{\"transcript\":\"Hello from OpenCode.\",\"confidence\":0.96972656,\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"paragraphs\":{\"transcript\":\"\\nSpeaker 0: Hello from OpenCode.\",\"paragraphs\":[{\"sentences\":[{\"text\":\"Hello from OpenCode.\",\"start\":0.16,\"end\":1.4399999}],\"speaker\":0,\"num_words\":3,\"start\":0.16,\"end\":1.4399999}]}}],\"detected_language\":\"en\",\"language_confidence\":0.9998003}],\"utterances\":[{\"start\":0.16,\"end\":1.4399999,\"confidence\":0.96875,\"channel\":0,\"transcript\":\"Hello from OpenCode.\",\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"speaker\":0,\"id\":\"6f46ea75-616b-4209-b644-aae3e8f1e3c0\"}]}}" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/google-transcription/streams-speaker-turns.json b/packages/ai/test/fixtures/recordings/google-transcription/streams-speaker-turns.json new file mode 100644 index 00000000000..70ea84c329b --- /dev/null +++ b/packages/ai/test/fixtures/recordings/google-transcription/streams-speaker-turns.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:google-transcription", + "provider:google", + "protocol:google-transcription" + ], + "name": "google-transcription/streams-speaker-turns", + "recordedAt": "2026-09-23T13:04:11.224Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-transcribe:streamGenerateContent?alt=sse", + "headers": { + "content-type": "application/json" + }, + "body": "{\"contents\":[{\"role\":\"user\",\"parts\":[{\"inlineData\":{\"mimeType\":\"audio/mpeg\",\"data\":\"//NAxAARMHY0AMsMJTyMAqIOTWHWOu8D4lk8mGBwACBAgQIEyZMnd2TtCAAAAAMPDw8PAAAAAAMPDw8PAAAAAAMPDw/YAAOAHf+YeAAHwFf8PQ1IcgAgGXgcE04UmsFKseuFA08QgqD/80LEFg/QNjgM4wQIqCoBEQoFApKtQtNTF/U8sj37+dxVz1e7XLX+215BJJFvYb9epThev4hVCbrr6oO/3L+UwWlRIyJWKu13NuTZJrfiu3+vVpfPo1mjSbE7k3ZDRf9f77ff/d71gcn/80DEMgvgHkT00gwAyQn1MM3fWZRj64i3fhpqehEbePQdNmlMYpiAOTnVxssQkGOEbHqMHGJ8XL9mxQuBl0yE+fH/iz7UL7GtjtRApMqJMdvWwYTMXJZGKxqfN1rDVnQ6obXGFRoqPP/zQsRdEbh+MALiRiDBDeTLWFrxWKQG6TqPoQN+6YW7a60AssagckLEqibUtaQmlsQjoVussapPabUz3m4V1YPeAUSn3Fg5KEbHZRwHBpJgQ2tGOZtzSLdS1L1MX7qryVLN+xVu7McL6v/zQMRyEXBSNALTBghaP3UU3+NrvWwwni6KFNVMHZmYQTRUCkgFUiZacULPPh5HWx5XuFpY6/h44P4ESlrlmMe2lDD3e5NXS5qFIDayFFPOV/n2EfWOQv4yM3ojDR10jYnFATUNpxZT//NCxIcOIC5ALNmEBOpGUhENN2LKIWG9/U+GSLYl5n5GX1ixJNFYHEEotG1Th8JWTy9J9SUJP6N7lij21oWB9sXonrYpDAf1JJdNPeB4EBBWMAJMCY8JzKRHxPuMEdU61YpNlqwpnwRv//NAxKoPsCJENNmGACShzRyHrFKhQiNrNwotAUzZvDjVCc0tCx+rcjMKH1uaketBE4bZFM29vDqR8aiofUlSVRGMXqWUs5Eeu0zPoKWMwg3gcVRhBodHpKZXS67jkH+mRlCXp38+zBr/80LExhMpEjQA2kYMRCcLg08QPSWCCkPYNGKaBY0ORdyses9apM6KjxvFb8YMIoGXCUwtYrAD2ELyt9iQ2hnSqgcB0G5XnKHDlU1lhL8u3KKXy6Wz9JUx72oPC6CIU6vm7hQdpmnnc63/80DE1RQQrjQA4kYI+fSyrKf93QuGbE5CmI3BUPuCU93FpSczui3c0Qy56e2aeRP47PUfX/prV96jGiFyMn1Hn5oxPevfb9exl/bV+n/f/f/3+/357LZjSfnGnIh+UpU4Im/sYbMLmv/zQsTfFikOOArbxgS+phXC8CXmaJs2F5FsHMKB4wfcuDzHIXFb0zdnGHCeBvCelJaKW/2HMPEeYvhVBxj/QzdezWa0mGZImSJRLxQHh+pffffy+SBwzMCSMTg9iMbjzHJ9Vlfb/75Jmv/zQMTiGjIyOAlbGAGOwxGxMvkqUy4mUx6HDqul/X////LxQcuAP/aE6sAHkGhH+llGQqAFage5ZS/MeLnhSOTk24SKtafeBcO6rOzl/PcA5e32YPM3tQ45+XGE9HYFp2FZ+8hiWAtN//NCxNQlU372X49oAltWsT3Xm5DPr0ucio9vn1bps98fTuMVnJpfrzr77Nt44NC2xE1l5npm1rV5xwUFMd/KnJ4VVz9vg8eFLcdoV/bdBSh36Bnhf9si64XVgMcB0OBjKYCXaBncW/Iw//NAxJoiigaVhdpgAfrLogZdgC1/E6axk4ECTn1JQl8/FPSZwYo7Tcq23oXbUtSijgmUc/Ks0SB7dqWOwFASOQxiel6Ps9upnkuarYv/koFO58mWGizYGq8rdtcidxdX1DeYgpjqcxT/80LEaiUSCo0E3ha8FWkAA+ASJlHb1D/3B25iDyrJD7hUXQFhS8XPGOsoWEAjapm084gMU+uJFYAFEgSa9thS1KGgi53BoawL8B22RJFSOrJ6XAiQGpFhKLnQSJV/6rP5mf39V6WxwCX/80DEMRuh5qG+yYS1HJMGdsolHqFAUo6BhT/KVjKUxlbQ3/zKAiSlLwxjuBP/i6X/yTTdN8UFdFQXf4rjeUxQrKSO/rvyx6b8IKUxeAFDERAMWBDMmW/PIihMIg2AQymDgWGGQnGGYP/zQsQdGUE2QAFdSAAyXqLKtRhgCKVbizavBIKnqb6zFUqmnZ3NBUMqefyJWak0KZQQAuWCoIwFKFD89bd7lf5fV5oKmaTsLViZcs5Z1y4KuHf//+lUg7z6E9eqQUFVT2Q6/v851X+LBP/zQMQUGPsKkAGPUACH/gMAUBY/x+hINP/AnAvEQLH//C8EQLAFcL9xF///iLG4Xg8ZB4SA3////4N5OeRk4sCoLEwkC/H/////+TqRnjwWGMPcRY/A4f///4IF0ICjrapv3U3VrxRU//NCxAsXOd64AZlAANG04zD1mZ5MSna7WLJJl0YTPEqYrlwZBWbSUIkCYG4IhU6iVEw+pgJRGLyOHzjphR41/lbMuOuakp4J//nn1Lt+sptaWCw6DXu3GfLBosCsKimEBh0IA21ruL0q//NAxAoT4Xq5b9hAAj0HvDOuwbrsycae+nkP/r63f5umBcdbEB+tILiFczAhh6sWiw12ldWrNwsM1+tSvqvzysxNXoWbni2IpYRZKVV////46ZWmRCrgAEPznbhCkXq4sMJJDuj6RTn/80LEFRP5cpj2w8Z2WXxgD9QmSCijdQnUhPBulx3mIZSuZpJjdY0N1KrYl3ARPsyl/sZN1f8v534zmWsNVwzetyiqEVjCjDIENSU5aKAF4/liAdYKoteeRqZK+LYPTGZQEIOU4oi+cSv/80DEIRRQxoGe08Z0qYXAx30XcoznKJQdjZNAnbY6FAMKPkEG2ISjzB58BPWUUfOnzd5qnOyKG//+ijaBqvNMypZcYS6cBIpiBgaCwBUGObgIctWRghZiFDsvksZTHcetfiRbdl8U3P/zQsQqE0jWgAjWGHCZG4zAXoXlw0gEAJWApAQHRbhiAGI/nJ0eUcTSPe679TIIRtUhCA0TDgAEh5hbdweyskUwf2Smd3JcxaJMxFQfe/nqqt/KRJZEpeIO9YtLoHYW13JI0ESL8QyU3f/zQMQ4FCECoh7T2Kwek2ChHSqd/OmFzymMp0b/J1O0QfqB8eH11jcoSQoKCW6NPgMwAo52siGwPjDBgArza1p2a3/vBWH6sML7MAGiY44mCcAyzpUQNcyNp2/o2X+VWX6VcPk1Cm6k//NCxEIUif6YANLK8PX2H7ovO/Qa+oebjX6D89UqSGqgFYIlAD9Z94/Q9+7bqOmXODy426nbj+S8WSyrVOvmRLTWleFDAcxgK2pW+YzoIdE+/Uvq3b/0bobzeK9CPqrd21KKNB9y0Z6W//NAxEsTMgKlvsvKcP+iYojyP09jf+GHbtRDuicztXsZiIChXLmCfIahBSN1C/0xul46wcR9nUDbgn6N7+f09X4I/dfX/q/07+DZs7dTenr/z+oqwiXG6JzP3uzf/vuqm5k5zmSMp3D/80LEWRQSYqpew0R0CnnCfhkZk5gacXmb4FV7+Mahn/DxkFhkIxiJPd8Fg4WAAAEEED1xF/+72O0Z7//bvzx0QfhWchGCCHB96ylAYatsmtP/p//0ZCqh5/cAunD9LWdEMNG73O7lWqf/80DEZBRRTpwqywxkezeQSLNPzMOIS4WNSt+tELKQjkE7TtsPSbVCweziIXV5FtEZ2j9p1suHaGzvGQ2E0+0jejgerYAAa03mEa7v/XiLHrZUDvTTJ8vmOBmepX5SkbmIyVn1KX9Flf/zQsRtEinSrADITLELthBMhJSYKKECJGTlEChE2gLrqSuF77+bVfPGdSjC39WSkzUkKReB6bL46Vx0rMm+1sJRe1DrX+F2Zjh7vKHsLG5+0EvxL1HGxmCys/xnErwbdhIFBQWDh8hf5v/zQMSAFHnmukrAkpnbTGQEitM7mf/38/Ov5bv//9aVW///0pnQekZ/0YgHttbGUmIA5w7a3Fjlp/jrxIFpmkptkJoo9xGrChMqHu+MuyFS4BvmlhaWRfW4qjTKGAYqVGujSNZLhV6A//NCxIkSusbqXnhFHmWNa+SF1mRMuQ0//+hfpdDLt//mG59gCFx7E17YjG1MfcRV+lonEKhlqewpUDboUG2yDXubgjeD///Mb/5t1gUC/+YyA8ACWQN5AAhOJErSjC+Zr31jp/ZhyEEy//NAxJoTMRrWXnjLBtggijco2+wsXn/3zv65gs69nJmd+aUo2v+85OVfxAB4YCHDsD4HgE/odo65t///fXh/0HpUNfN5IWufEFVAeeKBYuRzdnCypKGJyTNxVRpasUJiRikkkW0S99z/80LEqBuzOwZeQFlz7ILMyXabkxJnLgzNNadort3RrkhgnZcIzqMSF4FicohEiZiaASYykSlrRQnzcziaIrBEhLVEbVJvPZKxbyUL3X7We/6VeIeHl2ZrEjXglfcIpIp7WSRX8qQpTBL/80DElR9jLsWMONMRUNrSt7Dh2TRcghnYmlwiA6GMmRZIbM70rFhBXQw0wLilymxiqb6XFydj69X113hJiOcecUR3etWc2rq6uH0IAWBacgMnB7wKpEIdXAaxPBJ2gIpDgHeEFpAITf/zQsRyE5DrBxxKRhYBcikAOOGyCzqbGBx0UjSYs0Fnn0ni66oQJSCkZWfpWQV6bepznMY11y/lrzylrtzcy6l9AACoAAvJ0w1talkWdRcWg0wtkGUnZt5ofe1W3dIIaKNLDgoVd8kJZf/zQMR/FCCG5xw4RgBLhtJu0aRQsNWof5LRLopQfZPDRllE+TYBRo9WTr3EjBZWyZWd6728qYsIAKgjY6EiU1Pbw7IUY6y9lUFKHtAauQydGhgRPaBWgSC1IPowIlCOqNuUE+CmgGff//NCxIkUYOLrHkDEyL3CiHM9T+hxFthAk96EvqLH49u5X1a9aqypmqupjwEtSgIbXbYWtor5DyDo+wJNSFkDyo9+WjMw1lscA3AQXMLMwC+xT1gL1Hq2GZEFXIBpC/5uxbtRL5HW7ZuW//NAxJMTmYLrHjBE9Odom+d/9dWS+6W14LAAHPlNnK/MspCgMGTCooZcRwsQpdMFI7g5g+XUx3Ps831Zav674OYSLDppla05+am/byCiF8083rrv5zV4f4g4q/+4NoIkQ1utkETksMX/80LEnxKIgu8eY8wkAUDC6lqEXLiE7+Go0MiHOgfrZolgZeD5y42FS1QsAhl40V2Dyq4+0KikxzlXVUpbuOBOiohizba2mWSxpH+uDSL/ov//////uapoAFAeIgqOAcAx6Obq+MvLuLP/80DEsBPA7uJW1lJS/KtX0NHs9MIlqkjInalZWBztmaYwwTvQz2QNIY1+iLR2UpWEgQfOhprQBCgkCodDLjpFN6FWlp0r/////+2ZCUr03LvwAbhG+JF5mtdW4UVjy23yyQnKbf8apf/zQsS8E5jq3k4uRlqn+qqtCQYvW8Zvp/7bNG21KAQYCgIed6d4lOiJQ40VCgFLIhI9h1DPSTpK/7j3uFv11QQ4aKUBrzk0T7S99ZJGX1bEz8uK2tlaHRHMn1q1pKiXbatNx5a03+upl//zQMTJFAEq0v47ylSamSVI5fBTOJDBXI+kyyVHQcSBBQYGiJf/fy/323uI07dSyUCGqvtIcazl82OgGEETRijwCRAANHhdVQcEOVFRrjC6LVMsQ65ahCpOUmgvaq0yXhXr1+XXaHDt//NCxNQTiSbaPmGGlnfe0T4BKjjt4QUCwbFsrrueWJTlLww9KAUIQcAiGMwpIUC5IviXY0sDAxbM0MSOVdixPRZHGp4Ndty2ci3vyfhkMnClUVq1awvnfP2I59tFkakWTpMs6ZuaopH+//NAxOETySZYGNMGlWuWXSpO1lz4fMvHI8ipzLl7x+5efokzRRrri0CzdPtqapqD+FVqLM4vC8q9HIKKxN2+4/jbr6sCIbNrJoQufWxRRUVQIRdYqQQyyfOLLEr5a8JHaiaWymasbTr/80LE7BIoMkAM2kwAwytCMLMekmpT63o02KKXVrLmnNKtRQQGxEUUQOUKeMA6+Q8Kdw+Jkt5/2VACUSZAp/qQa+QFS6nNFGgFyHBBpPFgFnjvC2oFwFpXi4zhoWRGgXLC2Aap+QdIcZz/80DE/xnbDjAK4wYNNAbxBpwXyEEhBn8rkwVCYIgmcFeGkLSK8F6xWgsX/LjE4iboHTcvkPIiajkkAKwzI/FX/+fQYzdMvm5Pm8n3FxEiKBJIgBLHR1FYoESMiBf/+zoXQZBrLdN6ZP/zQsTyFChWQBVbGABxPmhNG5VOEagZl0jEEjf/0/+DQmJKErctrdstu2AHKybrXTzkuLml2rVrrVcmFhut6KrCZyASxydRjLNkNxlVtf4xbUlwqrmZsfr+1VT/4c4erR1/NTLnP/L8jf/zQMT9LZsaijWaiAC6cPT1L1Lh//BXCoKr2Fn8SoIhOYhQeE878rUILEBLNoBiVhjiksp+tXwlOt1s87mdqdxudtV8M8bYuYPoxoy7LUOoFQbB4I4AgqI5Ll/afrd1f9c3URI+JLZO//NCxKEYGmrSX8wYAh35ur06mzAQGg2DcDhjS46DIhebOz3l8YXKkaQscZUxReLoNlhQUkRZAJDJZACDeUAbDuC4Eg/Xm4pcZOmBELMA2ViOZJEmfP1SOMlZb3kmnPMh0IxxZ7MSHXVq//NAxJwaMYKBl1lAACFAXNrV6GDq1r0S8XXlXCoVavc7xbMT+9s4zKxRqqhWRVe3saol1597/zXOt7l/ywRGdjjxHkzP7+j7Vv///j0jf///v47hEcH8eO8eRH9/kbP8CBgHwIGAfAj/80DEjiVKWnjLm3gAGP//+D4EDCoCIMIGhXZ6km6Ms/zTjkGjV+t/mWlhkox3/8ChgdeB9a28CoAd0AVSvgLSB5IASiEX+DcQXBCeQuFFt/8UMFhQg4dIatGRGeU9bf8PmIUjRBUXC//zQsRTJAsyODGbkAAbCun///y6PpFEmTZhjiZLoxpNF7////FnEBIqKRIERYQmICRUUiOUOcITEBHNEc/////5AhzhCYckXKIC1UxBTUUzLjEwMFVVVVVVVVVVVVVVVVVVVVVVVVVVVf/zQMQAE9gafR5gxgAMpOS2jbgAMSmYGLcHwcBAEAQDB+D4PgTKBh58HwfBwEAQhgHwfB8HwcBAEAQBAHwfB8HwI4MKDD+j/lAQBAMPn/39H8+gocBAMA+D4Ph9CiAvJOOJxgFIcN0y//NCxAsSYIpACNvMBE6VCoQYkAgEAgEAiRIiRIkSJEiiRo46qOoGgVBUFQVBYGg6DQNAqCoKnWiIjPGACGlk1U9epX6P//r//1f+qjvsPNSBMIGRDk+hIKnD5JrLOEAfYH2j1yF94qPS//NAxB0R+DowAOPMAFjahaXIm0PadfOqscxI4vGMk7g+qm+9FlyBOlMUT238u9a3qF0+3B21dNEJ2EACqoHESPdWGQo6TTy7yGkEmYTJomxGrmbvPbTm5Hqc17fK/sReCKRePCP8uHL/80LEMBWBpjgI2YYkz+P/kgikQcDDGAwDZabe8izpH0klUhZ4nkFuRtx1GBJlTU0jy5d8jqqB6cgOUEyqyuCHwdOMELWhFDgwUnkrQzfdc4m469QuYKYuyOcowPUuSyepg1G+pQ/tMev/80DENg+QOkgK0kYER12dn5/xlmbVGrRySSW2e6Gbtrli/vmZKMBAfRHmb5VW+HvRH/2nTnbOHwlvtHPNapf4484VBqFw/JOIHl+y7A7cZclYxCe7/27dKhlH3cWnX41Sc3WEAQCBGf/zQsRSEJmCTBLYRnwMDjtJJKjj3h3N/03a1UnZI5MqH0iAe8qaggbY4CpKRZwgCD1j2Vl39Bggu3mOj5Om571Oj25OnYwNQoD9i82ACShQhzWmZ4pzrZI+UTn+XnJ0yzPnXOASjCxrY//zQMRrEgEeTBDQRFz46BcR3fFht4o7Q93JFdPEEDHlQjWEKF4GbjpEUv3PPKeVNXzPiKd/Se4hQhMSnTjIuLI+Uyg0OEAUJOA89xCdrVQBLRfrtttbrbbRR3YbNvaQE1igQctbHMOR//NCxH4Zwt5MMtJGCLAjB5y+hqMriclbO11w/GK8Nz9mEDCB643agxC2jL7GQ0GR9dswyCCB4XX/b+7pn88+7MhaE4ZGH3mLYvpldaETowmAx5DD09dmIWU+wUhjxkbnfWZDDD3HkLvb//NAxHMj0t6yXt4MTu5a4xN9+bGxXjDMfcbs/+P7VowaDfptkwKRDYRqAkjaiVwF8kJRkEQLvGWOjEM6Bc3mNNcFDgEFBRNOgOJLADwlGUD1mtTSh0bU1Kbn9v793PlZs0undtrmlAD/80LEPiQCfnpQ0kyd7IoAth2Yq0wo7ahKCb7UJubRkSD7hazyZDQQSgPAcIMboIW7FGfUtKZAq9JvORjd/81nQzLJu5Psh7TeU3pDR87SjnX/jnSd90n9XtyTNSn2pAk3Jba5E24xrBX/80DEChZqBspeeEa2CIsZETBeHep9S/UvdBg2N2WbICJmM4Z/zMJBwUn+kSJvE0s3HC3/2zzI0ImjISOaTcqX19BCJHIeMOijB7wM1mym7SWdpAK2hgjYVmh/rNVOW7bfWttygHoJYP/zQsQLFzka2l5uTA6JJMvJfr0VzhdNc9A6Exe0KumtOHTfI8n309KaTDyXtN7MpojfncgBk0DR8XuTbG77GA/aIVkCRYcdijEswRUdYSO3K7Q29q2dqpu/a9p0BgQsICkeIaL7/C3iz//zQMQKFalOvnTCUIiQF2rg6CE4rJOGm9voqKaGQfeGDp8jXcjTVFzEKJtbWOBiJYGEvlxHR8C4u6Gibcupe2RESGIp86aG7HWzwQV9vAIXnkfrLtJ2f/6YrYA59IB57DpgFCjstaUI//NCxA4WuSKthMpGtM2kEALEVgjKiSRXIovKigAARaggbpAPJoxXjZGayhBc9ZmjWKMECSrQlrRKiylWoYgmgRaYIYkZNjHlVIvGhdv6Htv/kUhJDkpXSSfUwAqgIg5QcMBINNswQnLF//NAxA8WgQK9lmYSFGgYFD0wjYgObEIBFrYQ8K35dXNNk5ZdAqlBOqXliDDzHYYoEwTGBsh1OayUIqeMO5U2C4FeIEuIT2xcmXPmGf//22oot/v2gKz/jokgH9akttwMagGCNgtJcRP/80LEEBRBpvI+YkYWkDcIbhAmNMbwugPsnIfCzEekclgY5FPhbFWKF/5aeXwzyLIjyUjUnKPAqEFr0McuKK2tsRsJKM3r/V+UVmfocyog5E7N/wKoKslzPJXLShLvKtEKh9nvxFMp977/80DEGxNpmtT+SMTYeGsATGvUUqtbzzIi2Ii16qUnXLbn6erWZBgC73NFY0aPPOTNiYSFGoKoCyqH7l3ruehdVgf5uSNyCrQRL6Yc1er/5bZ12Tq11fCCkKmzQ72oQWkpVFVhTqUar//zQsQoE5mO0Z5gxNqTVS4PAxrBU6WfSf/q2CBQFWkSzF/+0d7dudamLRccsysi0r5WNSoC1CSwy6+zYPXFzp6gs8Sx6ZnP8sTQMvE0ItEeQkUcKzF1RzuY8e/89XCInqRMEO+4wCSlFP/zQMQ1FEk2vl5JhjRwmOEoLERUF3EbBtGIHEFPaxndV6SrvSscl//11WDfEJSvDiVPZpxI5ESo1brtNYtLSYGLsiEpzkTt8xLsIr3ta1/85/Rl0J0Ja3PZUbQjHOucWBizKBi8ESeR//NCxD4VkyKsyljEFZ6nPDi0fP1fk1dDuyvnzoRnIRoRx4et1pVDRBSUmq3XRqdWj0r9/f2qhxhT223/8um2z7zvr6X7qdVnYrw6sZ5iTEssvraWYnIo6JPq9o9J1mDR6EmAoSOMLmnK//NAxEMacy60yGiS/fUUMdtCbNm3zUnJTL81IRnBSDDd2jfJRLVJvtR8LXni6QOfSxApkViHVmhkPUEiFYzb++u11eR32Wf290dKM6uamRkRojVHnXrRn////P76jzZ//TvlTBUzauf/80LENBQqmtcaMEb8vSV7O5uAKbrnRTY6EI6oQXTPxsSFrgyQLuqJpIKct4mbqI/0YGXJWa9B3PBMB9QTC9AoW7D5IGRz49pVBZIokq4Sw1FBdaxdiUikDdi7tT4ROQdP/2KKNEBJZJz/80DEPxO4suMYYIZEh4SSWJj0sqmhd6drW11bU2h2VtrbbIpBVXBE7NWKOVjnTdwpNhzdHC40yJiMYjg8QGdEk2xmFplp5LAe2qu72ZHlfMX2YuokmDMtdo2/LGhNcKQ4hrSi3Kkzvv/zQsRLFEGC7vpIxJ7P9eoaxmq/7XW22xoQcTqTKj1GaYE7D2KQNxDpximJIuiWNiqD9EC2QDKKY7ZtoLq0wsHA0bPB+bQpl0qtz9KrmwK/9drJXVekIzgU+r/xUSitFZR4mGeHhm2tc//zQMRWE4EK6lpaRg5hMPPiiPJIngu9ITigOvDtRoGDAgm5FpH1DGwQixJIUmpyKgYUTepr3biYjBN1mpb7u5T5OhLXb7N+LdLhtYiZ+JTdSHiA2niHd/v9tY58AkwsWs2lbXYmIWW1//NCxGMTqML/HDGGCh1DOpESkhCZyoTRjqgncDhEGBKMqWJRwsIywlET376NYq43s6G/5sXFTpKdLPO/7K4ahRJaTLURdxLw7Vrdb9HbmSBB9JqgIncuiJnaNNYTpVZqrMf3I2tGPbNj//NAxHAT6Lr2/GDGgm9f43ehjqrGiw7GY1Nf//9S/BuWln1hr02BoXBUQgqdBVlJXvBVMGs9DUqSYJVKElAGHLUjwpsz/ZrSLgtdoiIEsFxl9bGZjq0j9qpNeqQWCmmZN4Yo21h0GMb/80LEexOZkqpaWYYMpHRg2YQ0NBLXVJBUGn/6n2CVZ0JB0OnSQo8qdPSLSK5b/gValkPzQ1bJpgAEiiKAI0VpdahKVlzj3b28nQSzk2LQCareu+VpKEVBHQd7MtbSt3su0OhRVo7XcyH/80DEiBPBJlw60YZEBreMIT+EYrLqYqBo9q5rgAjGhNDCwOguCZgQsSoqKi7RCgAnUHIndDdaVONXSMbMLbKJY9g5uVRZVyKGMS2kwj/Wv0eA6fQVQi9/9BkJNC8oNhxoXGzptN7Eyv/zQsSUDzgiTBTRmACUu1V8dyEi5aG2Sr7nRj5grq19Nv6mTTWcJ7lf3t/FKj348MdgQmALQ4bbZYoFBsFCrI5NWOPUiQOjIfS66/o7aZtRbsvQNBCKZ67ZHc8UF008KM++HLPgI1vPlf/zQMSzEShaQADaRgRrTHhLvjb/vatqfiu/8DVtmuISbuP9x7mnGY0O5/5WV93aFkQJtAEms8YDrEgjRpbqhqsiEcJikzLnsXDwXelkpZW3u7N0yToII32Dr/rDIX3MkQxsPBY6hwNq//NCxMkMUBpYNNGGAGBtY9ecVMvU1djNCgwoDS7beLuY4QMeHFXDiQZfVEZ+dX3WHaDhbt2p+tQ7wwrZdZBTGSUKCmoo6YCctGr2WF++uZItJjkoJDZEiRSX6BmJxC28v0iZ1zC/+pb8//NAxPMXYJY0AOJMIZE534o0HHm0N3z35d1/WtqnwP3893lf7Ttz4kdbYdU8ohwAGkzUyqR35i1IblFJ6te0iZhil6fpdUvUS3zzh7w2GtG9t3Nmn+5efrrfwN75hh33KYfQ5crwzkL/80LE8BYxkjwS2kYIR6+ZaS5QpV4cyszpJczVfdDJVjIeh2EppDKondjFzp0yzeFzmDNiqr0v3QFZumj/PuRVI/4DCKmrtupKxgEMB0YHXKjhChxEfvtNjQsp8Kh9RagcXIFCjJE+xZf/80DE8xaZXkg02EZ5adKCkumJmmHnQ0KUD4Xy9LirKaHXpYIGtqxfKJS9ZNC9ZED6xciqPA6DCR9FaCqRyIzEK/J7EEYTswO1ZpR8LIEOraeWWZ4U8xAY1Ev5LReaJc2yniC+Dzc9r//zQsTzG5MaMADhhp0fAWMDI8ah/trx5xzAHmMnQRp1mLz0sVfLQI+3PtFEDk4/TC7b6Or+2t7GVNUGYGaSYOVoEAmeInx6DiWEEGjgqQFXuDV4GXOGkNQOky7A2sVcBgaA5QDGwodh4v/zQMTgE8iWPADYxDSB5B8DzrGNIvPEjJo+0xkR4STKtYuhK9Kj24Vaz9Zu3cLKHY6OwEjZxgeiHSRh0+pAatjGh2oRZv7EQLytp0i655ksu+RcLBYr6VXiT8941e3yFh25SIeQ5rgB//NCxOsYCL44ANmGUXJLqGa3hDOn5Hu/Cmp3/Y3Us+8T8T+e74VhCd77nsC//3k6GaAuBAgFp1Qz2V2qfLWGeN3t7PPmP529/Y9l2OqmqcPRNUS/TtrtcFS0wa4NwyEDU8RlEH+zPM5m//NAxOYUGCpEEtsGAH+UDlqrZMGQLnhFedeJxBU58/G0l3PSyxoknYLouvaReoc6LhF+kpVQBgVCwDCYUBkcgQCAJ97QdeS5MOGISP+W/L5mOPHNH/7pOoAEQ0N4VQxTA0m5S/AgQGL/80LE8BbhfjwK2YYNAUP60z41hzwCoAEL6DpFY8yRKYDnCXBaP5DPF8+ibpgYQZR2AjgVESIcX/Y3MCaPMpoku4lBLrC3i3CXKCiP////hwBzy+ShfZlJF0c5Ljs/////zxcMRzlZaML/80DE8BgZxkATXBgADDDxKZeLqRJEqPP//////9ieZDyMTdVALeaAvXIIcyjoVFLH+6PJksQGqs3caVtR3Uh584Yf6B4ArtZizsJ2oYz+hXOzJ4sHCtCcWn7DKExa6yJ1BBOwM3Ca4v/zQsTqKVPall+aaAAzBEfJSeXy8TDIq1EsxQtXL6MzzfWv9ar0ri5JcxeLiorAIJYMTcLC+ynW9OXzcvsx0GgRiXxe5r1Sif9CUMYCYMCBwkuVQKKJDB+0woJodkDb0ylB6bQsrBqyIP/zQMSgJCnmubfYYAC18EWCdOVFEyW9xicV1lXT+BG1SLe0XGM48T61eudvfduc368fb40CeEmdsS1GiKp5aKhT5lHErqKta71XWzJbd5vLQRSEhEokAodYBSmAIOh0oq+rK2/KqjDs//NCxGojEuKs5sPKvdv//RkEjMiBZBpjOZWqlP9VdSuUrC0YsCu38zORz49W6L8qB2iWUI9UoXsGGTOZgtktq7oJMHCmWVP1cf6nlsazpaXHm8axpIrWzM1Mx360VeSDUPQdAChMKB9e//NAxDkewmKU51hAAEmyTWwch6oKQhw5tLJqV+a1+GmaaabYW5Nh1i/29v/vX1/+VXv2bZhY762aRVwNHuVBV3KnSoypmJRp4RDj0RFnlrbdRqNhoNB5+vkPAIABmQAeEg8Ys/V22pj/80LEGRlKIupfjygCv4m22BzowwHDPchTDfHOQUERgmvzuhRchzFT1OKA44wBgjBNJLkZpCfICDjAQEHigo7FMpUVf/5mFHeZB0g4TB4Bnw8ESn//n4rQlFWAAAAWRpO20C2//////r3/80DEDxfzCv5fwigCKe+T/o3v57kS1CMi85z1fejUydiSdImYPjxg5xAhQ+MKd1kOiCgfIHw4HBMgCAQOKiKhBUSHK6WOcPkRkIJiynD7odZ3QpLEsLkwGQD5C/h9aKmrqpdaQAEAC//zQsQKFWlW3xwQRgDfILQPQ4dFdiQTCNiF2LsiylTf7lxXkYQaLL/5XjkiIKEjk0rthBiCTRGG0LbE5R6CdeQFeoaaSUE5C5IEOPY82RMVoE4tCKD8HmdCnLrKmHdNABcymCE7QiRRo//zQMQQFCme4xBBhhEnNTm5m17/ySyue856FlqbIOZc4ZGbvX5QSEnCgIoAGEBGuPdtfKkP9tW9mrJZ+5Eb+XVMuxnJpxBXuvz9/mfMrXl3h4h3WJJJOA1CCy4z3swOg7cZZtJdpusm//NAxBoS8UcPHjBHKj09PIaujO3QMYXEU/9BRvXOUyOiBcwEiTCQ2G23dUl7zkg208OUGaedAHWrFW6F/3exrMuruqdNBDABEeeyBzACzjDMiHe4akMrkyWhn4UkLu15eb5cf/z3zsf/80LEKRJpUusUMIaAGyyJpCBSCwPRFhYXjDgetXAKcgSsvz90X9PzTXQk1dt6hoFIeXQ8ADAGIIx4HQrgPEsvGlfl9bMHN09pj/37cm3EA+lS1XX6v5QUVQLnnw9ql1GNOGL0w7jWJEX/80DEOxNJXt8USwYE9T7KzsTLSAUrcp9OpxIaMvULVZT5YWAA09GaIKhBgaG4MqtigkfqCGJD40AKHpZ8pZVCQQbl/0u8hP3+i/yDFQeCgPV1F+aLM1BCFQDAljIcqf11qC6pGNCqrf/zQsRIE5lqvlDaBrSkqo6kaIkqQVIsQAANAgEv+49ZZqf1KRZM9ylYW7NzIgMagSCW5EbOgQGvk110PbLFYmelEVhmGropfwlttoToFOiwdh2CupZ361mkD2lK////+p8AopqAGRkpSf/zQMRVE+D6ul7J0pB+wDp+cEMPYrNgRQYnhxLM4ytRrYPtC4fNEMDI8QjDMMjE982YalE/1ZlM7G6UeYxtTf/Q1Dej0N+v//zPcyBnPLO/UDV0Gg6eI0UIUikAbWezFBzFs/NXgBQK//NCxGAT6lLRvmBE7uk719gAqJBd51qZOk2v/LYqrJThq0wqKXCFjaehZRL6zBqxw+UqAo97/qKQn12tM11rPVAlLMzoDmcj4U13dPyKGnLOsA1w4TYSI0B4JIiwyxPmKFS1v6ttrN6P//NAxGwUKZJcPMJGlEJ/pzddbOyqWi3drq6q5zI0okOBtoJgqT3HRLcfcUZ3UBR8aj2I/e76E/oqDCAEBqgrKgM3h8NTzkqpaWzMZa7vVvG836yMszFffR24zxMvtSz723v3btTa/+3/80LEdhGRXmA9VBAA3G0Woun10SVnH1GRn9/UaZePGtMS21cm5475f35+f3Ktj/ZzwbiRSZpqkmCjXI//593/N3f6dKYRO2kXQMg/gTEi6NESdZSyIKqvXz3///////7Zs////////5z/80DEiyOT2lwNmjAAlJqjqgEJMkeWSKZGmgiXoGeEgW4WadzpHCq+FPZieZjjKwLRnFlnoA+plVaXLdNBqTQJu9VsMVGv+m+YbmfcsxOHfspCSrZf35n340j2yPsllIqeIZzqb62ltv/zQsRXIfMWRCubMADLVrfb3O74rZ1HMqH+/60uXdxuXWTPb5vPd27tpj5myhGqLPbLgwYAwEe/iYT/6wuiQDLCYUDgkmt1EgSAAiNTao0Qv6bTjR/ikHf8JQjjm/hKNKFP+KugaDRGDf/zQMQrHzsCib+PQAH/8vTniAOMgd/MpJoKwbioLw/EMpRDdg9IZGYmOZD84IzxQR2Cg8kk81TVRf///8gyBSkHOKKeYXTPBVMzMPhv////+6fm3e89936jl+ujS4PNnZGaGRkBrY2X//NCxAkVsT6wAY8wAE9ji/W3jzyOji17rszJmXk/4xEeQSLJGwRRATqTdsj3ezLVUH6uHxvnz/DTtIQAYsj/nzQXQETZz/+AwSAYsYaa4rFB5h6nnflBV/+wVSqkv6SrA1SVYHKVDFnA//NAxA4VuT6sAZhYAFEA+RYJoSxOq5zZ43I2WqcWbNIqU1rVkNrB8XXYJBBMJA9oJr987qqJ/me/9qV1+xG3+U6/859976R5SSjR5qWT/e9YGo///9YAgIuW22gu22u1ccAieFI8mhP/80LEEhi65tZdjBAAbqDlpqV0dlVXqqMV1VWMUSVzKZ1hUdGWaj0KWq1kvgxBvVEoz3cn+vdyMgs8mdiST//pvpU53aRnkPdGIVfW6G/vs/yUO+QIAiGk////yhkhltu1olkkrwKQUZX/80DECxbjHwJdwhADlleds29ulbaf/PSjaVb/s3W/pq7XrTZLVOkxbHI4hQhwSlcgZxbO9GOt66Ki5Z9pGRUW6NyWs1WjHfIUplO5zKpEIdVFFM51UI+g3/HYwW513229tlsIEDshzv/zQsQKE6Dy8lqCBhYOkVSwWlqSbZsnHFir0yZronY5luVN80VXFdl+CGBIRnQyIwcYLA1Gn/7v173S9JNY5fImkPcoWiozr7n66X5B4bTTMIXb3bW1xsgeSubI8bEd5FtUIICi1JVFsv/zQMQXFAlG6lh4xFpKDREWIC0DE9vZ1Rszu9aenVnttXcge2KYx575I+KoHA2oJIIw8eFKFIrGJebhqlaMBdeRRdFXS3UqaZh331tsjsBs0WekxXAlA3X0xlsBhjquU1YEPQPOurZ1//NCxCEUOP7+/MmGZowhd0fPOMbMpBipNZ8/n4agUBkWDj2GjqQk+RgrxLasNMDR2SxE/9jTygKeEriUOwDcCW27HbQWC4kYAWHOJujzoTgJrMmBJDdm0+shddNwPaeYLJGzFIEQPsGr//NAxCwTqLahvnpGTHHihwOxKd7Io4XQxwul5Y0tX6/+k7Hp9vo7UIOvFtRKFXdKgPCoaBkE5VRHIcIm1LKlhBqwPCflHRgstj/NWZRqbgKHYJQFvA7sUeWBpplg+HxUBpQUJuQNsED/80LEOBRpBogg1hJUaQnTbnk6WrMySc9coqwSA4Ks8fv93///2X0KAAAFySAL+tOoDyraPCVmoTDa34ksIyyrdZW0+nibFV4IcQi6crChXFCVhAQskb/NusffrHWJMkok4304PlAwCHT/80DEQhPQ9p0e0kbI9N///qh28E/b5e486xUuUCCh1MSRRO2sWAMtrQYTJh2CAM1dLmhvLcyRDpKya49x51hWDLYewh42VqPIayIYqmEqGar15fPuWIFFELOiwaEP///6wh7c8u3////zQsRNE3lWpB7BhMDUGQFoubewA1hgBlgoBXjHDGQct4KD7JxHRs4sKyP7Yo6D1FKdqVu9pMmTWyla//rrYOHQRAgGDwAMOQYuumN4nf/9Ul/s/oiz2yBhuUPJ7A9BtWCgNm0Icef1pf/zQMRbEjlGxZ5rBFIcRq2P1sHlTQJXExFcToL7b2ndpWTjn5T244Q4rN9Uvt+z3djoV2MQYxFIiHX3xppRmsxV////rtX4xS2/raJ1qlbgBaHfL0FtBGcmoZ1UXBeZrA0TUJzWSoda//NCxG0UiZq43HsEllx5segPLBBJS4GikQzN9ZZvEYJoigkJemsXRCxciFmrY0HBcQlwkZ6///////S74sNaV/UdEtUOWkZL+BgMZM+LAPFKifoTKm4IgFUej6Nx82Zuv5zZaCLEYImW//NAxHYUCMa0HnsSkJJ/6wZ49W9Ht5TlFDw0IlYSUFBdqB4hDQhTgHIf///2hs0cCYfNjA6LBQqlCgA9mgEXVFjfLYHD4InhsxZ8ZMEQsRk6oFbTedXqicMR+ehWD/WsnFgFo0bg5k3/80LEgBRZItWeeYR6ATMYxBDzO7CIYcWAWarYMrEhp12nBeFzYCxZ5igRAx3Ns5uuoMcevAulW373ffx4y/jv9zHdDG1PaMJwZDn3rX2iP73vfyM+Y16/j0tTQ68nt//383mNm/ZXAmD/80DEiiOSznQe3gx5k2mg8WoGBdtAcEggwElpCq4mDi9+LP7EkbWAmqyONyChl2r3d8/HHDueWX93+/5cv1bVLH6bsU1S2pfez1yxbicSnock0ATOobgVwHaLwKCBVQUK1QtGgkUwXf/zQsRWIqtCqN7ABecsXeR22Vv1chy5Ec7EroZq9Un6G5Pz85clOVutlTZYY1fy13WuhKTrdUIf//PIWqEDtG7PhQkVoLt2kwgxAGkAcaSXbDCjfxkuQrUXxfWhCC7jlOh7H1Uf88tNTP/zQMQnHQNiwZ5bBWrdVMmvWO5dfSzmK2SSCQzHYlA3W5ZhbT3xyM2dsrdvdm2frlucUAnQVEupERS3ZHU5Twj3T/////72p7Mpf1a//s7A3Q5mEBSABzAIlwmruSculoiUGbUgsdI3//NCxA4YArrFnnmEuoap7L3gzqLVQKo1lphwxiTX/JfpJv3x85sW7kjS9KJwfy2dSlb2Z7WzXkMQzhyBVRrM3kRET/9EFEOJCt/T2////+inQEOzyhXcJVaLI2eaWUgWryXLZIE+ewh7//NAxAoWKXaw/npGPuBGnaDRmlg3xE0s3yWp4s1doXyRZJ/khXVg0z61ud8qBNDg/kFL/z6TGxxqawMGNQx0SwseBoFSvOtPJiIRWf+5H8k8SgV1aKaP0+GhYKoBlJOObbWgHdKBpcT/80LEDBMQ7qm+eYZqVaG7DbJdPUeQMXY647Nqik/GDqVx6EAvQSMfVjMpMwEVEqGSxE+ZD3yYNP3p63RZI0Iu3Z32gVAqhDPuDQVJiKonWCIA1YnLknRKpQBAwkAQOti3Ly01NK4OquT/80DEGxQBNmwe2ZCMkTCzy06+EzCHMgTakiDIiG1XuKlY0a4aNIqIgyiKFAlCI0LuMV7En0Vef/1//3ZIiv3N3iUGyyS227/7ABG0BpUmNrkAgp2IkFGhXsLKITd5hQLMA25CI66iGv/zQsQmE9kqtl5hho7suqnUeuDypw+lMUG3xY9XaUalDaH4Yqc9wPjoY149f5VruXAHBSjevTUVynbGgDXgVCiKJIqQVzWrXnTVKpFsDAB1Aa5d1zJTJsURGxlzWoJBC0obJ62c3Jcjl//zQMQyE8nebF7DBgjX/XOZh0vM9VWfw+rP5dxD5XqW9YDZ55Hiz2O/yyov+D5PxbG84WgOOsEjo9gsSAoVJPxTj0MgUgFQEJAKJWDEMCpn9v/62v/XHs4qZb6vxnx5J/8VTUxBTUUz//NCxD0N6C44DHmMAC4xMDBVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV\"}}]}],\"generationConfig\":{\"audioTranscriptionConfig\":{\"wordTimestamp\":true,\"diarization\":true}}}" + }, + "response": { + "status": 200, + "headers": { + "content-type": "text/event-stream" + }, + "body": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \"Did it release ship?\",\"audioTranscription\": {\"text\": \"Did it release ship?\",\"speakerLabel\": \"spk:0\",\"words\": [{\"word\": \"Did\",\"startOffset\": \"0.300s\",\"endOffset\": \"0.400s\"},{\"word\": \"it\",\"startOffset\": \"0.400s\",\"endOffset\": \"0.500s\"},{\"word\": \"release\",\"startOffset\": \"0.500s\",\"endOffset\": \"0.900s\"},{\"word\": \"ship?\",\"startOffset\": \"0.900s\",\"endOffset\": \"1.300s\"}]}},{\"text\": \"Yes, it shipped this morning.\",\"audioTranscription\": {\"text\": \"Yes, it shipped this morning.\",\"speakerLabel\": \"spk:1\",\"words\": [{\"word\": \"Yes,\",\"startOffset\": \"1.600s\",\"endOffset\": \"2.200s\"},{\"word\": \"it\",\"startOffset\": \"2.400s\",\"endOffset\": \"2.600s\"},{\"word\": \"shipped\",\"startOffset\": \"2.600s\",\"endOffset\": \"2.900s\"},{\"word\": \"this\",\"startOffset\": \"2.900s\",\"endOffset\": \"3.100s\"},{\"word\": \"morning.\",\"startOffset\": \"3.100s\",\"endOffset\": \"3.500s\"}]}}],\"role\": \"model\"},\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 1,\"totalTokenCount\": 1,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 1}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-3.5-transcribe\",\"responseId\": \"Sc6zarfDKPbFjuMP44vIkA8\"}\r\n\r\ndata: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \"\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 90,\"totalTokenCount\": 90,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 1},{\"modality\": \"AUDIO\",\"tokenCount\": 89}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-3.5-transcribe\",\"responseId\": \"Sc6zarfDKPbFjuMP44vIkA8\"}\r\n\r\n" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/google-transcription/transcribes-with-speakers-and-word-timestamps.json b/packages/ai/test/fixtures/recordings/google-transcription/transcribes-with-speakers-and-word-timestamps.json new file mode 100644 index 00000000000..540d6e37cb7 --- /dev/null +++ b/packages/ai/test/fixtures/recordings/google-transcription/transcribes-with-speakers-and-word-timestamps.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:google-transcription", + "provider:google", + "protocol:google-transcription" + ], + "name": "google-transcription/transcribes-with-speakers-and-word-timestamps", + "recordedAt": "2026-09-23T12:32:14.599Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-transcribe:generateContent", + "headers": { + "content-type": "application/json" + }, + "body": "{\"contents\":[{\"role\":\"user\",\"parts\":[{\"inlineData\":{\"mimeType\":\"audio/mpeg\",\"data\":\"//NAxAAPyC4wNMpGAAjLUgSIZ7JEeBBAqL0cOD4PlwcBAEDkPlw/KHCgYDAPny4814P59Cjigwt5fUCDuH/7+UygPn/5d///y4P/yaoOpQOtDQywMA4hBzNscmCxY8QlQVEoSIh15EP/80LEGxCwLjQs49IAQBMTLzdgTFyVbeiaSYV906ydQtZz92u8ZZZay9El9d/ro3WD/YwmegFiZpZ6DoTEFGEjHqFrGCywiixta6L79N4zkUo+3Z6bqydz/Zf1O6atdt5RaS+upqfbmh7/80DENAzAIkGU2IYADxQ8SK05AndWsKWuDrpDSDGzjmCYuC9rFFCk6VlbUYo1QwI35CObFLnGKyCqQSeeUbwYLUxE2H7Ejw6Kn4wkqpekgqwmxDxW/afP1SzNUhy4CNdUQBBSDQZWLP/zQsRcEwhSMArbBgAoo6tTUIcwtxmTmq60LSsKosrR6EtuX2oqtckD8j6DqfctL6loq8gUXY7Z0gBl76v7A+lxrYN3maqL6tOje07f+v7P/RTSvxn9Gl//3f/r1zw0MWO7QYUJIoUjJ//zQMRrDrgeRDTQggCxK0qMYfDxISHXF2HiqHndRaHHjUvGhV95pSYSCblXpm7A5Ru1HS1J1oatOrIcylN1bfIB+RvqF6PbLIGOqXAcHuAwPFB+LLyX5mtOKIhgWdMKnyryKmweYHZq//NCxIsI+BZVlMjEAKcdrlxBQtFe4fO2T3TURQk+37lsna6F9gvRPZMUecP6pJ8wbU8zH3MtCC4KANTB2UVqxuwxO6BHUIdHDsUbBeqIErcEctTIeHGkre8nTkhMX+7n8CdvE/zNzs6y//NAxMMRuGI0ANpGBPvMetEK6jTmEEwBfhsuldpE4k0DI6MSaFpSHY94BKC4oHGCiVJqGpBoOEli2ZwL88okiMNZijJti0sUINHp5LlThEpYopurdEC7gk/1NsQbKpvUe8O5sZx8dFT/80LE1xFAUjgK2kwAVpWjVqkYth309O+/MnkyJcD+74P8/wZFbSD3Yof6/aln6HnBI9d8uwYR8jaLSsBsEdl2VwEHcwx+vTaOHkZEeYclRmYrhsXbdyM3N/STPQH+AVcwTlcov5r8P4n/80DE7hcRtiwA2wYI5cfzy32t3pEaPkdY8iKHo/C9gR6Uvc7UWkOf650ua097NCY99PydDl5kYRcfWOVhRQGuPMhpsAKIxofghbTTsDqNCoHSEJ1k4pa2sRl0tdc4smRAe18+44thHv/zQsTsF4CuNAzZhjF500nlSzn3Usv3tmDdhOmZFmqYuosaHJeKDRWY76KBvEWAiiRBIWTkvKY/iQ3kjrUQhgcLDhUNCWflQKBDAemMBILBgRM2z94LC3PTBFGPURmPZ++wjLIDMQPJ4//zQMTqGXMaLALZhi0CIk0/GCAgtNyaf7k9jD9f2mCI8WBM0CAYnA5ayzh94nr2qam4/vc1q0f1ZliktfURChykzzqZfKJdLKt+Vw3NOwxCFTkkdyPw5EpQ7bc2VopwfTyhYSB0gJjB//NCxN8RcDZADNJEBDDg1AnGJEetesOB0hM636aLFjd+YEA8O15n11kQJlsSCubkhxVpgYOdGsTqGKUr7EUwvv9Ruxw6ifYWLHNgcvNA3CACeEEJXN45//C/676MAAN8MqGYY1gcMB8P//NAxPUbWZY8AtsMHJzg+8hwG8mqOHtmKSyy99SXfpaZgePoD9dncph8CvlQTE6kGiDdnZ283FO09HqBgQuVgi7T51s0VM4OijatxW9+2LDBVgevFPSSIulDK+iqRMNqHnqZakTEjwj/80LE4iOKbmgAwwcYB5BQs4YrpEw5YcQo8HSpBeQEMieu0McOCIDPN9C3zObpiCd/LidIXS0YH7UyML/8CIJqDBH/9/Aw+O/vNePquYgjEUrvwL+aHiS8qHmAhe6MGIuWxoUGmuvdty//80DEryRScowAeYel2UbY/eZgbhTV1AgUtF0/pBevUmeJ/HLAWcza/cuMoqsnLzzztWi0ygEr0EjIfb1tr/v5Z4ymZKVZu7fIBJMHUknEq2NyU++NKcRzGaexjI8nUum7VWGIMheV6v/zQsR4JTtiuh55k3X9WppXTPEJITrY1sLwvA0dZmZPanezq5y61pqSFdpK4VYX+eF+ZVKo3ATKjOvBtes9pzoz0bnUbjFnTQoxYXCaCkmn3TBCo4CEDEY5mKKuzuHeLuPO6ONFeU6aaf/zQMQ/H/nisgTT0OhRuESHNjlba+L4qZ2ae7SWtaZVOEDHbmnCxgKxOHZ/FU5FInN2OLaZ2pbEh2XNE3Oee3rxIDuoPbFfqPfMHGxcPrm4AC3hTmwGTsVOGoAtSXW5mtZ+naSIQFC8//NCxBoa2dLCPjZGWPf95804KuaPPX/jKY1KrXldlqqEPzp0BGc2MuAxAvn8NmL/pNf7sYUS1fz2DChIGeTqFEgs+gGg6db/M3HyJMICV9v7lmQ+6KcDAEOnCQuRuRWsBg/7s3JWAfi3//NAxAoWObrWPnrKcn/16SivHBmX11ZUJqvWrcVB6EE26pqIihG/ERAi9mQDEIzetk640OOsxu/+gsX+IiosjI0wPwqFbohEI6skHBMQLQ2B3FtbIs1gJRanW9KsDIOS7aOS0DNj4/z/80LEDBYZUuJeekcKfdsnDOsle3fP9WFnH/fP/1BVoSQXgJv3z3NEoFrMVe+1TFZD/+1QIVzmbygbws46BxBlQyIEI78yZI6nE+tWEluE590Vt623wENd/VU0BoBpEhiTyYGYagEdm+n/80DEDxOhkro6HgpcDd5dw/luhBhwWd59c5+t9lQWSDttvKL++4V8rznvxb7xOQRCYuevxU3/TOfkYo46U9DMQPh9loD1iZ5Rn/T+tbwI0AJeiZAAXnVZ1JWHKBEsWMuJsdoHTQRmAv/zQsQbE4EWxl6ZhSDQFga3ZAzLoNj5Pp262ojRwQg2moCIAxelUcMKK3WCooh8FQm7iUKgTEoSf5L0kfcKfBpQACAPelAAC2gdW3JKLCNTnN6z/O5lHwvEFyVJ2L/4fl2pBLsNDn7PP//zQMQpE6miujYWSlhqBEC6tfmDz/+n+rmFhUXMp692FyN2nGgcOO2ItW88kMfVjH6w0tmFInfjd3AEoq4k9ZLX/oTAeUur3m+ITgpjcR1fMcIwTg8S39dQiDFb/OSKDzsUXQxwx/XY//NCxDUUaariPnoK5qMYPHL/7X1KQRIPb9DUiw06SSVjJMPEBnt/t/9VlQaIRsriu2Bb5Oobs7IhCCDVzVfFwDoWX1O1PECLLeGVHAxQs3wxGHmnFh0k3ioTlxzRZQa2g0/+BHqIOFmp//NAxD8UEMbeXkqMkgm4cwMuhuhKIDP/+nidSFql1SAkH4DhO7RWkNnA05GUL1DCxLB7HiUBFBOpwDPRd4gtMoffVk3xBkAstCYD6//8BjgQLrHgMMNeJQ4bMzS2CoNiAWGCcQvagk3/80LESRO4Zq1MQkYkeFmgNSBAmktkjf+CWgGBWypr1uGYbYxVrvhni9NF1/lkRDNFOcefnnLWmy82fnnz////////r1X+/cpXAn0r1KRgnPcUjM6A9M2d8PBSMDpM8yF1aVQ0jiTNSAH/80DEVhNbXuZWQEc+I0PaiUDyjtm5HJSc0DiFMw7uLZtMcExsCoZFKE+KCDzSFjAMYLvWhh1C3TxKdCVYCN2ET63RZiddWtKtfRal3/7Eu5p/qe6rgJKhgJytTnMdWZG5pq2gMwve6f/zQsRjE3C+uvR6RghAsKjFnF0Cz6Occss7KrWdHmGJm1MkJkxs3AVYx25oRmFsAx8KETHqA92ypgd9h5kUZXMzrgV//7fokpQBrkgBfp86LVu4zMgQPJMSGrTdy/hMypsJZZz7OWXex//zQMRxFADGrZzLzMji6gTzWf77KGWsOep5rPujSKQMl+3izACrspRLsAl30U3T0NtZWKUqaWKJZL6y9F08rlDOKQesPcWFyP/+taGpCTstuAeql3Ux/EQMFjsre7ledtOwuV5f3vH6//NCxHwXogKpVsmFJkcSHt7/89NwDcf3hIIZuvMCg82vtBQ8PVr4p6/qBdnVfSpZu+XKKJVfe1pZrLadaUs9tbqT7lRyr6f+vkQn/fb6POc4IljQ5GFMBvu2APln9/GmVsFQhOc/y5lj//NAxHkZQt6tnsIFGtvEWcwfgdAhQyR/mmpx6x643pmPupcS5ufvukQpJZ9axqGTfcVLaUfcnoz3u8ILh+QKVLh2H7xN/EdpXsRT6KLg3PioFz3LF2Mp1PFAAhoNxhjyD4fkJ2nqLwT/80LEbx/6gqmWy9BeUXdcEnh1XSXN8SWC41/i5hKINKn3LddcGv1OfzHU23it2Felypmvnaeu2T03/VWyKATtH8VqyAQQMzU5ucLEIJe5wvW2dop1/r/TbH392zGU0XWvWuDBU0WXTm//80DESyIbKs4+y8T+XqQBOMDhLJrC+XxV2xFlchvCEilsKPRcFPkKLuwqe1njKwqZZgaxmpRa9zKn/2Y5qs+pvzf////1/OCLTTWzAAuldgAM/qK+qkKgbA2X54QWZhPfDdDzn1V8o//zQsQdGCGivZY2ED7kIuflg8QjUDly/Puv0LgcLn3fKB+FJeri4av/Z0/qVVk7qBg8RbhP9qv9ijTqc97neUUcEZMEHKp2AukKoJJ//k/ctQxUIoZLG79R5vFDMDiOdC86vz+K8NMypv/zQMQYFkre3l4Lyjqf9SswWxyMUSLEBhCwib6CQSF15ZjfoYWbNmEj/QxivpVVT8rPrlMLN/X5ns+rFKU1/6W1nb////8wq/WekoJ7NiqQghObOO60HoyjaREHwoGWa1CqkTJibY55//NCxBkUea7Bv0koAi4pcEg0pStUOnFWurCQ9HsJDyloYrFZQ6yP0D3/N5Zn0c2hqFlLURa/zsO1A0PEQayrq///AT/4NQMBvEaiBqBHVg5d8w4oEkMxETLxLod+SLhy7FLZESwweRY4//NAxCMaqwpIAZs4AIIpJXZjShCdV6nCwkJASTvXcIwnCcqJf9PwDxEICQ0h6/+uPkwnKMYz/X///qeTHCBBXPW5n////+xink0M2IHlxAr///OPSjzkFynGJFZDrTcEhADGz7TCuhP/80LEExfJkqwBj1gAI83PklqDE7Yyia4Mb6HRTzhmHkbXIxTIOH4OFJNJKxt/69Pk3PpzBKROOb/39stld74lzrce/cXAaSEtY6o9+t75NJRw49Z/+5KP9dWHwt2xgKBa7GebZWWrELH/80DEDxfB6rTX2BgCk7hQ1Iofa2vhVdpt6rS2Ksih2tPdcwgspO2AKHsqgyFUAw6fkc3cw7E+ZJaZMN2POuynHdWs8vs2s////yFhA2gNmXetP/zN30SXUj6iSxVwBUgDAsG7W1wkZv/zQsQLFlnWlD7RhpJs5CPqEqEv1Dg4AgttQFcA7UkfCZoTRqmRnaLia3xBrQEgECHayk3sagJrIVXPBiVgEcaMxqxkrZf/5lNw/a+xJuOHGAb//////+TDBQMKCAgF1bYB3t9pyAAMoP/zQMQNE3nikP7ZhJRI54zVpo5tr5f+GK2WdY692GKjNewlRAAIkLoowwyFZ8hEJWxE6qqXVbjmMZ39F7EI3on//1O/jhfW3/EH/0STXR1AuyzDbhlSowOwk1ymktq+zTqHV2xyruvX//NCxBoUIKKMDVkwAuS+anO3Lw0DLuvqkUUUCgIjXvI+pKVO/Qu3fJtLIi4KkaQaEoebwghPFXJ1vjR7xA5gAoGzmQWnfTZAtg9talOzW1H8I3/LteULNGOEcbCHDIB86JVdkUxAUMZi//NAxCUdqlKQAZiQAIA1pKNzRZ4VsMiHrh6alspSbpqQCAgPgCxoBpBW9le2/E4ixkUJQqk+h//+TB4qJlxIuLMfv///m6BBDE3TONM3Kf/+goQNEAwaD6f/Ev/gcPoY7ecFRqduM7H/80LECRZZyrABj1gAG+jtUTNeDkdzzirWS9m19bT9bnkxb4Jj3prmwsD6VLEti6OTJijiJku9qNdTbLunIu/mphspaP/83f/6VfKIkBkCO4DCQzZKEejxdX2/+iL1KxAv2r1ABEceVlX/80DECxQRJtI32EACEOGOGZGzfdPr/ww/OH6SxymgFuz8wOHQTjw8PNEjOKHCrSy42XkRWXr//j6+l9DyBCDBlGKIirGWOLB4TipFH7avVVt7VYAwASafWAWAjVDkHAjFFDfFgSyhN//zQsQVFAmWsKx4y0QANBzMgxE6jmhdM6rW6sTx/SR1LDlZ4CLVZ/rpgYLVFgi3h8r+HfoQRwECIcwm+d9O3XU7znFBzwx/9qqRRqScnGBIm0MwYSt9LqiWbbhBlDqakjo6lO7x+/t6EP/zQMQgFEGa1X5JhpZCr2X6Kt1b/P9CN4EgmIBCrJ3v5EbFqMbqqUVRIbArtyzKg4WPXfnqHFfZ///3yL4fAKACwsamMtpRPYYE52tdVEvKYWwsI6TxRjMJRXhLscLp/L6FhwRYBpGB//NCxCoTiR6sFHsMy4iXPq3edre8T/FZ83xOyHImFUUGT3f3vc6Ij0Axsk//+rZJbtXQQkU7gMBuynscPvNZE63Zhjz9Kq/qszTHirQw8WGFBjiqh0iOYtEcye6f1ViGHrcydvZlokUI//NAxDcTifLI/0woAmFrMtSt/+lEFhZBwdEwTRvs/////+JVqoJQIYSlbfdNdhEAAAJOa5AoXStCC38a+YjFVECpHKLIPF6S25pky5s9LD6mLDlqO27qE7lKtlMAVSEPaU77hNaz/wH/80DEQyUiApW/mMAAJVuaA3sfjMSrcz72okWwBYzJUf87Os5TEYTA2+d6+6qCAyGo7P0bwpeNmlU9EcfuVL/9//fhnatqC8ug+xn/0r1OtN48//////////1u9XN+qiYLqFvX4ZcdKf/zQsQJFlGehAGYeAApilRsA+g71LXt7e2iupfbcGkSEcDK1duV2MP2Aegmm/951jBY94gf/Hxm26vIivV6v1Dv9++Xr7OLc6FZEf3u8xSBrf/8JT3rjP34Bf/xxKoezJqDxRZYLEJamf/zQMQLFvE2cAGayAB2AA9tqGFTIgVgDIAGloD44pLNpSYwNC3S2ef7IVihR/UNRr/iR3qBCyupaAAVm02rHf+K0V1kNFWhpuLe//7/3Jlt+q7MZtyq1Wx+r/Q8AkYCARtJq022sUAP//NCxAoPwVJw35kQAHQZ0RGIVGiLLhqp1pJdos57t78e3U28+nfQfv0I1Na+hO/0/6oQR433u6/9Vv63f13s//GKAe0kyUy1iYEttVoJhrmDKGmQDMUlZXGghGYi9PZmXBcHBhgJ630h//NAxCcd2aaQAZh4AIynWAmWYnxfv6vBYJmSsdTEqgro5lLLb94rGQW8Q8saMXKdITRSq2BvVn1q1hx3V95v37OdOo72n/tiZ5aH/lwRkljP+n6hZIULKBq3+j/9dVK425Ao3dgAP1T/80LEChO5LrZfzzACOcsCtVTI9pSeDI93SubyXtXesoXlYc0YyViP//WwO2Xmf/+6JsbL54QxoKpYEIoFByhWHTlRo3U+5J0VBlCMqrZ2XmfWJjHQFLcAAFXc6yUQCxa9v2ceYRkO8mz/80DEFxO58n2ewYYcjUzo6DmHUngmC6o25UgoUCAmMs/yzXyXlVBQQCGsGWPxmalYSkT/rjGUIof2fw/HFdCjhqg7xRliJMwAyGJExprkXI9LKsurbyr0v3Ldn7ON38LcqtOyo1LA+f/zQsQjFFFyYD1aQACBWbJoUD2bCIGxDF1V8rU1Wuqw0s1kqocjjuWi7nmog5AdsJKeRVF1uDKaO4rmVQCUhYuYAwIAD2LdA8LEFVkwpZlOT7OpTvdb3Jcny7k7BgEAv2IEEJmf3cBk0//zQMQtGZGOZNeaMAAI//z/8ggnfqnNI/7+3z9oyziRKZNbW/+f89NCITB8MslQlFUPu9AYB8Gip0t/a+kMfbsK2L9/84XVAMAdtOS3A4BAIKs1FgZh2CsFZY8WfBWGnKiJ/1nen93///NCxCEMyCIFn8YIAPz3EvUeViWGv/Wd4ikazpV3DqpMQU1FMy4xMDCqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq\"}}]}],\"generationConfig\":{\"audioTranscriptionConfig\":{\"languageCodes\":[\"en-US\"],\"wordTimestamp\":true,\"diarization\":true}}}" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json; charset=UTF-8" + }, + "body": "{\n \"candidates\": [\n {\n \"content\": {\n \"parts\": [\n {\n \"text\": \"Hello from Open Code.\",\n \"audioTranscription\": {\n \"text\": \"Hello from Open Code.\",\n \"speakerLabel\": \"spk:0\",\n \"words\": [\n {\n \"word\": \"Hello\",\n \"startOffset\": \"0.400s\",\n \"endOffset\": \"0.600s\"\n },\n {\n \"word\": \"from\",\n \"startOffset\": \"0.600s\",\n \"endOffset\": \"0.800s\"\n },\n {\n \"word\": \"Open\",\n \"startOffset\": \"0.800s\",\n \"endOffset\": \"1.200s\"\n },\n {\n \"word\": \"Code.\",\n \"startOffset\": \"1.200s\",\n \"endOffset\": \"1.500s\"\n }\n ]\n }\n }\n ],\n \"role\": \"model\"\n },\n \"finishReason\": \"STOP\",\n \"index\": 0\n }\n ],\n \"usageMetadata\": {\n \"promptTokenCount\": 40,\n \"totalTokenCount\": 40,\n \"promptTokensDetails\": [\n {\n \"modality\": \"TEXT\",\n \"tokenCount\": 1\n },\n {\n \"modality\": \"AUDIO\",\n \"tokenCount\": 39\n }\n ],\n \"serviceTier\": \"standard\"\n },\n \"modelVersion\": \"gemini-3.5-transcribe\",\n \"responseId\": \"zcazavHzD734juMPuKac8Aw\"\n}\n" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/openai-transcription/diarizes-speakers-into-segments.json b/packages/ai/test/fixtures/recordings/openai-transcription/diarizes-speakers-into-segments.json new file mode 100644 index 00000000000..dfe61f9ccfd --- /dev/null +++ b/packages/ai/test/fixtures/recordings/openai-transcription/diarizes-speakers-into-segments.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:openai-transcription", + "provider:openai", + "protocol:openai-transcription" + ], + "name": "openai-transcription/diarizes-speakers-into-segments", + "recordedAt": "2026-09-23T12:32:01.274Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.openai.com/v1/audio/transcriptions", + "headers": { + "content-type": "multipart/form-data; boundary=----WebKitFormBoundaryf476753a62264fa88f2d753a7e85c3a4" + }, + "body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\ngpt-4o-transcribe-diarize\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"response_format\"\r\n\r\ndiarized_json\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"chunking_strategy\"\r\n\r\nauto\r\n--BOUNDARY--\r\n" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"text\":\"Hello from OpenCode!\",\"task\":\"transcribe\",\"duration\":1.54125,\"segments\":[{\"type\":\"transcript.text.segment\",\"text\":\" Hello from OpenCode!\",\"speaker\":\"A\",\"start\":0.30000000000000004,\"end\":1.3499999999999999,\"id\":\"seg_0\"}],\"usage\":{\"type\":\"tokens\",\"total_tokens\":107,\"input_tokens\":15,\"input_token_details\":{\"text_tokens\":0,\"audio_tokens\":15},\"output_tokens\":92}}" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/openai-transcription/returns-whisper-word-timestamps.json b/packages/ai/test/fixtures/recordings/openai-transcription/returns-whisper-word-timestamps.json new file mode 100644 index 00000000000..c3048160ffb --- /dev/null +++ b/packages/ai/test/fixtures/recordings/openai-transcription/returns-whisper-word-timestamps.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:openai-transcription", + "provider:openai", + "protocol:openai-transcription" + ], + "name": "openai-transcription/returns-whisper-word-timestamps", + "recordedAt": "2026-09-23T12:32:02.972Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.openai.com/v1/audio/transcriptions", + "headers": { + "content-type": "multipart/form-data; boundary=----WebKitFormBoundarydfe85869a7d6448994e94f2f41d9ce55" + }, + "body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\nwhisper-1\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"response_format\"\r\n\r\nverbose_json\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"timestamp_granularities[]\"\r\n\r\nword\r\n--BOUNDARY--\r\n" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"task\":\"transcribe\",\"language\":\"english\",\"duration\":1.5399999618530273,\"text\":\"Hello from OpenCode!\",\"words\":[{\"word\":\"Hello\",\"start\":0.0,\"end\":0.6200000047683716},{\"word\":\"from\",\"start\":0.6200000047683716,\"end\":0.8799999952316284},{\"word\":\"OpenCode\",\"start\":0.8799999952316284,\"end\":1.399999976158142}],\"usage\":{\"type\":\"duration\",\"seconds\":2}}" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/openai-transcription/streams-text-deltas.json b/packages/ai/test/fixtures/recordings/openai-transcription/streams-text-deltas.json new file mode 100644 index 00000000000..cb1bcb351f4 --- /dev/null +++ b/packages/ai/test/fixtures/recordings/openai-transcription/streams-text-deltas.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:openai-transcription", + "provider:openai", + "protocol:openai-transcription" + ], + "name": "openai-transcription/streams-text-deltas", + "recordedAt": "2026-09-23T12:31:59.149Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.openai.com/v1/audio/transcriptions", + "headers": { + "content-type": "multipart/form-data; boundary=----WebKitFormBoundarya44da5204f47452ea1ef8dec145321dc" + }, + "body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\ngpt-4o-mini-transcribe\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"stream\"\r\n\r\ntrue\r\n--BOUNDARY--\r\n" + }, + "response": { + "status": 200, + "headers": { + "content-type": "text/event-stream; charset=utf-8" + }, + "body": "data: {\"type\":\"transcript.text.delta\",\"delta\":\"Hello\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\" from\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\" Open\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\"Code\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\".\"}\r\n\r\ndata: {\"type\":\"transcript.text.done\",\"text\":\"Hello from OpenCode.\",\"usage\":{\"type\":\"tokens\",\"total_tokens\":22,\"input_tokens\":15,\"input_token_details\":{\"text_tokens\":0,\"audio_tokens\":15},\"output_tokens\":7}}\r\n\r\ndata: [DONE]\r\n\r\n" + } + } + ] +} diff --git a/packages/ai/test/fixtures/recordings/openai-transcription/transcribes-with-a-language-hint.json b/packages/ai/test/fixtures/recordings/openai-transcription/transcribes-with-a-language-hint.json new file mode 100644 index 00000000000..1353e8ed0d1 --- /dev/null +++ b/packages/ai/test/fixtures/recordings/openai-transcription/transcribes-with-a-language-hint.json @@ -0,0 +1,32 @@ +{ + "version": 1, + "metadata": { + "tags": [ + "prefix:openai-transcription", + "provider:openai", + "protocol:openai-transcription" + ], + "name": "openai-transcription/transcribes-with-a-language-hint", + "recordedAt": "2026-09-23T12:31:56.657Z" + }, + "interactions": [ + { + "transport": "http", + "request": { + "method": "POST", + "url": "https://api.openai.com/v1/audio/transcriptions", + "headers": { + "content-type": "multipart/form-data; boundary=----WebKitFormBoundary484bf529b0064c43ac4a965c37433c9e" + }, + "body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\ngpt-transcribe\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"languages[]\"\r\n\r\nen\r\n--BOUNDARY--\r\n" + }, + "response": { + "status": 200, + "headers": { + "content-type": "application/json" + }, + "body": "{\"text\":\"Hello from OpenCode.\",\"languages\":[{\"code\":\"en\"}],\"usage\":{\"type\":\"duration\",\"seconds\":2}}" + } + } + ] +} diff --git a/packages/ai/test/promise.test.ts b/packages/ai/test/promise.test.ts index 4716f53e208..a4fee352353 100644 --- a/packages/ai/test/promise.test.ts +++ b/packages/ai/test/promise.test.ts @@ -1,10 +1,10 @@ import { describe, expect, test } from "bun:test" import { Effect, Layer } from "effect" import { HttpClientRequest } from "effect/unstable/http" -import { AIError, LLMEvent, Media, SpeechEvent } from "../src/index.js" +import { AIError, LLMEvent, Media, SpeechEvent, TranscriptionEvent } from "../src/index.js" import { RequestExecutor } from "../src/route.js" import { AI } from "../src/promise.js" -import { OpenAI, Runway } from "../src/providers.js" +import { AssemblyAI, OpenAI, Runway } from "../src/providers.js" import { handlerLayer } from "./lib/http.js" import { sseEvents } from "./lib/sse.js" @@ -18,7 +18,8 @@ const chatBody = sseEvents( /** * Executor layer that answers chat completions with SSE text, image generations with one base64 PNG, Runway video - * tasks with a queued submission that succeeds on the second poll, and speech with raw audio or SSE audio deltas. + * tasks with a queued submission that succeeds on the second poll, speech with raw audio or SSE audio deltas, OpenAI + * transcription with JSON or SSE text deltas, and AssemblyAI transcripts that complete on the first poll. */ const executor = (seen: Array) => RequestExecutor.layer.pipe( @@ -44,6 +45,27 @@ const executor = (seen: Array) => { headers: { "content-type": "text/event-stream" } }, ) : input.respond(Uint8Array.from([1, 2, 3]), { headers: { "content-type": "audio/pcm" } }) + if (web.url.endsWith("/audio/transcriptions")) + return input.text.includes('name="stream"') + ? input.respond( + sseEvents( + { type: "transcript.text.delta", delta: "Hello" }, + { type: "transcript.text.delta", delta: " there." }, + { type: "transcript.text.done", text: "Hello there." }, + ), + { headers: { "content-type": "text/event-stream" } }, + ) + : input.respond(JSON.stringify({ text: "Hello there." }), { + headers: { "content-type": "application/json" }, + }) + if (web.url.endsWith("/v2/transcript")) + return input.respond(JSON.stringify({ id: "tr_1", status: "queued" }), { + headers: { "content-type": "application/json" }, + }) + if (web.url.endsWith("/v2/transcript/tr_1")) + return input.respond(JSON.stringify({ id: "tr_1", status: "completed", text: "Hello there." }), { + headers: { "content-type": "application/json" }, + }) if (web.url.endsWith("/text_to_video")) return input.respond(JSON.stringify({ id: "task_1" }), { headers: { "content-type": "application/json" } }) if (web.url.endsWith("/tasks/task_1")) { @@ -134,6 +156,32 @@ describe("AI promise client", () => { await ai.dispose() }) + test("generates, streams, and starts transcriptions over the same runtime", async () => { + const ai = AI.make({ layer: executor([]) }) + const audio = Media.url("https://audio.test/hello.mp3") + const bytes = Media.bytes(Uint8Array.from([0x49, 0x44, 0x33]), "audio/mpeg") + + expect( + (await ai.transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio: bytes })).text, + ).toBe("Hello there.") + + const deltas: Array = [] + for await (const event of ai.transcription.stream({ + model: openai.transcription("gpt-4o-mini-transcribe"), + audio: bytes, + })) + if (TranscriptionEvent.is.textDelta(event)) deltas.push(event.delta) + expect(deltas).toEqual(["Hello", " there."]) + + const model = AssemblyAI.configure({ apiKey: "test", baseURL: "https://assemblyai.test" }).transcription( + "universal-2", + ) + const generation = await ai.transcription.start({ model, audio }) + expect(generation.token).toEqual({ transcriptID: "tr_1" }) + expect((await generation.await({ poll: { interval: 10 } })).text).toBe("Hello there.") + await ai.dispose() + }) + test("generates and streams speech over the same runtime", async () => { const ai = AI.make({ layer: executor([]) }) const model = openai.speech("gpt-4o-mini-tts") diff --git a/packages/ai/test/provider/assemblyai-transcription.recorded.test.ts b/packages/ai/test/provider/assemblyai-transcription.recorded.test.ts new file mode 100644 index 00000000000..bfbc5767486 --- /dev/null +++ b/packages/ai/test/provider/assemblyai-transcription.recorded.test.ts @@ -0,0 +1,39 @@ +import { describe, expect } from "bun:test" +import { Effect } from "effect" +import { Transcription } from "../../src/index.js" +import { AssemblyAI } from "../../src/providers.js" +import { recordedTests } from "../recorded-test.js" +import { TRANSCRIPT, audio, audioRecording } from "./transcription-recording.js" +import { queuedPoll } from "./queued-recording.js" + +// Recorded against the EU region: the US endpoint was unreachable from the recording network. Same API and shapes. +const model = AssemblyAI.configure({ + apiKey: process.env.ASSEMBLYAI_API_KEY ?? "fixture", + baseURL: "https://api.eu.assemblyai.com", +}).transcription("universal-3-5-pro") + +const recorded = recordedTests({ + prefix: "assemblyai-transcription", + provider: "assemblyai", + protocol: "assemblyai-transcription", + requires: ["ASSEMBLYAI_API_KEY"], + options: audioRecording, +}) + +describe("AssemblyAI Transcription recorded", () => { + recorded.effect( + "uploads, submits, and polls a transcript", + () => + Effect.gen(function* () { + const generation = yield* Transcription.start({ model, audio: yield* audio, diarize: true }) + const response = yield* generation.await({ poll: queuedPoll }) + + expect(response.text).toMatch(TRANSCRIPT) + expect(response.words?.[0]).toMatchObject({ text: "Hello", speaker: "A" }) + expect(response.words?.every((word) => word.startSeconds < 5 && word.endSeconds < 5)).toBe(true) + expect(response.segments?.[0]?.speaker).toBe("A") + expect(response.usage).toEqual({ type: "seconds", seconds: expect.any(Number) }) + }), + { timeout: 15 * 60 * 1000 }, + ) +}) diff --git a/packages/ai/test/provider/deepgram-transcription.recorded.test.ts b/packages/ai/test/provider/deepgram-transcription.recorded.test.ts new file mode 100644 index 00000000000..080432b7a8d --- /dev/null +++ b/packages/ai/test/provider/deepgram-transcription.recorded.test.ts @@ -0,0 +1,47 @@ +import { describe, expect } from "bun:test" +import { Effect, Stream } from "effect" +import { Transcription } from "../../src/index.js" +import { Deepgram } from "../../src/providers.js" +import { recordedTests } from "../recorded-test.js" +import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js" + +const model = Deepgram.configure({ apiKey: process.env.DEEPGRAM_API_KEY ?? "fixture" }).transcription("nova-3") + +const recorded = recordedTests({ + prefix: "deepgram-transcription", + provider: "deepgram", + protocol: "deepgram-transcription", + requires: ["DEEPGRAM_API_KEY"], + options: audioRecording, +}) + +describe("Deepgram Transcription recorded", () => { + recorded.effect("transcribes raw audio with speakers and word timestamps", () => + Effect.gen(function* () { + const request = Transcription.request({ model, audio: yield* audio, diarize: true, timestamps: "word" }) + const response = yield* Transcription.generate(request) + + expect(response.text).toMatch(TRANSCRIPT) + expect(response.words?.map((word) => word.text)).toEqual(["Hello", "from", "OpenCode."]) + expect(response.words?.every((word) => word.speaker === "0" && word.confidence !== undefined)).toBe(true) + expect(response.segments).toEqual([ + { text: response.text, startSeconds: expect.any(Number), endSeconds: expect.any(Number), speaker: "0" }, + ]) + expect(response.language).toBe("en") + expect(response.usage).toEqual({ type: "seconds", seconds: response.durationSeconds }) + + const events = Array.from(yield* Stream.runCollect(Transcription.stream(request))) + expect(events.map((event) => event.type)).toEqual(["finish"]) + }), + ) + + recorded.effect("splits utterances at speaker changes", () => + Effect.gen(function* () { + const response = yield* Transcription.generate({ model, audio: yield* dialog, diarize: true }) + + expect(response.segments?.map((segment) => segment.speaker)).toEqual(["0", "1"]) + expect(response.segments?.[0].text).toMatch(/release ship\?$/) + expect(response.segments?.[1].text).toMatch(/shipped this morning\.$/) + }), + ) +}) diff --git a/packages/ai/test/provider/fal-video.recorded.test.ts b/packages/ai/test/provider/fal-video.recorded.test.ts index 27a70aa738a..e6d42770580 100644 --- a/packages/ai/test/provider/fal-video.recorded.test.ts +++ b/packages/ai/test/provider/fal-video.recorded.test.ts @@ -3,7 +3,7 @@ import { Effect } from "effect" import { Video } from "../../src/index.js" import { Fal } from "../../src/providers.js" import { recordedTests } from "../recorded-test.js" -import { videoPoll } from "./video-recording.js" +import { queuedPoll } from "./queued-recording.js" const model = Fal.configure({ apiKey: process.env.FAL_KEY ?? "fixture", @@ -29,7 +29,7 @@ describe("fal Video recorded", () => { resolution: "720p", providerOptions: { duration: "4s" }, }, - { poll: videoPoll }, + { poll: queuedPoll }, ) expect(response.videos).toHaveLength(1) diff --git a/packages/ai/test/provider/google-transcription.recorded.test.ts b/packages/ai/test/provider/google-transcription.recorded.test.ts new file mode 100644 index 00000000000..c92cfdc52ad --- /dev/null +++ b/packages/ai/test/provider/google-transcription.recorded.test.ts @@ -0,0 +1,54 @@ +import { describe, expect } from "bun:test" +import { Effect, Stream } from "effect" +import { Transcription, TranscriptionEvent } from "../../src/index.js" +import { Google } from "../../src/providers.js" +import { recordedTests } from "../recorded-test.js" +import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js" + +const model = Google.configure({ apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY ?? "fixture" }).transcription( + "gemini-3.5-transcribe", +) + +const recorded = recordedTests({ + prefix: "google-transcription", + provider: "google", + protocol: "google-transcription", + requires: ["GOOGLE_GENERATIVE_AI_API_KEY"], + options: audioRecording, +}) + +describe("Google Transcription recorded", () => { + recorded.effect("transcribes with speakers and word timestamps", () => + Effect.gen(function* () { + const response = yield* Transcription.generate({ model, audio: yield* audio, diarize: true, language: "en-US" }) + + expect(response.text).toMatch(TRANSCRIPT) + expect(response.words?.length).toBeGreaterThan(2) + expect(response.words?.every((word) => word.endSeconds > word.startSeconds)).toBe(true) + expect(response.segments).toEqual([ + { + text: response.text, + startSeconds: response.words?.[0]?.startSeconds, + endSeconds: response.words?.at(-1)?.endSeconds, + speaker: expect.any(String), + }, + ]) + expect(response.usage?.type).toBe("tokens") + }), + ) + + recorded.effect("streams speaker turns", () => + Effect.gen(function* () { + const events = Array.from( + yield* Stream.runCollect(Transcription.stream({ model, audio: yield* dialog, diarize: true })), + ) + const finish = events.at(-1) + if (finish === undefined || !TranscriptionEvent.is.finish(finish)) throw new Error("Expected a finish event") + + expect(finish.text).toMatch(/release ship\? Yes, it shipped this morning\.$/) + const speakers = events.filter(TranscriptionEvent.is.segment).map((event) => event.segment.speaker) + expect(speakers).toHaveLength(2) + expect(new Set(speakers).size).toBe(2) + }), + ) +}) diff --git a/packages/ai/test/provider/google-video.recorded.test.ts b/packages/ai/test/provider/google-video.recorded.test.ts index db23091c2f7..b8daf170fb0 100644 --- a/packages/ai/test/provider/google-video.recorded.test.ts +++ b/packages/ai/test/provider/google-video.recorded.test.ts @@ -3,7 +3,7 @@ import { Effect } from "effect" import { Video } from "../../src/index.js" import { Google } from "../../src/providers.js" import { recordedTests } from "../recorded-test.js" -import { videoPoll } from "./video-recording.js" +import { queuedPoll } from "./queued-recording.js" const model = Google.configure({ apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY ?? "fixture", @@ -28,7 +28,7 @@ describe("Google Veo recorded", () => { aspectRatio: "16:9", durationSeconds: 4, }, - { poll: videoPoll }, + { poll: queuedPoll }, ) expect(response.videos).toHaveLength(1) diff --git a/packages/ai/test/provider/openai-transcription.recorded.test.ts b/packages/ai/test/provider/openai-transcription.recorded.test.ts new file mode 100644 index 00000000000..e8ce460bf79 --- /dev/null +++ b/packages/ai/test/provider/openai-transcription.recorded.test.ts @@ -0,0 +1,86 @@ +import { describe, expect } from "bun:test" +import { Effect, Stream } from "effect" +import { Transcription, TranscriptionEvent } from "../../src/index.js" +import { OpenAI } from "../../src/providers.js" +import { recordedTests } from "../recorded-test.js" +import { TRANSCRIPT, audio, audioRecording } from "./transcription-recording.js" + +const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY ?? "fixture" }) + +const recorded = recordedTests({ + prefix: "openai-transcription", + provider: "openai", + protocol: "openai-transcription", + requires: ["OPENAI_API_KEY"], + options: audioRecording, +}) + +describe("OpenAI Transcription recorded", () => { + recorded.effect("transcribes with a language hint", () => + Effect.gen(function* () { + const response = yield* Transcription.generate({ + model: openai.transcription("gpt-transcribe"), + audio: yield* audio, + language: "en", + }) + + expect(response.text).toMatch(TRANSCRIPT) + expect(response.language).toBe("en") + expect(response.usage?.type).toBeDefined() + }), + ) + + recorded.effect("streams text deltas", () => + Effect.gen(function* () { + const events = Array.from( + yield* Stream.runCollect( + Transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio: yield* audio }), + ), + ) + const finish = events.at(-1) + if (finish === undefined || !TranscriptionEvent.is.finish(finish)) throw new Error("Expected a finish event") + + const deltas = events.filter(TranscriptionEvent.is.textDelta).map((event) => event.delta) + expect(deltas.length).toBeGreaterThan(1) + expect(deltas.join("")).toBe(finish.text) + expect(finish.text).toMatch(TRANSCRIPT) + expect(finish.usage).toMatchObject({ type: "tokens", input: expect.any(Number), output: expect.any(Number) }) + }), + ) + + recorded.effect("diarizes speakers into segments", () => + Effect.gen(function* () { + const response = yield* Transcription.generate({ + model: openai.transcription("gpt-4o-transcribe-diarize"), + audio: yield* audio, + diarize: true, + }) + + expect(response.text).toMatch(TRANSCRIPT) + expect(response.segments).toEqual([ + { + text: expect.stringMatching(TRANSCRIPT), + startSeconds: expect.any(Number), + endSeconds: expect.any(Number), + speaker: "A", + }, + ]) + expect(response.durationSeconds).toBeGreaterThan(1) + }), + ) + + recorded.effect("returns whisper word timestamps", () => + Effect.gen(function* () { + const response = yield* Transcription.generate({ + model: openai.transcription("whisper-1"), + audio: yield* audio, + timestamps: "word", + }) + + expect(response.words?.map((word) => word.text.toLowerCase())).toEqual(["hello", "from", "opencode"]) + expect(response.words?.every((word) => word.endSeconds > word.startSeconds)).toBe(true) + expect(response.language).toBe("english") + expect(response.usage).toEqual({ type: "seconds", seconds: expect.any(Number) }) + }), + ) +}) diff --git a/packages/ai/test/provider/video-recording.ts b/packages/ai/test/provider/queued-recording.ts similarity index 63% rename from packages/ai/test/provider/video-recording.ts rename to packages/ai/test/provider/queued-recording.ts index bbb95897e84..257ca06a67b 100644 --- a/packages/ai/test/provider/video-recording.ts +++ b/packages/ai/test/provider/queued-recording.ts @@ -1,10 +1,10 @@ import type { Poll } from "../../src/index.js" /** - * Video generations take minutes live, so record with a realistic interval. Replay consumes the recorded polls in + * Queued generations take minutes live, so record with a realistic interval. Replay consumes the recorded polls in * cassette order regardless of timing, so it runs them back-to-back instead of sleeping between each one. */ -export const videoPoll: Poll = { +export const queuedPoll: Poll = { interval: process.env.RECORD === "true" ? "10 seconds" : "10 millis", timeout: "15 minutes", } diff --git a/packages/ai/test/provider/runway-video.recorded.test.ts b/packages/ai/test/provider/runway-video.recorded.test.ts index 672119efc40..0bec3c7b66f 100644 --- a/packages/ai/test/provider/runway-video.recorded.test.ts +++ b/packages/ai/test/provider/runway-video.recorded.test.ts @@ -3,7 +3,7 @@ import { Effect } from "effect" import { Video } from "../../src/index.js" import { Runway } from "../../src/providers.js" import { recordedTests } from "../recorded-test.js" -import { videoPoll } from "./video-recording.js" +import { queuedPoll } from "./queued-recording.js" const model = Runway.configure({ apiKey: process.env.RUNWAYML_API_SECRET ?? "fixture", @@ -28,7 +28,7 @@ describe("Runway recorded", () => { aspectRatio: "1280:720", durationSeconds: 2, }, - { poll: videoPoll }, + { poll: queuedPoll }, ) expect(response.videos).toHaveLength(1) diff --git a/packages/ai/test/provider/transcription-recording.ts b/packages/ai/test/provider/transcription-recording.ts new file mode 100644 index 00000000000..d01b4aa0e22 --- /dev/null +++ b/packages/ai/test/provider/transcription-recording.ts @@ -0,0 +1,41 @@ +import type { HttpRecorder } from "@opencode/http-recorder" +import { Effect } from "effect" +import { Media } from "../../src/index.js" + +const fixture = (name: string) => + Effect.promise(() => Bun.file(new URL(`../fixtures/audio/${name}`, import.meta.url)).bytes()).pipe( + Effect.map((bytes) => Media.bytes(bytes, "audio/mpeg")), + ) + +/** "Hello from OpenCode." spoken by Deepgram Aura (`aura-2-thalia-en`, 32 kbps MP3). */ +export const audio = fixture("hello.mp3") + +/** "Did the release ship?" (`aura-2-thalia-en`), then "Yes, it shipped this morning." (`aura-2-orion-en`). */ +export const dialog = fixture("dialog.mp3") + +/** Providers disagree on "OpenCode" versus "Open Code" and on the final punctuation. */ +export const TRANSCRIPT = /^hello,? from open ?code[.!]?$/i + +/** + * Request bodies are snapshotted as text, so the audio would be stored as mangled UTF-8. Replace it with a marker and + * pin the random multipart boundary, which keeps the remaining form fields and query checked on replay. + */ +const redactAudio = (body: string) => { + const boundary = /^--(\S+)\r\n/.exec(body)?.[1] + if (boundary === undefined) return body.includes("\uFFFD") ? "[audio]" : body + return body + .replaceAll(boundary, "BOUNDARY") + .replace(/(filename="[^"]*"\r\nContent-Type: [^\r]*\r\n\r\n)[\s\S]*?(\r\n--BOUNDARY)/, "$1[audio]$2") +} + +const contentType = (snapshot: HttpRecorder.RequestSnapshot) => + snapshot.headers["content-type"]?.replace(/boundary=.*/, "") + +export const audioRecording: HttpRecorder.RecorderOptions = { + redact: { body: redactAudio }, + match: (incoming, recorded) => + incoming.method === recorded.method && + incoming.url === recorded.url && + contentType(incoming) === contentType(recorded) && + incoming.body === recorded.body, +} diff --git a/packages/ai/test/provider/xai-video.recorded.test.ts b/packages/ai/test/provider/xai-video.recorded.test.ts index b7dfd3791f2..96b25de013b 100644 --- a/packages/ai/test/provider/xai-video.recorded.test.ts +++ b/packages/ai/test/provider/xai-video.recorded.test.ts @@ -3,7 +3,7 @@ import { Effect, Stream } from "effect" import { Video } from "../../src/index.js" import { XAI } from "../../src/providers.js" import { recordedTests } from "../recorded-test.js" -import { videoPoll } from "./video-recording.js" +import { queuedPoll } from "./queued-recording.js" const model = XAI.configure({ apiKey: process.env.XAI_API_KEY ?? "fixture", @@ -28,7 +28,7 @@ describe("xAI Video recorded", () => { resolution: "480p", durationSeconds: 2, }) - const observed = yield* generation.events({ poll: videoPoll }).pipe(Stream.runCollect) + const observed = yield* generation.events({ poll: queuedPoll }).pipe(Stream.runCollect) const progress = Array.from(observed).flatMap((event) => event.type === "generation-progress" && event.progress !== undefined ? [event.progress] : [], ) diff --git a/packages/ai/test/recorded-test.ts b/packages/ai/test/recorded-test.ts index 7998c03c5dd..2c507f5a0dd 100644 --- a/packages/ai/test/recorded-test.ts +++ b/packages/ai/test/recorded-test.ts @@ -8,11 +8,13 @@ import { LLMClient, RequestExecutor } from "../src/route.js" import { ImageClient } from "../src/image-client.js" import { VideoClient } from "../src/video-client.js" import { SpeechClient } from "../src/speech-client.js" +import { TranscriptionClient } from "../src/transcription-client.js" import { EvaluationClient } from "../src/experimental/evaluation-client.js" import type { Service as EvaluationClientService } from "../src/experimental/evaluation-client.js" import type { Service as ImageClientService } from "../src/image-client.js" import type { Service as VideoClientService } from "../src/video-client.js" import type { Service as SpeechClientService } from "../src/speech-client.js" +import type { Service as TranscriptionClientService } from "../src/transcription-client.js" import type { Service as LLMClientService } from "../src/route/client.js" import type { Service as RequestExecutorService } from "../src/route/executor.js" import { @@ -30,6 +32,7 @@ type RecordedEnv = | ImageClientService | VideoClientService | SpeechClientService + | TranscriptionClientService | EvaluationClientService | Socket.WebSocketConstructor @@ -107,6 +110,7 @@ export const recordedTests = (options: RecordedTestsOptions) => ImageClient.layer.pipe(Layer.provide(requestExecutor)), VideoClient.layer.pipe(Layer.provide(requestExecutor)), SpeechClient.layer.pipe(Layer.provide(requestExecutor)), + TranscriptionClient.layer.pipe(Layer.provide(requestExecutor)), EvaluationClient.layer.pipe(Layer.provide(requestExecutor)), webSocket, ) diff --git a/packages/ai/test/transcription.test.ts b/packages/ai/test/transcription.test.ts new file mode 100644 index 00000000000..aa469a141b2 --- /dev/null +++ b/packages/ai/test/transcription.test.ts @@ -0,0 +1,174 @@ +import { describe, expect } from "bun:test" +import { Effect, Fiber, Layer, Stream } from "effect" +import * as TestClock from "effect/testing/TestClock" +import { HttpClientRequest } from "effect/unstable/http" +import { Media, Transcription, TranscriptionClient } from "../src/index.js" +import { AssemblyAI, Deepgram, Google, OpenAI } from "../src/providers.js" +import { it } from "./lib/effect.js" +import { dynamicResponse } from "./lib/http.js" + +const layer = (handler: Parameters[0]) => + TranscriptionClient.layer.pipe(Layer.provideMerge(dynamicResponse(handler))) + +const audio = Media.bytes(Uint8Array.from([0x49, 0x44, 0x33, 1, 2, 3]), "audio/mpeg") +const openai = OpenAI.configure({ apiKey: "test", baseURL: "https://openai.test/v1" }) +const deepgram = Deepgram.configure({ apiKey: "test", baseURL: "https://deepgram.test" }).transcription("nova-3") +const google = Google.configure({ apiKey: "test", baseURL: "https://google.test/v1beta" }).transcription( + "gemini-3.5-transcribe", +) +const assemblyai = AssemblyAI.configure({ apiKey: "aai-key", baseURL: "https://assemblyai.test" }).transcription( + "universal-3-5-pro", +) + +describe("Transcription", () => { + it.effect("rejects what a route cannot honor before sending anything", () => + Effect.gen(function* () { + const errors = yield* Effect.all( + [ + Stream.runCollect(Transcription.stream({ model: openai.transcription("whisper-1"), audio })), + Transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio, diarize: true }), + Transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio, timestamps: "word" }), + Transcription.generate({ model: openai.transcription("gpt-4o-transcribe-diarize"), audio, prompt: "Names" }), + Transcription.generate({ model: deepgram, audio, prompt: "OpenCode" }), + Transcription.generate({ model: google, audio, speakers: 2 }), + Transcription.start({ model: deepgram, audio }), + Transcription.generate({ model: deepgram, audio, http: { body: { callback: "https://hook.test" } } }), + Transcription.generate({ + model: openai.transcription("gpt-transcribe"), + audio: Media.url("https://a.test/x.mp3"), + }), + Transcription.start({ + model: assemblyai, + audio: Media.ref("file_1", { provider: "openai", mediaType: "audio/mpeg" }), + }), + Transcription.generate({ + model: openai.transcription("gpt-transcribe"), + audio: Media.bytes(Uint8Array.from([1, 2, 3]), "audio/x-unknown"), + }), + Transcription.generate({ + model: Google.configure({ apiKey: "test" }).transcription("gemini-3.6-flash"), + audio, + }), + ].map((effect) => Effect.flip(effect)), + ) + expect(errors.map((error) => [error.reason._tag, "operation" in error.reason && error.reason.operation])).toEqual( + [ + ["UnsupportedOperation", "media.stream"], + ["UnsupportedOperation", "media.diarize"], + ["UnsupportedOperation", "media.timestamps"], + ["UnsupportedOperation", "media.prompt"], + ["UnsupportedOperation", "media.prompt"], + ["UnsupportedOperation", "media.speakers"], + ["UnsupportedOperation", "transcription.start"], + ["InvalidRequest", false], + ["InvalidRequest", false], + ["InvalidRequest", false], + ["InvalidRequest", false], + ["UnsupportedOperation", "transcription.model"], + ], + ) + }).pipe(Effect.provide(layer(() => Effect.die("an unsupported request reached the network")))), + ) + + it.effect( + "uploads inline audio to AssemblyAI, resumes polling from a persisted token, and surfaces failed transcripts", + () => + Effect.gen(function* () { + const calls: Array<{ + readonly method: string + readonly url: string + readonly auth: string | null + readonly body: string + }> = [] + const json = (value: unknown) => ({ + body: JSON.stringify(value), + init: { headers: { "content-type": "application/json" } }, + }) + const polls = () => calls.filter((call) => call.url.endsWith("/tr_1")).length + const failed = JSON.stringify({ id: "tr_2", status: "error", error: "Audio file could not be decoded" }) + const program = Effect.gen(function* () { + const started = yield* Transcription.start({ model: assemblyai, audio, diarize: true, speakers: 2 }) + expect(started.status).toBe("queued") + const resumed = yield* Transcription.resume(assemblyai, JSON.parse(JSON.stringify(started.token))) + const response = yield* resumed.await({ poll: { interval: "1 second" } }) + const failure = yield* Transcription.resume(assemblyai, { transcriptID: "tr_2" }).pipe( + Effect.flatMap((generation) => generation.await()), + Effect.flip, + ) + return { response, failure } + }).pipe( + Effect.provide( + layer((input) => + Effect.gen(function* () { + const web = yield* HttpClientRequest.toWeb(input.request).pipe(Effect.orDie) + calls.push({ + method: web.method, + url: web.url, + auth: web.headers.get("authorization"), + body: input.text, + }) + if (web.url.endsWith("/tr_2")) + return input.respond(failed, { headers: { "content-type": "application/json" } }) + const reply = web.url.endsWith("/v2/upload") + ? json({ upload_url: "https://cdn.assemblyai.test/upload/1" }) + : web.method === "POST" + ? json({ id: "tr_1", status: "queued" }) + : polls() < 3 + ? json({ id: "tr_1", status: "processing" }) + : json({ + id: "tr_1", + status: "completed", + text: "Hello there.", + words: [ + { text: "Hello", start: 250, end: 700, confidence: 0.9, speaker: "A" }, + { text: "there.", start: 700, end: 1250, confidence: 0.8, speaker: "B" }, + ], + utterances: [ + { text: "Hello", start: 250, end: 700, speaker: "A" }, + { text: "there.", start: 700, end: 1250, speaker: "B" }, + ], + language_code: "en_us", + audio_duration: 2, + }) + return input.respond(reply.body, reply.init) + }), + ), + ), + ) + + const fiber = yield* Effect.forkChild(program) + yield* TestClock.adjust("5 seconds") + const { response, failure } = yield* Fiber.join(fiber) + + expect(calls.slice(0, 2).map((call) => [call.method, call.url, call.auth])).toEqual([ + ["POST", "https://assemblyai.test/v2/upload", "aai-key"], + ["POST", "https://assemblyai.test/v2/transcript", "aai-key"], + ]) + expect(JSON.parse(calls[1].body)).toEqual({ + audio_url: "https://cdn.assemblyai.test/upload/1", + speech_models: ["universal-3-5-pro"], + language_detection: true, + speaker_labels: true, + speakers_expected: 2, + }) + expect(calls[2].url).toBe("https://assemblyai.test/v2/transcript/tr_1") + expect(response.segments).toEqual([ + { text: "Hello", startSeconds: 0.25, endSeconds: 0.7, speaker: "A" }, + { text: "there.", startSeconds: 0.7, endSeconds: 1.25, speaker: "B" }, + ]) + expect(response.words?.[1]).toEqual({ + text: "there.", + startSeconds: 0.7, + endSeconds: 1.25, + speaker: "B", + confidence: 0.8, + }) + expect(response).toMatchObject({ + text: "Hello there.", + language: "en_us", + usage: { type: "seconds", seconds: 2 }, + }) + expect(failure.reason).toMatchObject({ _tag: "ProviderInternal", body: failed }) + }), + ) +}) diff --git a/packages/ai/test/transcription.types.ts b/packages/ai/test/transcription.types.ts new file mode 100644 index 00000000000..e14ca86624e --- /dev/null +++ b/packages/ai/test/transcription.types.ts @@ -0,0 +1,34 @@ +import type { Stream } from "effect" +import { Media, Transcription, type TranscriptionEvent } from "../src/index.js" +import { AssemblyAI, Deepgram, OpenAI } from "../src/providers.js" + +type StreamItem = T extends Stream.Stream ? A : never +type Equal = [A, B] extends [B, A] ? true : false +type Assert = T + +const audio = Media.url("https://example.com/call.mp3") +const deepgram = Deepgram.configure({ apiKey: "test" }).transcription("nova-3") + +Transcription.generate({ + model: deepgram, + audio, + timestamps: "word", + providerOptions: { keyterm: ["OpenCode"], diarize_model: "future-model", futureOption: true }, +}) +Transcription.start({ model: AssemblyAI.configure({ apiKey: "test" }).transcription("universal-3-5-pro"), audio }) + +// @ts-expect-error Known provider options are inferred from the selected model. +Transcription.generate({ model: deepgram, audio, providerOptions: { keyterm: "OpenCode" } }) +// @ts-expect-error Timestamp granularity is a closed set. +Transcription.generate({ model: deepgram, audio, timestamps: "character" }) +// @ts-expect-error Audio is required. +Transcription.request({ model: deepgram }) +// @ts-expect-error Speech models cannot be used for transcription requests. +Transcription.request({ model: OpenAI.configure({ apiKey: "test" }).speech("gpt-4o-mini-tts"), audio }) + +const streamed = Transcription.stream({ + model: OpenAI.configure({ apiKey: "test" }).transcription("gpt-transcribe"), + audio, +}) +type StreamEvent = Assert, TranscriptionEvent>> +void (true satisfies StreamEvent)