feat(ai): add transcription across inline, streamed, and queued routes (#50910)

This commit is contained in:
Shoubhit Dash
2026-09-23 21:15:50 +05:30
committed by GitHub
parent b6bc55764c
commit dbaa57a21b
56 changed files with 2840 additions and 286 deletions
+3 -1
View File
@@ -100,7 +100,9 @@ Media does not fit the SSE-frames-to-event-state-machine LLM route. `MediaRoute.
`MediaProtocol.queued` is the submit-then-poll kind every video route uses: `start` (body + decode into `{ token, snapshot }`), `status`, `result`, and optional `cancel`, each addressed by a route-owned `token` whose `Schema.Codec` makes it serializable. `MediaRoute.inline` and `MediaRoute.queued` compose the two kinds with `Endpoint` and `Auth`; the queued route decodes the token once at the boundary (`start` output or `resume` input) and closes over it in a token-free `GenerationRoute` (`status`/`result`/`cancel` are plain Effects), so `Generation` never sees the token's shape and only carries the encoded JSON for persistence. Polls reuse the route's auth and deployment headers plus the request's `http` overlay after `start`, and resolve relative paths against the route base URL (provider-issued absolute URLs such as fal's `status_url` pass through). `result` is always its own GET even when the provider returns output inside the status document, so `Generation.await` behaves the same after `start` and after `resume`. `PollContext.auth` carries only what `Auth` added so protocols can hand download credentials to output assets as transient `Media.Asset.headers` (Veo) — never part of `source` or JSON. Status strings map through a per-protocol `STATUS` table via `MediaProtocol.status`; terminal generations without output fail through `output.ended` / `output.contentPolicy` with the provider document on `reason.body`. `Generation.AwaitOptions` (`{ poll?: Poll }`) is the one options type for `await`, `events`, `Video.generate`, and `Video.stream`.
`MediaProtocol.stream` is the incremental kind every speech route uses, with the same discipline as LLM protocols. `MediaRoute.stream` submits the caller's request as `MediaProtocol.Addressed<Request>` (`{ ...request, mode }`, `mode: "generate" | "stream"`), so one provider stays one protocol: `body.from`, the endpoint path, and `frames` read `request.mode` to pick the body, path, and framing. `frames(bytes, context)` returns frames — `Framing.sse`, `Framing.lines`, `Framing.document` (a single-document response shaped like a streamed record), or the raw `bytes` for chunked audio. `initial()` is fresh per-response parser state; `step` folds each frame into it and emits modality events; `finish(state, context)` runs once after the last frame with the request, body, and observed `http` (header-only usage lives there) and emits exactly one terminal event or fails with `MediaProtocol.incomplete`. Keep parser state to real accumulators and derive anything the request or body determines in `finish`. `generate` runs the same stream and folds it with the modality's `collect`. Request-derived URL parameters go on the JSON body's `query`, applied before route and caller `http.query`. Decode frames with `MediaProtocol.decodeFrame` and raise stream-time failures with `MediaProtocol.frameError` (the frame stays on `reason.body`); protocols never thread HTTP context, because the route fills `reason.http` on stream errors that lack it. Speech protocols share `protocols/utils/speech-stream.ts` for deltas, timestamps, voice ids, PCM and container descriptions, and the terminal asset.
`MediaProtocol.stream` is the incremental kind every speech route uses, with the same discipline as LLM protocols. `MediaRoute.stream` submits the caller's request as `MediaProtocol.Addressed<Request>` (`{ ...request, mode }`, `mode: "generate" | "stream"`), so one provider stays one protocol: `body.from`, the endpoint path, and `frames` read `request.mode` to pick the body, path, and framing. `frames(bytes, context)` returns frames — `Framing.sse`, `Framing.lines`, `Framing.document` (a single-document response shaped like a streamed record), or the raw `bytes` for chunked audio. `initial()` is fresh per-response parser state; `step` folds each frame into it and emits modality events; `finish(state, context)` runs once after the last frame with the request, body, and observed `http` (header-only usage lives there) and emits exactly one terminal event or fails with `MediaProtocol.incomplete`. Keep parser state to real accumulators and derive anything the request or body determines in `finish`. `generate` runs the same stream and folds it with the modality's `collect`. Request-derived URL parameters go on the body's `query` (array values repeat the parameter), applied before route and caller `http.query`. Decode frames with `MediaProtocol.decodeFrame` and raise stream-time failures with `MediaProtocol.frameError` (the frame stays on `reason.body`); protocols never thread HTTP context, because the route fills `reason.http` on stream errors that lack it. Speech protocols share `protocols/utils/speech-stream.ts` for deltas, timestamps, voice ids, PCM and container descriptions, and the terminal asset.
Transcription uses all three kinds (OpenAI and Gemini stream, Deepgram is inline, AssemblyAI is queued): every route carries its `kind` and `TranscriptionClient` dispatches on it. Bodies are `json`, `multipart`, or `binary` (a raw upload), and a queued protocol that must upload media before submitting implements `start.prepare` (`MediaProtocol.Prepare`; AssemblyAI `/v2/upload`).
### URL Construction
+65 -1
View File
@@ -766,6 +766,69 @@ for await (const event of ai.speech.stream({ model, text: "Hello from OpenCode."
}
```
## Transcription
Transcription (speech-to-text) is the one modality whose providers use every route kind: OpenAI and Gemini stream,
Deepgram answers inline, and AssemblyAI is queued. `Transcription.generate` and `Transcription.stream` work on all of
them; `Transcription.start` / `resume` return a `Generation` on queued routes and fail with `UnsupportedOperation`
elsewhere. Models come from `.transcription(...)` selectors on the `OpenAI`, `Google`, `Deepgram`, and `AssemblyAI`
facades. Common fields (`language`, `prompt`, `timestamps: "none" | "segment" | "word"`, `diarize`, `speakers`) lower
natively or fail with a typed `AIError` before any network call; a route may return more than asked.
```ts
import { Media, Transcription, TranscriptionEvent } from "@opencode/ai"
import { AssemblyAI, Deepgram, OpenAI } from "@opencode/ai/providers"
const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY })
const program = Effect.gen(function* () {
const audio = yield* Media.file("./call.mp3")
// Speaker-labelled segments; labels are provider-native strings ("A", "0", "spk:0").
const response = yield* Transcription.generate({
model: Deepgram.configure({ apiKey }).transcription("nova-3"),
audio,
diarize: true,
timestamps: "word",
})
response.text // "Hello from OpenCode."
response.segments // [{ text, startSeconds, endSeconds, speaker: "0" }]
response.words // [{ text, startSeconds, endSeconds, speaker, confidence }]
response.language // the provider's own value, lowercased ("en", "english", "en_us")
// Text deltas as the model transcribes, then one finish carrying the whole transcript.
yield* Transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio }).pipe(
Stream.tap((event) => (TranscriptionEvent.is.textDelta(event) ? Console.log(event.delta) : Effect.void)),
Stream.runDrain,
)
// Queued: persist the token, resume from another process, and await.
const model = AssemblyAI.configure({ apiKey }).transcription("universal-3-5-pro")
const generation = yield* Transcription.start({ model, audio })
const resumed = yield* Transcription.resume(model, JSON.parse(JSON.stringify(generation.token)))
const transcript = yield* resumed.await({ poll: { interval: "3 seconds" } })
})
```
Inline routes emit only `finish` from `stream` (no faked deltas); queued routes emit `generation-queued` /
`generation-progress` before it. `TranscriptionClient.layer` needs `RequestExecutor.Service`.
Provider notes:
- **OpenAI** takes inline audio only; `diarize` needs `gpt-4o-transcribe-diarize`, timestamps need `whisper-1`, and `whisper-1` does not stream.
- **Gemini** needs a transcribe model (`gemini-3.5-transcribe`); `prompt` and `speakers` fail typed.
- **Deepgram** detects the language unless `language` is set; vocabulary goes in `providerOptions.keyterm`.
- **AssemblyAI** uploads inline audio before submitting and is the only route that accepts `speakers`.
The promise client mirrors the Effect API:
```ts
const text = (await ai.transcription.generate({ model, audio })).text
for await (const event of ai.transcription.stream({ model, audio })) if (event.type === "text-delta") write(event.delta)
const generation = await ai.transcription.start({ model: assemblyai, audio })
const transcript = await generation.await({ poll: { interval: 3_000 } })
```
## Public API
- **`LLM.request({...})`** — build a provider-neutral `LLMRequest`. Accepts ergonomic inputs (`system: string`, `prompt: string`) that normalize into the canonical Schema classes.
@@ -778,7 +841,8 @@ for await (const event of ai.speech.stream({ model, text: "Hello from OpenCode."
- **`Media`** — the shared asset type (`Media.Asset`, `Media.Source`) and constructors used by messages, tool results, and media requests.
- **`Generation`** — provider-neutral handle for an in-flight media generation (`await`, `refresh`, `cancel`, `events`) used by queued media routes.
- **`Speech.request` / `Speech.generate` / `Speech.stream`** — text-to-speech through a provider-neutral request; `SpeechClient` is its Effect service and layer.
- **`@opencode/ai/promise`** — `AI.make({ layer? })` and a default `ai` client exposing `llm`, `image`, `video`, and `speech` as Promise / `AsyncIterable` APIs.
- **`Transcription.request` / `generate` / `stream` / `start` / `resume`** — speech-to-text over inline, streaming, and queued routes; `TranscriptionClient` is its Effect service and layer.
- **`@opencode/ai/promise`** — `AI.make({ layer? })` and a default `ai` client exposing `llm`, `image`, `video`, `speech`, and `transcription` as Promise / `AsyncIterable` APIs.
## Testing
+63 -14
View File
@@ -1,6 +1,6 @@
# Media generation in `@opencode/ai` — public API direction
Status: phases 1–2 implemented; phase 3 Speech implemented, Transcription pending; phases 4–5 proposal.
Status: phases 1–3 implemented (Speech and Transcription); phases 4–5 proposal.
## Goal
@@ -249,23 +249,71 @@ Deferred: `Speech.session(...)` — input-streaming TTS where text arrives incre
#### Transcription (STT)
Shipped as the second half of phase 3 (`src/transcription.ts`, `src/transcription-client.ts`, protocols
`openai-transcription`, `google-transcription`, `deepgram-transcription`, `assemblyai-transcription`; new `AssemblyAI`
facade).
```ts
const request = Transcription.request({
model: openai.transcription("gpt-4o-transcribe"),
audio: Media.file("./call.wav"),
language: "en",
prompt: "Names: Shoubhit, OpenCode.",
timestamps: "word", // none | segment | word
model: openai.transcription("gpt-4o-transcribe-diarize"),
audio: yield* Media.file("./call.wav"),
language: "en", // provider-native passthrough
timestamps: "segment", // none | segment | word
diarize: true,
providerOptions: { chunkingStrategy: "auto" },
speakers: 2, // expected count, hint only (AssemblyAI)
providerOptions: { known_speaker_names: ["agent"] },
})
const response = yield* Transcription.generate(request)
response.text; response.segments; response.words; response.language; response.durationSeconds
yield* Transcription.stream(request) // Stream<TranscriptionEvent>: text-delta | segment | finish
response.text; response.segments; response.words; response.language; response.durationSeconds; response.usage
yield* Transcription.stream(request) // Stream<TranscriptionEvent>: generation-queued | generation-progress | text-delta | segment | finish
const generation = yield* Transcription.start(request) // queued routes only
yield* Transcription.resume(model, token)
```
Realtime STT over WebSocket is the same future `session` shape as input-streaming TTS.
Transcription is the first modality whose providers span all three protocol kinds, and it needed no fourth kind.
Every `MediaRoute` now carries its `kind`; `TranscriptionRoute` is the union of the inline, stream, and queued routes;
`TranscriptionModel.fromRoute` is overloaded per protocol kind (arity picks the overload: `<Options>`,
`<Options, Frame, State>`, `<Options, Token>`) and composes through `MediaRoute.inline` / `stream` / `queued`; and
`TranscriptionClient` dispatches on `route.kind`. `generate` on a queued route is `start` then `await`; `stream` on an
inline route is the response as a single `finish`, and on a queued route it is the status observations followed by
`finish`. `start` / `resume` on a non-queued route fail with `UnsupportedOperation` (`transcription.start`). The
`finish` event carries the whole transcript (text, segments, words, language, duration, usage), so the stream route's
`collect` is just "take `finish`".
The route layer gained a `binary` body with array-valued `query` (Deepgram) and `Queued.start.prepare` (AssemblyAI's
upload); `packages/ai/AGENTS.md` (Media Routes) describes both.
Settled rules:
- **Timestamps.** A granularity the selected route or model cannot produce fails as `UnsupportedOperation`
(`media.timestamps`), following Speech; a route that returns more than asked (Deepgram and AssemblyAI always return
words) is not stripped. Segments always carry start and end times: Gemini times each transcription part from its
word offsets, so segment timestamps and diarization also request word offsets there.
- **Diarization.** `diarize` means segments (and words, where the provider labels them) carry `speaker`. Labels are
provider-native strings — OpenAI `A` or a known speaker name, Deepgram `0`, Gemini `spk:0`, AssemblyAI `A` — with no
cross-provider speaker model. `speakers` is a hint; only AssemblyAI (`speakers_expected`) accepts it.
- **Language** is passed through (`language`, OpenAI `gpt-transcribe` `languages[]`, Gemini `languageCodes`,
AssemblyAI `language_code`). `response.language` is the provider's own value, lowercased but not normalized: an
ISO code on most routes, `english` from whisper-1, `en_us` from AssemblyAI. Deepgram and AssemblyAI assume English
unless asked to detect, so a missing `language` enables their detection.
- **Gemini** requires a transcribe model; other model ids fail with `UnsupportedOperation` before the call, because
general models ignore `audioTranscriptionConfig` and answer conversationally. Streamed chunks carry whole speaker
turns (one part per turn), which join with a space.
- **Streaming inline providers** emit only `finish`; deltas are never faked.
- **Units.** AssemblyAI milliseconds and Gemini protobuf durations (`"0.400s"`) are normalized to seconds at the
protocol boundary.
| Provider | Kind | Audio input | `timestamps` | `diarize` | Unsupported | Usage |
|---|---|---|---|---|---|---|
| OpenAI | stream (`stream: true` in `stream` mode) | multipart `file` (inline only) | `whisper-1` (`verbose_json`); diarize model: `segment` | `gpt-4o-transcribe-diarize` (`diarized_json`) | `speakers`; `prompt` on the diarize model; streaming on `whisper-1` | `tokens` or `seconds` |
| Gemini | stream (`generateContent` / `streamGenerateContent`) | `inlineData` or Gemini Files `fileData` | `audioTranscriptionConfig.wordTimestamp` | `audioTranscriptionConfig.diarization` | `prompt`, `speakers` | `tokens` |
| Deepgram | inline | raw body, or JSON `{ url }` | words always; `segment` → `utterances` | `diarize_model=latest` + `utterances` | `prompt`, `speakers` | `seconds` (`metadata.duration`) |
| AssemblyAI | queued (upload → submit → poll) | `/v2/upload` then `audio_url`, or a URL | words always; `segment` → `speaker_labels` | `speaker_labels` | — | `seconds` (`audio_duration`) |
Deferred: `Transcription.session(...)` — realtime STT over WebSocket (Deepgram live, AssemblyAI streaming, ElevenLabs
realtime, OpenAI realtime transcription) — is the same future scoped `session` shape as input-streaming TTS and ships
with the realtime work in phase 5. ElevenLabs Scribe is not implemented yet.
### `Generation` — shared async execution
@@ -338,13 +386,14 @@ Existing facades gain per-modality selectors; the modality routes each facade pr
| Facade | llm | image | video | speech | transcription | other |
|---|---|---|---|---|---|---|
| `OpenAI` | responses (default), chat | Images API | Sora (deprecated 2026-09-24) | ✓ | ✓ | |
| `Google` | Gemini | Gemini-native (default), `imagen` | Veo | Gemini TTS | Gemini transcribe | |
| `Google` | Gemini | Gemini-native (default), `imagen` | Veo | Gemini TTS | `gemini-3.5-transcribe` | |
| `XAI` | ✓ | ✓ | ✓ | | | |
| `ElevenLabs` | | | | ✓ | Scribe | soundEffect, music |
| `Cartesia` | | | | ✓ | | |
| `Deepgram` | | | | Aura | ✓ | |
| `Fal` | | ✓ | ✓ | | | |
| `Replicate`, `Runway`, `Luma`, `Kling`, `MiniMax`, `AssemblyAI`, `BlackForestLabs`, `Stability` | | per provider | | | | |
| `AssemblyAI` | | | | | ✓ (queued) | |
| `Replicate`, `Runway`, `Luma`, `Kling`, `MiniMax`, `BlackForestLabs`, `Stability` | | per provider | | | | |
New facades follow the existing one-file-per-provider rule. Package entrypoints are modality-specific, such as `@opencode/ai/providers/openai/images`, and return the concrete model.
@@ -358,7 +407,7 @@ Media does not fit the LLM four-axis route (SSE frames → event state machine)
- `MediaProtocol.queued` — `start` (body + decode to `{ token, snapshot }`), `status`, `result`, optional `cancel`, `pollHint`, and a `token` codec. `result` is always a separate GET (against the status document for Veo/xAI/Runway, fal's `response_url` otherwise) so `await` after `start` and after `resume` share one path. `PollContext.auth` hands the auth headers the route sent to the protocol for output URLs that need them (Veo downloads); they become transient `Media.Asset.headers`, never part of `source`. There is no separate `download` step: `Media.Asset.bytes()` downloads through the executor with those headers. `MediaRoute.inline(...)` / `MediaRoute.queued(...)` compose each kind with endpoint and auth; the queued route decodes the token once and hands `Generation` a token-free `{ status, result, cancel? }`.
- `MediaProtocol.stream` — `body.from(request)` over the request plus its `mode`, `frames` (a function that picks the framing for the call: `Framing.sse`, `lines`, `document`, or the raw bytes), fresh per-response `initial()` state, `step` emitting modality events, and `finish(state, context)` — with the observed response for header-only usage — emitting exactly one terminal event or failing as an incomplete stream. The route fills `reason.http` on stream errors. `MediaRoute.stream(...)` exposes `stream` and `generate` (the same stream folded by the modality's `collect`).
`MediaRoute.inline` / `MediaRoute.queued` / `MediaRoute.stream` compose one protocol kind with endpoint/auth; `ImageModel`/`VideoModel`/`SpeechModel` share the `MediaModel` base (`src/media-model.ts`).
`MediaRoute.inline` / `MediaRoute.queued` / `MediaRoute.stream` compose one protocol kind with endpoint/auth and tag the route with its `kind`; `ImageModel`/`VideoModel`/`SpeechModel`/`TranscriptionModel` share the `MediaModel` base (`src/media-model.ts`).
### LLM integration
@@ -388,7 +437,7 @@ Foundation + Image ship together as the reference implementation, serially. Vide
1. **Foundation** — per-modality selectors, `Media`, `Generation`, `Poll`, `Usage` union, `MediaProtocol` kinds, `@opencode/ai/promise` with `llm` + `image`. Port the five existing image protocols onto it. Unify `MediaPart` and add the `media` LLM event (fixes Gemini image output being dropped).
2. **Video** — ✅ Veo, xAI, fal, Runway shipped (`MediaProtocol.queued`, `Video.start/generate/resume/stream`, promise `ai.video`). Deferred: `Video.complete` (webhooks), Luma, Kling, MiniMax, Replicate.
3. **Speech + Transcription** — ✅ Speech: OpenAI, Gemini TTS, ElevenLabs, Cartesia, Deepgram shipped (`MediaProtocol.stream`, `Speech.generate/stream`, promise `ai.speech`). Pending: Transcription (OpenAI, ElevenLabs Scribe, Gemini, Deepgram, AssemblyAI). Deferred: `Speech.session` (WebSocket input streaming).
3. **Speech + Transcription** — ✅ Speech: OpenAI, Gemini TTS, ElevenLabs, Cartesia, Deepgram shipped (`MediaProtocol.stream`, `Speech.generate/stream`, promise `ai.speech`). ✅ Transcription: OpenAI, Gemini, Deepgram, AssemblyAI shipped across all three route kinds (`Transcription.generate/stream/start/resume`, promise `ai.transcription`). Pending: ElevenLabs Scribe. Deferred: `Speech.session` and `Transcription.session` (WebSocket streaming).
4. **Image queued routes and partials** — BFL, fal, Replicate, Stability; OpenAI `partial_images` streaming.
5. **Later** — ElevenLabs music/SFX, Lyria, `Speech.session` / `Transcription.session`, realtime.
+13 -1
View File
@@ -161,7 +161,7 @@ const PROVIDERS: ReadonlyArray<Provider> = [
id: "deepgram",
label: "Deepgram",
tier: "canary",
note: "Deepgram Aura text-to-speech recorded tests",
note: "Deepgram Aura text-to-speech and Nova transcription recorded tests",
vars: [{ name: "DEEPGRAM_API_KEY" }],
validate: (env) =>
HttpClientRequest.get("https://api.deepgram.com/v1/projects").pipe(
@@ -169,6 +169,18 @@ const PROVIDERS: ReadonlyArray<Provider> = [
executeRequest,
),
},
{
id: "assemblyai",
label: "AssemblyAI",
tier: "canary",
note: "AssemblyAI queued transcription recorded tests",
vars: [{ name: "ASSEMBLYAI_API_KEY" }],
validate: (env) =>
HttpClientRequest.get("https://api.assemblyai.com/v2/transcript?limit=1").pipe(
HttpClientRequest.setHeader("authorization", Redacted.value(Redacted.make(env.ASSEMBLYAI_API_KEY))),
executeRequest,
),
},
{
id: "cloudflare-ai-gateway",
label: "Cloudflare AI Gateway",
+25 -4
View File
@@ -40,10 +40,21 @@ export interface AwaitOptions {
export const DEFAULT_POLL_INTERVAL = Duration.seconds(5)
export const DEFAULT_POLL_TIMEOUT = Duration.minutes(10)
export type Event =
| { readonly type: "generation-queued"; readonly id: string; readonly position?: number }
| { readonly type: "generation-progress"; readonly id: string; readonly progress?: number }
| { readonly type: "generation-finished"; readonly id: string; readonly status: Status }
export const QueuedEvent = Schema.Struct({
type: Schema.tag("generation-queued"),
id: Schema.String,
position: Schema.optional(Schema.Number),
}).annotate({ identifier: "Generation.Event.Queued" })
export const ProgressEvent = Schema.Struct({
type: Schema.tag("generation-progress"),
id: Schema.String,
progress: Schema.optional(Schema.Number),
}).annotate({ identifier: "Generation.Event.Progress" })
export type Observation = Schema.Schema.Type<typeof QueuedEvent> | Schema.Schema.Type<typeof ProgressEvent>
export type Event = Observation | { readonly type: "generation-finished"; readonly id: string; readonly status: Status }
const TERMINAL: ReadonlySet<Status> = new Set(["completed", "failed", "cancelled", "expired"])
@@ -168,3 +179,13 @@ export class Generation<Response> {
)
}
}
export const resultEvents = <Response, A>(
generation: Generation<Response>,
expand: (response: Response) => ReadonlyArray<A>,
options?: AwaitOptions,
): Stream.Stream<Observation | A, AIError> =>
generation.events(options).pipe(
Stream.filter((event): event is Observation => event.type !== "generation-finished"),
Stream.concat(Stream.fromIterableEffect(Effect.map(generation.result(), expand))),
)
+19
View File
@@ -67,6 +67,25 @@ export type {
SpeechRoute,
} from "./speech.js"
export { Speech } from "./speech.js"
export { TranscriptionClient } from "./transcription-client.js"
export {
TranscriptionEvent,
TranscriptionModel,
TranscriptionModelSchema,
TranscriptionRequest,
TranscriptionResponse,
TranscriptionSegment,
TranscriptionTimestamps,
TranscriptionWord,
} from "./transcription.js"
export type {
TranscriptionModelOptions,
TranscriptionOptions,
TranscriptionRequestFor,
TranscriptionRequestInput,
TranscriptionRoute,
} from "./transcription.js"
export { Transcription } from "./transcription.js"
export { Media } from "./media.js"
export { Generation } from "./generation.js"
export type {
+41 -5
View File
@@ -9,12 +9,21 @@ import { LanguageModel, LLMRequest } from "./schema/index.js"
import type { RequestInput } from "./llm.js"
import { Speech, SpeechModel, SpeechRequest, type SpeechRequestInput } from "./speech.js"
import { SpeechClient } from "./speech-client.js"
import {
Transcription,
TranscriptionModel,
TranscriptionRequest,
type TranscriptionOptions,
type TranscriptionRequestInput,
} from "./transcription.js"
import { TranscriptionClient } from "./transcription-client.js"
import { Video, VideoModel, VideoRequest, type VideoOptions, type VideoRequestInput } from "./video.js"
import { VideoClient } from "./video-client.js"
/**
* Promise-first entrypoint for scripts and non-Effect callers. One `ManagedRuntime` hosts the LLM, image, video, and
* speech clients over a request executor; every method runs the corresponding Effect API and rethrows `AIError` unchanged.
* Promise-first entrypoint for scripts and non-Effect callers. One `ManagedRuntime` hosts the LLM, image, video, speech,
* and transcription clients over a request executor; every method runs the corresponding Effect API and rethrows
* `AIError` unchanged.
*/
export interface Options {
/** Executor layer; defaults to `RequestExecutor.fetchLayer`. Inject a recorder or middleware here. */
@@ -30,6 +39,7 @@ export type Services =
| Layer.Success<typeof ImageClient.layer>
| Layer.Success<typeof VideoClient.layer>
| Layer.Success<typeof SpeechClient.layer>
| Layer.Success<typeof TranscriptionClient.layer>
| RequestExecutor.Service
/** Promise view of a `Generation`: its snapshot plus `await`, `refresh`, and `cancel` returning promises. */
@@ -56,9 +66,13 @@ const abortEffect = (signal: AbortSignal | undefined) =>
export const make = (options: Options = {}) => {
const runtime = ManagedRuntime.make(
Layer.mergeAll(LLMClient.layer, ImageClient.layer, VideoClient.layer, SpeechClient.layer).pipe(
Layer.provideMerge(options.layer ?? RequestExecutor.fetchLayer),
),
Layer.mergeAll(
LLMClient.layer,
ImageClient.layer,
VideoClient.layer,
SpeechClient.layer,
TranscriptionClient.layer,
).pipe(Layer.provideMerge(options.layer ?? RequestExecutor.fetchLayer)),
)
/** Run any package Effect (for example `asset.bytes()`) inside this runtime. */
@@ -93,6 +107,8 @@ export const make = (options: Options = {}) => {
input instanceof VideoRequest ? input : Video.request(input)
const speechRequest = (input: SpeechRequestInput | SpeechRequest) =>
input instanceof SpeechRequest ? input : Speech.request(input)
const transcriptionRequest = (input: TranscriptionRequestInput | TranscriptionRequest) =>
input instanceof TranscriptionRequest ? input : Transcription.request(input)
return {
run,
@@ -138,6 +154,26 @@ export const make = (options: Options = {}) => {
options?: RunOptions,
) => iterate(Speech.stream(speechRequest(input)), options),
},
transcription: {
request: Transcription.request,
generate: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: AwaitOptions & RunOptions,
) => run(Transcription.generate(transcriptionRequest(input), { poll: options?.poll }), options),
stream: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: AwaitOptions & RunOptions,
) => iterate(Transcription.stream(transcriptionRequest(input), { poll: options?.poll }), options),
start: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: RunOptions,
) => run(Transcription.start(transcriptionRequest(input)), options).then(handle),
resume: <Options extends TranscriptionOptions>(
model: TranscriptionModel<Options>,
token: unknown,
options?: RunOptions,
) => run(Transcription.resume(model, token), options).then(handle),
},
dispose: () => runtime.dispose(),
}
}
@@ -0,0 +1,211 @@
import { Effect, Schema } from "effect"
import type { HttpClientResponse } from "effect/unstable/http"
import type { Status } from "../generation.js"
import { Media } from "../media.js"
import { MediaProtocol } from "../route/media-protocol.js"
import { MediaRoute } from "../route/media.js"
import { ProviderID, mergeJsonRecords } from "../schema/index.js"
import { TranscriptionModel, TranscriptionResponse, type TranscriptionRequestFor } from "../transcription.js"
import { ProviderShared, optionalNull } from "./shared.js"
import { MediaInput } from "./utils/media-input.js"
const ADAPTER = "assemblyai-transcription"
const NAME = "AssemblyAI"
const PROVIDER = ProviderID.make("assemblyai")
export const DEFAULT_BASE_URL = "https://api.assemblyai.com"
export const PATH = "/v2/transcript"
export const UPLOAD_PATH = "/v2/upload"
// ---------------------------------------------------------------------------
// 1. Public model input
// ---------------------------------------------------------------------------
export type AssemblyAITranscriptionOptions = {
readonly keyterms_prompt?: ReadonlyArray<string>
readonly punctuate?: boolean
readonly format_text?: boolean
readonly disfluencies?: boolean
readonly filter_profanity?: boolean
readonly temperature?: number
readonly speaker_options?: { readonly min_speakers_expected?: number; readonly max_speakers_expected?: number }
readonly language_detection_options?: {
readonly expected_languages?: ReadonlyArray<string>
readonly fallback_language?: string
readonly code_switching?: boolean
}
readonly speech_models?: ReadonlyArray<"universal-3-5-pro" | "universal-2" | (string & {})>
} & Record<string, unknown>
export type Request = TranscriptionRequestFor<AssemblyAITranscriptionOptions>
// ---------------------------------------------------------------------------
// 2. Token and response schemas
// ---------------------------------------------------------------------------
export const Token = Schema.Struct({ transcriptID: Schema.String })
export type Token = Schema.Schema.Type<typeof Token>
const Upload = Schema.Struct({ upload_url: Schema.String })
/** Word and utterance times are milliseconds. */
const Transcript = Schema.Struct({
id: Schema.String,
status: Schema.String,
text: optionalNull(Schema.String),
words: optionalNull(
Schema.Array(
Schema.Struct({
text: Schema.String,
start: Schema.Number,
end: Schema.Number,
confidence: optionalNull(Schema.Number),
speaker: optionalNull(Schema.String),
}),
),
),
utterances: optionalNull(
Schema.Array(
Schema.Struct({
text: Schema.String,
start: Schema.Number,
end: Schema.Number,
speaker: optionalNull(Schema.String),
}),
),
),
language_code: optionalNull(Schema.String),
audio_duration: optionalNull(Schema.Number),
speech_model_used: optionalNull(Schema.String),
error: optionalNull(Schema.String),
})
const STATUS = {
queued: "queued",
processing: "running",
completed: "completed",
error: "failed",
} as const satisfies Record<string, Status>
// ---------------------------------------------------------------------------
// 5. Request body construction
// ---------------------------------------------------------------------------
const decodeUpload = MediaProtocol.decodeJson(ADAPTER, NAME, Upload)
/** `/v2/transcript` only takes a URL, so inline audio is uploaded to `/v2/upload` first. */
const prepare = Effect.fn("AssemblyAITranscription.prepare")(function* (request: Request, send: MediaProtocol.Send) {
if (request.audio.source.type !== "bytes" && request.audio.source.type !== "base64") return request
const audio = yield* MediaInput.inlineBytes(ADAPTER, request.audio)
const uploaded = yield* send(UPLOAD_PATH, MediaProtocol.binary(audio, "application/octet-stream")).pipe(
Effect.flatMap(decodeUpload),
)
return { ...request, audio: Media.url(uploaded.value.upload_url, { mediaType: request.audio.mediaType }) }
})
const fromRequest = Effect.fn("AssemblyAITranscription.fromRequest")(function* (request: Request) {
const audio = yield* ProviderShared.mediaReference(request.audio, PROVIDER, NAME)
return MediaProtocol.json(
mergeJsonRecords(
{
audio_url: audio.value,
speech_models: [request.model.id],
language_code: request.language,
language_detection: request.language === undefined ? true : undefined,
prompt: request.prompt,
// Turn-level `utterances`, the only segments AssemblyAI returns, require speaker labels.
speaker_labels: request.diarize === true || request.timestamps === "segment" ? true : undefined,
speakers_expected: request.speakers,
},
request.providerOptions,
request.http?.body,
) ?? {},
)
})
// ---------------------------------------------------------------------------
// 6. Response decoding
// ---------------------------------------------------------------------------
const decodeTranscript = MediaProtocol.decodeJson(ADAPTER, NAME, Transcript)
const decodeStart = Effect.fn("AssemblyAITranscription.decodeStart")(function* (
response: HttpClientResponse.HttpClientResponse,
) {
const output = yield* decodeTranscript(response)
const status = yield* MediaProtocol.status(STATUS, output.value.status, output)
return { token: { transcriptID: output.value.id }, snapshot: { id: output.value.id, status } }
})
const decodeStatus = Effect.fn("AssemblyAITranscription.decodeStatus")(function* (
response: HttpClientResponse.HttpClientResponse,
context: MediaProtocol.PollContext<Token>,
) {
const output = yield* decodeTranscript(response)
const status = yield* MediaProtocol.status(STATUS, output.value.status, output)
return { id: context.token.transcriptID, status }
})
const seconds = (milliseconds: number) => milliseconds / 1000
const decodeResult = Effect.fn("AssemblyAITranscription.decodeResult")(function* (
response: HttpClientResponse.HttpClientResponse,
context: MediaProtocol.PollContext<Token>,
) {
const output = yield* decodeTranscript(response)
const transcript = output.value
const status = yield* MediaProtocol.status(STATUS, transcript.status, output)
const error = transcript.error ?? undefined
if (status === "failed")
return yield* output.ended("failed", `${NAME} transcription failed${error === undefined ? "" : `: ${error}`}`)
if (status !== "completed")
return yield* output.invalid(`${NAME} transcript ${context.token.transcriptID} has not finished`)
const duration = transcript.audio_duration ?? undefined
return new TranscriptionResponse({
text: transcript.text ?? "",
segments: transcript.utterances?.map((utterance) => ({
text: utterance.text,
startSeconds: seconds(utterance.start),
endSeconds: seconds(utterance.end),
speaker: utterance.speaker ?? undefined,
})),
words: transcript.words?.map((word) => ({
text: word.text,
startSeconds: seconds(word.start),
endSeconds: seconds(word.end),
speaker: word.speaker ?? undefined,
confidence: word.confidence ?? undefined,
})),
language: transcript.language_code?.toLowerCase(),
durationSeconds: duration,
usage: duration === undefined ? undefined : { type: "seconds", seconds: duration },
providerMetadata: {
assemblyai: { transcriptId: transcript.id, speechModel: transcript.speech_model_used ?? undefined },
},
})
})
// ---------------------------------------------------------------------------
// 7. Protocol and route
// ---------------------------------------------------------------------------
const transcriptPath = (token: Token) => `${PATH}/${token.transcriptID}`
export const protocol = MediaProtocol.queued<Request, TranscriptionResponse, Token>({
id: ADAPTER,
name: NAME,
token: Token,
start: { prepare, body: { from: fromRequest }, decode: decodeStart },
status: { path: transcriptPath, decode: decodeStatus },
result: { path: transcriptPath, decode: decodeResult },
})
export const model = (input: MediaRoute.ModelInput) =>
TranscriptionModel.fromRoute<AssemblyAITranscriptionOptions, Token>(
{ id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: PATH },
input,
)
export const AssemblyAITranscription = {
protocol,
model,
} as const
+22 -23
View File
@@ -3,6 +3,7 @@ import { MediaProtocol } from "../route/media-protocol.js"
import { MediaRoute } from "../route/media.js"
import { ProviderID, mergeJsonRecords } from "../schema/index.js"
import { SpeechModel, type SpeechEvent, type SpeechRequestFor } from "../speech.js"
import { MediaInput } from "./utils/media-input.js"
import { SpeechStream } from "./utils/speech-stream.js"
const ADAPTER = "deepgram-speech"
@@ -49,26 +50,24 @@ const FORMATS: Readonly<Record<string, { readonly encoding: string; readonly con
aac: { encoding: "aac" },
}
const queryValue = (value: unknown) =>
typeof value === "string" || typeof value === "number" || typeof value === "boolean" ? String(value) : undefined
const audioFormat = (request: Request) => {
const format = request.format === undefined ? undefined : FORMATS[request.format]
return {
encoding: request.providerOptions?.encoding ?? format?.encoding,
container: request.providerOptions?.container ?? format?.container,
}
}
const queryParameters = (request: Request) => {
const format = request.format === undefined ? undefined : FORMATS[request.format]
const { encoding, container, sampleRate, bitRate, ...native } = request.providerOptions ?? {}
return Object.fromEntries(
Object.entries({
...native,
model: request.model.id,
encoding: encoding ?? format?.encoding,
container: container ?? format?.container,
sample_rate: sampleRate,
bit_rate: bitRate,
speed: request.speed,
}).flatMap(([key, value]) => {
const text = queryValue(value)
return text === undefined ? [] : [[key, text]]
}),
)
const { encoding: _encoding, container: _container, sampleRate, bitRate, ...native } = request.providerOptions ?? {}
return MediaInput.query(ADAPTER, {
...native,
model: request.model.id,
...audioFormat(request),
sample_rate: sampleRate,
bit_rate: bitRate,
speed: request.speed,
})
}
const fromRequest = Effect.fn("DeepgramSpeech.fromRequest")(function* (request: Request) {
@@ -84,7 +83,7 @@ const fromRequest = Effect.fn("DeepgramSpeech.fromRequest")(function* (request:
)
return MediaProtocol.json(
mergeJsonRecords({ text: request.text }, request.http?.body) ?? {},
queryParameters(request),
yield* queryParameters(request),
)
})
@@ -101,15 +100,15 @@ const HEADERLESS_ENCODINGS: Readonly<Record<string, SpeechStream.PcmEncoding>> =
const finish = (state: State, context: MediaProtocol.ResponseContext<Request>) => {
const headers = context.http.headers
const mediaType = headers["content-type"]
const query = queryParameters(context.request)
const encoding = HEADERLESS_ENCODINGS[query.encoding ?? ""]
const format = audioFormat(context.request)
const encoding = HEADERLESS_ENCODINGS[format.encoding ?? ""]
const requestID = headers["dg-request-id"]
const modelName = headers["dg-model-name"]
return SpeechStream.finish(ADAPTER, state, {
...(query.container === "none" && encoding !== undefined
...(format.container === "none" && encoding !== undefined
? SpeechStream.pcm(encoding, SpeechStream.sampleRate(mediaType), mediaType)
: // Deepgram's default encoding is MP3; WAV is a container around any encoding.
{ mediaType, info: { format: query.container === "wav" ? "wav" : (query.encoding ?? "mp3") } }),
{ mediaType, info: { format: format.container === "wav" ? "wav" : (format.encoding ?? "mp3") } }),
usage: SpeechStream.headerUsage("characters", headers["dg-char-count"]),
providerMetadata:
requestID === undefined && modelName === undefined
@@ -0,0 +1,191 @@
import { Effect, Schema } from "effect"
import type { HttpClientResponse } from "effect/unstable/http"
import { MediaProtocol } from "../route/media-protocol.js"
import { MediaRoute } from "../route/media.js"
import { ProviderID, mergeJsonRecords } from "../schema/index.js"
import { TranscriptionModel, TranscriptionResponse, type TranscriptionRequestFor } from "../transcription.js"
import { ProviderShared } from "./shared.js"
import { MediaInput } from "./utils/media-input.js"
const ADAPTER = "deepgram-transcription"
const NAME = "Deepgram"
const PROVIDER = ProviderID.make("deepgram")
export const DEFAULT_BASE_URL = "https://api.deepgram.com"
export const PATH = "/v1/listen"
// ---------------------------------------------------------------------------
// 1. Public model input
// ---------------------------------------------------------------------------
export type DeepgramTranscriptionOptions = {
readonly smart_format?: boolean
readonly punctuate?: boolean
readonly paragraphs?: boolean
readonly utterances?: boolean
readonly detect_language?: boolean | ReadonlyArray<string>
readonly keyterm?: ReadonlyArray<string>
readonly diarize_model?: "latest" | "v1" | "v2" | (string & {})
readonly filler_words?: boolean
readonly numerals?: boolean
readonly mip_opt_out?: boolean
readonly tag?: string | ReadonlyArray<string>
} & Record<string, unknown>
export type Request = TranscriptionRequestFor<DeepgramTranscriptionOptions>
// ---------------------------------------------------------------------------
// 2. Response schema
// ---------------------------------------------------------------------------
const Word = Schema.Struct({
word: Schema.String,
start: Schema.Number,
end: Schema.Number,
confidence: Schema.optional(Schema.Number),
speaker: Schema.optional(Schema.Number),
punctuated_word: Schema.optional(Schema.String),
})
const ListenResponse = Schema.Struct({
metadata: Schema.optional(
Schema.Struct({ request_id: Schema.optional(Schema.String), duration: Schema.optional(Schema.Number) }),
),
results: Schema.Struct({
channels: Schema.Array(
Schema.Struct({
alternatives: Schema.optional(
Schema.Array(Schema.Struct({ transcript: Schema.String, words: Schema.optional(Schema.Array(Word)) })),
),
detected_language: Schema.optional(Schema.String),
}),
),
utterances: Schema.optional(
Schema.Array(
Schema.Struct({
start: Schema.Number,
end: Schema.Number,
transcript: Schema.String,
speaker: Schema.optional(Schema.Number),
words: Schema.optional(Schema.Array(Word)),
}),
),
),
}),
})
// ---------------------------------------------------------------------------
// 5. Request body construction
// ---------------------------------------------------------------------------
const query = (request: Request) =>
MediaInput.query(
ADAPTER,
mergeJsonRecords(
{
model: request.model.id,
smart_format: true,
language: request.language,
// Deepgram assumes English unless asked to detect, unlike the other routes' auto-detection.
detect_language: request.language === undefined ? true : undefined,
// `diarize=true` is deprecated in favor of choosing a diarization model.
diarize_model: request.diarize === true ? "latest" : undefined,
utterances: request.diarize === true || request.timestamps === "segment" ? true : undefined,
},
request.providerOptions,
) ?? {},
)
const fromRequest = Effect.fn("DeepgramTranscription.fromRequest")(function* (request: Request) {
const url = ProviderShared.mediaUrl(request.audio)
if (url !== undefined)
return MediaProtocol.json(mergeJsonRecords({ url }, request.http?.body) ?? {}, yield* query(request))
if (request.http?.body !== undefined)
return yield* ProviderShared.invalidRequest(`${NAME} sends inline audio as the raw body, so http.body cannot apply`)
const audio = yield* MediaInput.inlineBytes(ADAPTER, request.audio)
return MediaProtocol.binary(audio, request.audio.mediaType, yield* query(request))
})
// ---------------------------------------------------------------------------
// 6. Response decoding
// ---------------------------------------------------------------------------
const decodeListen = MediaProtocol.decodeJson(ADAPTER, NAME, ListenResponse)
const speaker = (value: number | undefined) => (value === undefined ? undefined : String(value))
const wordText = (word: typeof Word.Type) => word.punctuated_word ?? word.word
// Utterances split on pauses, not speakers: the v2 diarizer labels a whole utterance with one speaker even when its
// words change speaker, so segments split each utterance at speaker changes.
const speakerTurns = (words: ReadonlyArray<typeof Word.Type>) =>
words.reduce<Array<Array<typeof Word.Type>>>((turns, word) => {
const last = turns.at(-1)
if (last === undefined || last[0].speaker !== word.speaker) return [...turns, [word]]
last.push(word)
return turns
}, [])
const decodeResponse = Effect.fn("DeepgramTranscription.decodeResponse")(function* (
response: HttpClientResponse.HttpClientResponse,
) {
const output = yield* decodeListen(response)
const channel = output.value.results.channels[0]
const alternative = channel?.alternatives?.[0]
if (alternative === undefined) return yield* output.invalid(`${NAME} returned no transcript`)
const duration = output.value.metadata?.duration
const requestID = output.value.metadata?.request_id
return new TranscriptionResponse({
text: alternative.transcript,
segments: output.value.results.utterances?.flatMap((utterance) =>
utterance.words === undefined || utterance.words.length === 0
? [
{
text: utterance.transcript,
startSeconds: utterance.start,
endSeconds: utterance.end,
speaker: speaker(utterance.speaker),
},
]
: speakerTurns(utterance.words).map((turn) => ({
text: turn.map(wordText).join(" "),
startSeconds: turn[0].start,
endSeconds: turn[turn.length - 1].end,
speaker: speaker(turn[0].speaker),
})),
),
words: alternative.words?.map((word) => ({
text: wordText(word),
startSeconds: word.start,
endSeconds: word.end,
speaker: speaker(word.speaker),
confidence: word.confidence,
})),
language: channel?.detected_language?.toLowerCase(),
durationSeconds: duration,
usage: duration === undefined ? undefined : { type: "seconds", seconds: duration },
providerMetadata: requestID === undefined ? undefined : { deepgram: { requestId: requestID } },
})
})
// ---------------------------------------------------------------------------
// 7. Protocol and route
// ---------------------------------------------------------------------------
export const protocol = MediaProtocol.inline<Request, TranscriptionResponse>({
id: ADAPTER,
name: NAME,
unsupported: ["prompt", "speakers"],
body: { from: fromRequest },
response: { decode: decodeResponse },
})
export const model = (input: MediaRoute.ModelInput) =>
TranscriptionModel.fromRoute<DeepgramTranscriptionOptions>(
{ id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: PATH },
input,
)
export const DeepgramTranscription = {
protocol,
model,
} as const
+2 -6
View File
@@ -22,6 +22,7 @@ import {
import { classifyProviderFailure } from "../provider-error.js"
import { Media } from "../media.js"
import { JsonObject, knownString, lenient, optionalArray, optionalNull, ProviderShared } from "./shared.js"
import { GeminiGenerateContent } from "./utils/gemini-generate-content.js"
import { GeminiToolSchema } from "./utils/gemini-tool-schema.js"
import { Lifecycle } from "./utils/lifecycle.js"
import { ToolSchemaProjection } from "./utils/tool-schema.js"
@@ -305,14 +306,9 @@ const lowerToolConfig = (toolChoice: NonNullable<LLMRequest["toolChoice"]>) =>
tool: (name) => ({ functionCallingConfig: { mode: "ANY" as const, allowedFunctionNames: [name] } }),
})
// Gemini does not fetch public URLs; inline payloads and Gemini Files references are the accepted inputs.
const lowerContentPart = Effect.fn("Gemini.lowerContentPart")(function* (part: TextPart | MediaPart) {
if (part.type === "text") return { text: part.text }
const source = part.media.source
if (source.type === "ref" && source.provider === "google")
return { fileData: { mimeType: part.media.mediaType, fileUri: source.id } }
const media = yield* ProviderShared.requireInlineMedia("Gemini", part.media)
return { inlineData: { mimeType: media.mime, data: media.base64 } }
return yield* GeminiGenerateContent.mediaPart("Gemini", part.media)
})
const providerMetadata = (key: string, metadata: Record<string, unknown>): ProviderMetadata => ({ [key]: metadata })
+3 -22
View File
@@ -1,11 +1,11 @@
import { Effect, Schema } from "effect"
import type { HttpClientResponse } from "effect/unstable/http"
import { ImageModel, ImageResponse, type ImageRequestFor } from "../image.js"
import { Media } from "../media.js"
import { MediaProtocol } from "../route/media-protocol.js"
import { MediaRoute } from "../route/media.js"
import { ProviderID, mergeJsonRecords, type AIError } from "../schema/index.js"
import { ProviderID, mergeJsonRecords } from "../schema/index.js"
import { ProviderShared } from "./shared.js"
import { GeminiGenerateContent } from "./utils/gemini-generate-content.js"
import { MediaInput } from "./utils/media-input.js"
const ADAPTER = "google-images"
@@ -102,25 +102,6 @@ const generationConfig = (request: Request) => {
)
}
// Gemini does not fetch public URLs; inline payloads or Gemini Files references are the only accepted inputs.
const imagePart = (asset: Media.Asset): Effect.Effect<Record<string, unknown>, AIError> => {
const inline = asset.inline()
if (inline) return Effect.succeed({ inlineData: { mimeType: inline.mime, data: inline.base64 } })
const id = MediaInput.refID(asset, PROVIDER)
if (id) return Effect.succeed({ fileData: { mimeType: asset.mediaType, fileUri: id } })
if (asset.source.type === "ref")
return Effect.fail(
ProviderShared.invalidRequest(
"Google generateContent requires Gemini file references rather than other providers' file IDs",
),
)
return Effect.fail(
ProviderShared.invalidRequest(
"Google generateContent does not fetch public image URLs; use bytes, a data URL, or a Gemini file reference",
),
)
}
const fromRequest = Effect.fn("GoogleImages.fromRequest")(function* (request: Request) {
if (request.n !== undefined && request.n > 1)
return yield* ProviderShared.unsupportedOperation({
@@ -129,7 +110,7 @@ const fromRequest = Effect.fn("GoogleImages.fromRequest")(function* (request: Re
route: ADAPTER,
message: `${NAME} generates one image per request; call it once per image instead of n=${request.n}`,
})
const parts = yield* Effect.forEach(request.images ?? [], imagePart)
const parts = yield* Effect.forEach(request.images ?? [], (image) => GeminiGenerateContent.mediaPart(NAME, image))
return MediaProtocol.json(
mergeJsonRecords(
{
+24 -92
View File
@@ -1,10 +1,9 @@
import { Effect, Schema } from "effect"
import { Framing } from "../route/framing.js"
import { MediaProtocol } from "../route/media-protocol.js"
import { MediaRoute } from "../route/media.js"
import { AIError, ContentPolicyError, ProviderID, mergeJsonRecords } from "../schema/index.js"
import { ProviderID, mergeJsonRecords } from "../schema/index.js"
import { SpeechModel, type SpeechEvent, type SpeechRequestFor } from "../speech.js"
import { ProviderShared } from "./shared.js"
import { GeminiGenerateContent } from "./utils/gemini-generate-content.js"
import { SpeechStream } from "./utils/speech-stream.js"
const ADAPTER = "google-speech"
@@ -37,42 +36,12 @@ export type Request = SpeechRequestFor<GoogleSpeechOptions>
// 3. Streaming event schema
// ---------------------------------------------------------------------------
const GenerateContentChunk = Schema.Struct({
candidates: Schema.optional(
Schema.Array(
Schema.Struct({
content: Schema.optional(
Schema.Struct({
parts: Schema.optional(
Schema.Array(
Schema.Struct({
text: Schema.optional(Schema.String),
inlineData: Schema.optional(
Schema.Struct({ mimeType: Schema.String, data: Schema.Uint8ArrayFromBase64 }),
),
}),
),
),
}),
),
finishReason: Schema.optional(Schema.String),
}),
),
),
promptFeedback: Schema.optional(
Schema.Struct({ blockReason: Schema.optional(Schema.String), blockReasonMessage: Schema.optional(Schema.String) }),
),
usageMetadata: Schema.optional(
Schema.Struct({
promptTokenCount: Schema.optional(Schema.Number),
candidatesTokenCount: Schema.optional(Schema.Number),
totalTokenCount: Schema.optional(Schema.Number),
}),
),
modelVersion: Schema.optional(Schema.String),
responseId: Schema.optional(Schema.String),
})
type GenerateContentChunk = Schema.Schema.Type<typeof GenerateContentChunk>
const GenerateContentChunk = GeminiGenerateContent.chunk(
Schema.Struct({
text: Schema.optional(Schema.String),
inlineData: Schema.optional(Schema.Struct({ mimeType: Schema.String, data: Schema.Uint8ArrayFromBase64 })),
}),
)
const decodeChunk = MediaProtocol.decodeFrame(ADAPTER, NAME, GenerateContentChunk)
@@ -80,12 +49,8 @@ const decodeChunk = MediaProtocol.decodeFrame(ADAPTER, NAME, GenerateContentChun
// 4. Parser state
// ---------------------------------------------------------------------------
interface State extends SpeechStream.Audio {
interface State extends SpeechStream.Audio, GeminiGenerateContent.Metadata {
readonly mimeType?: string
readonly usage?: GenerateContentChunk["usageMetadata"]
readonly finishReason?: string
readonly modelVersion?: string
readonly responseId?: string
}
// ---------------------------------------------------------------------------
@@ -120,65 +85,27 @@ const fromRequest = Effect.fn("GoogleSpeech.fromRequest")(function* (request: Me
)
})
// Only `gemini-3.1-flash-tts-preview` and later stream; earlier TTS models reject `streamGenerateContent`.
const path = (request: MediaProtocol.Addressed<Request>) =>
request.mode === "stream"
? `/models/${request.model.id}:streamGenerateContent?alt=sse`
: `/models/${request.model.id}:generateContent`
// ---------------------------------------------------------------------------
// 6. Stream parsing
// ---------------------------------------------------------------------------
const step = Effect.fn("GoogleSpeech.step")(function* (state: State, frame: string) {
const chunk = yield* decodeChunk(frame)
const blockReason = chunk.promptFeedback?.blockReason
if (blockReason !== undefined)
return yield* new AIError({
reason: new ContentPolicyError({
message: `${NAME} blocked the prompt (${blockReason})${
chunk.promptFeedback?.blockReasonMessage === undefined ? "" : `: ${chunk.promptFeedback.blockReasonMessage}`
}`,
body: frame,
}),
})
const candidate = chunk.candidates?.[0]
const audio = (candidate?.content?.parts ?? []).flatMap((part) =>
const blocked = GeminiGenerateContent.blocked(NAME, chunk, frame)
if (blocked !== undefined) return yield* blocked
const audio = (chunk.candidates?.[0]?.content?.parts ?? []).flatMap((part) =>
part.inlineData === undefined ? [] : [part.inlineData],
)
const next: State = {
...state,
mimeType: state.mimeType ?? audio[0]?.mimeType,
usage: chunk.usageMetadata ?? state.usage,
finishReason: candidate?.finishReason ?? state.finishReason,
modelVersion: chunk.modelVersion ?? state.modelVersion,
responseId: chunk.responseId ?? state.responseId,
}
const next: State = { ...GeminiGenerateContent.track(state, chunk), mimeType: state.mimeType ?? audio[0]?.mimeType }
return [next, audio.flatMap((part) => SpeechStream.delta(next, part.data)[1])] as const
})
const finish = (state: State) => {
const usage = state.usage
const sampleRate = SpeechStream.sampleRate(state.mimeType) ?? DEFAULT_SAMPLE_RATE
return SpeechStream.finish(ADAPTER, state, {
...SpeechStream.pcm("pcm_s16le", sampleRate, state.mimeType ?? `audio/L16;codec=pcm;rate=${sampleRate}`),
usage:
usage === undefined
? undefined
: {
type: "tokens",
input: usage.promptTokenCount,
output: usage.candidatesTokenCount,
total: ProviderShared.totalTokens(
usage.promptTokenCount,
usage.candidatesTokenCount,
usage.totalTokenCount,
),
details: { google: usage },
},
providerMetadata: {
google: { finishReason: state.finishReason, modelVersion: state.modelVersion, responseId: state.responseId },
},
usage: GeminiGenerateContent.usage(state.usage),
providerMetadata: GeminiGenerateContent.providerMetadata(state),
detail: state.finishReason === undefined ? undefined : `finish reason: ${state.finishReason}`,
})
}
@@ -192,9 +119,7 @@ export const protocol = MediaProtocol.stream<Request, SpeechEvent, string, State
name: NAME,
unsupported: ["instructions", "speed", "timestamps"],
body: { from: fromRequest },
// `generateContent` answers with one document shaped exactly like a streamed chunk, so it is a single frame.
frames: (bytes, context) =>
context.request.mode === "stream" ? Framing.sse.frame(bytes) : Framing.document.frame(bytes),
frames: (bytes, context) => GeminiGenerateContent.frames(bytes, context.request.mode),
initial: () => ({ chunks: [] }),
step,
finish,
@@ -202,7 +127,14 @@ export const protocol = MediaProtocol.stream<Request, SpeechEvent, string, State
export const model = (input: MediaRoute.ModelInput) =>
SpeechModel.fromRoute<GoogleSpeechOptions, string, State>(
{ id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: ({ request }) => path(request) },
{
id: ADAPTER,
provider: PROVIDER,
protocol,
baseURL: DEFAULT_BASE_URL,
// Only `gemini-3.1-flash-tts-preview` and later stream; earlier TTS models reject `streamGenerateContent`.
path: ({ request }) => GeminiGenerateContent.path(request.model.id, request.mode),
},
input,
)
@@ -0,0 +1,214 @@
import { Effect, Schema, SchemaGetter } from "effect"
import { MediaProtocol } from "../route/media-protocol.js"
import { MediaRoute } from "../route/media.js"
import { ProviderID, mergeJsonRecords } from "../schema/index.js"
import {
TranscriptionFinishEvent,
TranscriptionModel,
TranscriptionSegmentEvent,
TranscriptionTextDeltaEvent,
type TranscriptionRequestFor,
type TranscriptionSegment,
type TranscriptionWord,
type TranscriptionEvent,
} from "../transcription.js"
import { ProviderShared } from "./shared.js"
import { GeminiGenerateContent } from "./utils/gemini-generate-content.js"
const ADAPTER = "google-transcription"
const NAME = "Google Transcription"
const PROVIDER = ProviderID.make("google")
export const DEFAULT_BASE_URL = "https://generativelanguage.googleapis.com/v1beta"
// ---------------------------------------------------------------------------
// 1. Public model input
// ---------------------------------------------------------------------------
/**
* Merged into `generationConfig`. The API rejects `customVocabulary` and `mode: "SMART"` alongside diarization or word
* timestamps.
*/
export type GoogleTranscriptionOptions = {
readonly audioTranscriptionConfig?: {
readonly mode?: "VERBATIM" | "SMART" | (string & {})
readonly customVocabulary?: ReadonlyArray<string>
readonly languageCodes?: ReadonlyArray<string>
}
} & Record<string, unknown>
export type Request = TranscriptionRequestFor<GoogleTranscriptionOptions>
// ---------------------------------------------------------------------------
// 3. Streaming event schema
// ---------------------------------------------------------------------------
const Seconds = Schema.String.check(Schema.isPattern(/^\d+(\.\d+)?s$/)).pipe(
Schema.decodeTo(Schema.Number, {
decode: SchemaGetter.transform((value) => Number.parseFloat(value)),
encode: SchemaGetter.transform((value) => `${value}s`),
}),
)
const AudioTranscription = Schema.Struct({
text: Schema.String,
speakerLabel: Schema.optional(Schema.String),
words: Schema.optional(
Schema.Array(
Schema.Struct({
word: Schema.String,
startOffset: Seconds,
endOffset: Seconds,
speakerLabel: Schema.optional(Schema.String),
}),
),
),
})
const decodeChunk = MediaProtocol.decodeFrame(
ADAPTER,
NAME,
GeminiGenerateContent.chunk(Schema.Struct({ audioTranscription: Schema.optional(AudioTranscription) })),
)
// ---------------------------------------------------------------------------
// 4. Parser state
// ---------------------------------------------------------------------------
interface State extends GeminiGenerateContent.Metadata {
readonly text: string
readonly segments: Array<TranscriptionSegment>
readonly words: Array<TranscriptionWord>
}
// ---------------------------------------------------------------------------
// 5. Request body construction
// ---------------------------------------------------------------------------
const fromRequest = Effect.fn("GoogleTranscription.fromRequest")(function* (request: MediaProtocol.Addressed<Request>) {
// General Gemini models ignore `audioTranscriptionConfig` and answer the audio conversationally.
if (!request.model.id.includes("transcribe"))
return yield* ProviderShared.unsupportedOperation({
operation: "transcription.model",
provider: PROVIDER,
route: ADAPTER,
message: `${request.model.id} is not a transcription model; use a transcribe model such as gemini-3.5-transcribe`,
})
return MediaProtocol.json(
mergeJsonRecords(
{
contents: [{ role: "user", parts: [yield* GeminiGenerateContent.mediaPart(ADAPTER, request.audio)] }],
generationConfig: mergeJsonRecords(
{
audioTranscriptionConfig: {
languageCodes: request.language === undefined ? undefined : [request.language],
// Parts carry no offsets of their own, so segment times come from word offsets.
wordTimestamp:
request.diarize === true || request.timestamps === "word" || request.timestamps === "segment"
? true
: undefined,
diarization: request.diarize === true ? true : undefined,
},
},
request.providerOptions,
),
},
request.http?.body,
) ?? {},
)
})
// ---------------------------------------------------------------------------
// 6. Stream parsing
// ---------------------------------------------------------------------------
const turn = (part: Schema.Schema.Type<typeof AudioTranscription>) => {
const words = (part.words ?? []).map((word) => ({
text: word.word,
startSeconds: word.startOffset,
endSeconds: word.endOffset,
speaker: word.speakerLabel ?? part.speakerLabel,
}))
const first = words[0]
const last = words.at(-1)
return {
text: part.text,
words,
segment:
first === undefined || last === undefined
? undefined
: {
text: part.text,
startSeconds: first.startSeconds,
endSeconds: last.endSeconds,
speaker: part.speakerLabel,
},
}
}
const step = Effect.fn("GoogleTranscription.step")(function* (state: State, frame: string) {
const chunk = yield* decodeChunk(frame)
const blocked = GeminiGenerateContent.blocked(NAME, chunk, frame)
if (blocked !== undefined) return yield* blocked
const turns = (chunk.candidates?.[0]?.content?.parts ?? []).flatMap((part) =>
part.audioTranscription === undefined ? [] : [turn(part.audioTranscription)],
)
const segments = turns.flatMap((item) => (item.segment === undefined ? [] : [item.segment]))
state.words.push(...turns.flatMap((item) => item.words))
state.segments.push(...segments)
// Each part is one whole speaker turn without surrounding whitespace, so turns join with a space.
const text = turns
.map((item) => item.text)
.filter((item) => item.length > 0)
.join(" ")
const delta = text.length === 0 || state.text.length === 0 ? text : ` ${text}`
const events: ReadonlyArray<TranscriptionEvent> = [
...(delta.length === 0 ? [] : [TranscriptionTextDeltaEvent.make({ delta })]),
...segments.map((segment) => TranscriptionSegmentEvent.make({ segment })),
]
return [{ ...GeminiGenerateContent.track(state, chunk), text: state.text + delta }, events] as const
})
const finish = (state: State) => {
if (state.finishReason === undefined) return Effect.fail(MediaProtocol.incomplete(ADAPTER))
return Effect.succeed([
TranscriptionFinishEvent.make({
text: state.text,
segments: state.segments.length === 0 ? undefined : state.segments,
words: state.words.length === 0 ? undefined : state.words,
usage: GeminiGenerateContent.usage(state.usage),
providerMetadata: GeminiGenerateContent.providerMetadata(state),
}),
])
}
// ---------------------------------------------------------------------------
// 7. Protocol and route
// ---------------------------------------------------------------------------
export const protocol = MediaProtocol.stream<Request, TranscriptionEvent, string, State>({
id: ADAPTER,
name: NAME,
unsupported: ["prompt", "speakers"],
body: { from: fromRequest },
frames: (bytes, context) => GeminiGenerateContent.frames(bytes, context.request.mode),
initial: () => ({ text: "", segments: [], words: [] }),
step,
finish,
})
export const model = (input: MediaRoute.ModelInput) =>
TranscriptionModel.fromRoute<GoogleTranscriptionOptions, string, State>(
{
id: ADAPTER,
provider: PROVIDER,
protocol,
baseURL: DEFAULT_BASE_URL,
path: ({ request }) => GeminiGenerateContent.path(request.model.id, request.mode),
},
input,
)
export const GoogleTranscription = {
protocol,
model,
} as const
+5 -8
View File
@@ -72,12 +72,6 @@ const isEdit = (request: Request) => (request.images?.length ?? 0) > 0
const isInline = (asset: Media.Asset) => asset.inline() !== undefined
const blob = (data: Uint8Array, mediaType: string) => {
const buffer = new ArrayBuffer(data.byteLength)
new Uint8Array(buffer).set(data)
return new Blob([buffer], { type: mediaType })
}
const reference = (asset: Media.Asset) =>
ProviderShared.mediaReference(asset, PROVIDER, NAME).pipe(
Effect.map((item) => (item.type === "ref" ? { file_id: item.value } : { image_url: item.value })),
@@ -104,9 +98,11 @@ const fromRequest = Effect.fn("OpenAIImages.fromRequest")(function* (request: Re
form.append(key, typeof value === "string" ? value : ProviderShared.encodeJson(value))
})
const uploads = yield* Effect.forEach(images, (image) => MediaInput.inlineBytes(ADAPTER, image))
uploads.forEach((data, index) => form.append("image[]", blob(data, images[index].mediaType), `image-${index}`))
uploads.forEach((data, index) =>
form.append("image[]", MediaInput.blob(data, images[index].mediaType), `image-${index}`),
)
if (mask !== undefined)
form.append("mask", blob(yield* MediaInput.inlineBytes(ADAPTER, mask), mask.mediaType), "mask")
form.append("mask", MediaInput.blob(yield* MediaInput.inlineBytes(ADAPTER, mask), mask.mediaType), "mask")
return MediaProtocol.multipart(form)
}
@@ -130,6 +126,7 @@ const fromRequest = Effect.fn("OpenAIImages.fromRequest")(function* (request: Re
// ---------------------------------------------------------------------------
const requestedFormat = (body: MediaProtocol.Body) => {
if (body.type === "binary") return undefined
const value = body.type === "json" ? body.value.output_format : body.value.get("output_format")
return typeof value === "string" ? value : undefined
}
@@ -0,0 +1,277 @@
import { Effect, Schema, Stream } from "effect"
import { Framing } from "../route/framing.js"
import { MediaProtocol } from "../route/media-protocol.js"
import { MediaRoute } from "../route/media.js"
import { ProviderID, mergeJsonRecords, type MediaUsage } from "../schema/index.js"
import {
TranscriptionFinishEvent,
TranscriptionModel,
TranscriptionSegmentEvent,
TranscriptionTextDeltaEvent,
type TranscriptionEvent,
type TranscriptionRequestFor,
type TranscriptionSegment,
} from "../transcription.js"
import { mediaTypeExtension } from "../utils/media-type.js"
import { ProviderShared } from "./shared.js"
import { MediaInput } from "./utils/media-input.js"
const ADAPTER = "openai-transcription"
const NAME = "OpenAI Transcription"
const PROVIDER = ProviderID.make("openai")
export const DEFAULT_BASE_URL = "https://api.openai.com/v1"
export const PATH = "/audio/transcriptions"
// ---------------------------------------------------------------------------
// 1. Public model input
// ---------------------------------------------------------------------------
export type OpenAITranscriptionOptions = {
readonly chunking_strategy?:
| "auto"
| {
readonly type: "server_vad"
readonly prefix_padding_ms?: number
readonly silence_duration_ms?: number
readonly threshold?: number
}
readonly include?: ReadonlyArray<"logprobs">
readonly keywords?: ReadonlyArray<string>
readonly known_speaker_names?: ReadonlyArray<string>
readonly known_speaker_references?: ReadonlyArray<string>
readonly temperature?: number
} & Record<string, unknown>
export type Request = TranscriptionRequestFor<OpenAITranscriptionOptions>
// ---------------------------------------------------------------------------
// 3. Streaming event schema
// ---------------------------------------------------------------------------
const Segment = Schema.Struct({
text: Schema.String,
start: Schema.Number,
end: Schema.Number,
speaker: Schema.optional(Schema.String),
})
const Usage = Schema.Union([
Schema.Struct({
type: Schema.Literal("tokens"),
input_tokens: Schema.optional(Schema.Number),
output_tokens: Schema.optional(Schema.Number),
total_tokens: Schema.optional(Schema.Number),
}),
Schema.Struct({ type: Schema.Literal("duration"), seconds: Schema.Number }),
])
const transcriptFields = {
text: Schema.String,
language: Schema.optional(Schema.String),
languages: Schema.optional(Schema.Array(Schema.Struct({ code: Schema.String }))),
duration: Schema.optional(Schema.Number),
segments: Schema.optional(Schema.Array(Segment)),
words: Schema.optional(
Schema.Array(Schema.Struct({ word: Schema.String, start: Schema.Number, end: Schema.Number })),
),
usage: Schema.optional(Usage),
}
const Event = Schema.Union([
Schema.Struct({ type: Schema.Literal("transcript.text.delta"), delta: Schema.String }),
Schema.Struct({ type: Schema.Literal("transcript.text.segment"), ...Segment.fields }),
Schema.Struct({ type: Schema.Literal("transcript.text.done"), ...transcriptFields }),
])
const Transcript = Schema.Struct(transcriptFields)
type Transcript = Schema.Schema.Type<typeof Transcript>
const decodeEvent = MediaProtocol.decodeFrame(ADAPTER, NAME, Event)
const decodeTranscript = MediaProtocol.decodeFrame(ADAPTER, NAME, Transcript)
type Frame = string | { readonly document: string }
// ---------------------------------------------------------------------------
// 4. Parser state
// ---------------------------------------------------------------------------
interface State {
readonly segments: Array<TranscriptionSegment>
readonly transcript?: Transcript
}
// ---------------------------------------------------------------------------
// 5. Request body construction
// ---------------------------------------------------------------------------
interface Capabilities {
readonly stream: boolean
readonly timestamps: ReadonlyArray<"segment" | "word">
readonly diarize: boolean
readonly languageField: "language" | "languages"
}
const TRANSCRIBE: Capabilities = { stream: true, timestamps: [], diarize: false, languageField: "language" }
const capabilities = (model: string): Capabilities => {
if (model.startsWith("whisper")) return { ...TRANSCRIBE, stream: false, timestamps: ["segment", "word"] }
if (model.includes("diarize")) return { ...TRANSCRIBE, timestamps: ["segment"], diarize: true }
// `gpt-transcribe` replaces `language` with `languages[]` and rejects both together.
if (model.startsWith("gpt-transcribe")) return { ...TRANSCRIBE, languageField: "languages" }
return TRANSCRIBE
}
const unsupported = (operation: string, message: string) =>
Effect.fail(ProviderShared.unsupportedOperation({ operation, provider: PROVIDER, route: ADAPTER, message }))
const validate = (request: MediaProtocol.Addressed<Request>, model: Capabilities) => {
const id = request.model.id
if (request.mode === "stream" && !model.stream)
return unsupported("media.stream", `${id} does not stream; use Transcription.generate`)
if (request.diarize === true && !model.diarize)
return unsupported("media.diarize", `${id} does not diarize; use gpt-4o-transcribe-diarize`)
if (request.prompt !== undefined && model.diarize)
return unsupported("media.prompt", `${id} does not accept a prompt`)
if (
request.timestamps === undefined ||
request.timestamps === "none" ||
model.timestamps.includes(request.timestamps)
)
return Effect.void
return unsupported("media.timestamps", `${id} does not return ${request.timestamps} timestamps`)
}
const RESERVED_FORM_FIELDS = new Set([
"file",
"model",
"prompt",
"language",
"response_format",
"timestamp_granularities",
"stream",
])
const appendField = (form: FormData, key: string, value: unknown) => {
if (Array.isArray(value)) return value.forEach((item) => form.append(`${key}[]`, String(item)))
form.append(key, typeof value === "object" && value !== null ? ProviderShared.encodeJson(value) : String(value))
}
const fromRequest = Effect.fn("OpenAITranscription.fromRequest")(function* (request: MediaProtocol.Addressed<Request>) {
const model = capabilities(request.model.id)
yield* validate(request, model)
// The API detects the audio format from the upload's filename extension.
const extension = mediaTypeExtension(request.audio.mediaType)
if (extension === undefined)
return yield* ProviderShared.invalidRequest(
`${NAME} cannot name a ${request.audio.mediaType} upload; send mp3, mp4, m4a, wav, webm, ogg, or flac audio`,
)
const audio = yield* MediaInput.inlineBytes(ADAPTER, request.audio)
const responseFormat = model.diarize
? "diarized_json"
: request.timestamps === undefined || request.timestamps === "none"
? undefined
: "verbose_json"
const native = Object.entries(mergeJsonRecords(request.providerOptions, request.http?.body) ?? {}).filter(
([key]) => !RESERVED_FORM_FIELDS.has(key),
)
const fields = mergeJsonRecords(
{
model: request.model.id,
language: model.languageField === "language" ? request.language : undefined,
languages: model.languageField === "languages" && request.language !== undefined ? [request.language] : undefined,
prompt: request.prompt,
response_format: responseFormat,
timestamp_granularities: responseFormat === "verbose_json" ? [request.timestamps] : undefined,
// Diarizing audio longer than 30 seconds requires a chunking strategy.
chunking_strategy: model.diarize ? "auto" : undefined,
stream: request.mode === "stream" ? true : undefined,
},
Object.fromEntries(native),
)
const form = new FormData()
form.append("file", MediaInput.blob(audio, request.audio.mediaType), `audio.${extension}`)
Object.entries(fields ?? {}).forEach(([key, value]) => appendField(form, key, value))
return MediaProtocol.multipart(form)
})
// ---------------------------------------------------------------------------
// 6. Stream parsing
// ---------------------------------------------------------------------------
const segment = (value: Schema.Schema.Type<typeof Segment>): TranscriptionSegment => ({
text: value.text.trim(),
startSeconds: value.start,
endSeconds: value.end,
speaker: value.speaker,
})
const onEvent = Effect.fn("OpenAITranscription.onEvent")(function* (state: State, frame: string) {
const event = yield* decodeEvent(frame)
if (event.type === "transcript.text.done") return [{ ...state, transcript: event }, []] as const
if (event.type === "transcript.text.delta")
return [state, event.delta.length === 0 ? [] : [TranscriptionTextDeltaEvent.make({ delta: event.delta })]] as const
const next = segment(event)
state.segments.push(next)
return [state, [TranscriptionSegmentEvent.make({ segment: next })]] as const
})
const step = (state: State, frame: Frame) =>
typeof frame === "string"
? onEvent(state, frame)
: decodeTranscript(frame.document).pipe(Effect.map((transcript) => [{ ...state, transcript }, []] as const))
const usage = (value: Transcript["usage"]): MediaUsage | undefined => {
if (value === undefined) return undefined
if (value.type === "duration") return { type: "seconds", seconds: value.seconds }
return {
type: "tokens",
input: value.input_tokens,
output: value.output_tokens,
total: value.total_tokens,
details: { openai: value },
}
}
const finish = (state: State) => {
const transcript = state.transcript
if (transcript === undefined) return Effect.fail(MediaProtocol.incomplete(ADAPTER))
const segments = transcript.segments?.map(segment) ?? state.segments
return Effect.succeed([
TranscriptionFinishEvent.make({
text: transcript.text,
segments: segments.length === 0 ? undefined : segments,
words: transcript.words?.map((word) => ({ text: word.word, startSeconds: word.start, endSeconds: word.end })),
language: (transcript.language ?? transcript.languages?.[0]?.code)?.toLowerCase(),
durationSeconds: transcript.duration,
usage: usage(transcript.usage),
}),
])
}
// ---------------------------------------------------------------------------
// 7. Protocol and route
// ---------------------------------------------------------------------------
export const protocol = MediaProtocol.stream<Request, TranscriptionEvent, Frame, State>({
id: ADAPTER,
name: NAME,
unsupported: ["speakers"],
body: { from: fromRequest },
frames: (bytes, context) =>
context.request.mode === "stream"
? Framing.sse.frame(bytes)
: Framing.document.frame(bytes).pipe(Stream.map((document) => ({ document }))),
initial: () => ({ segments: [] }),
step,
finish,
})
export const model = (input: MediaRoute.ModelInput) =>
TranscriptionModel.fromRoute<OpenAITranscriptionOptions, Frame, State>(
{ id: ADAPTER, provider: PROVIDER, protocol, baseURL: DEFAULT_BASE_URL, path: PATH },
input,
)
export const OpenAITranscription = {
protocol,
model,
} as const
@@ -0,0 +1,113 @@
import { Effect, Schema, type Stream } from "effect"
import type { Media } from "../../media.js"
import { Framing } from "../../route/framing.js"
import type { MediaProtocol } from "../../route/media-protocol.js"
import { AIError, ContentPolicyError, ProviderID, type MediaUsage, type ProviderMetadata } from "../../schema/index.js"
import { ProviderShared } from "../shared.js"
import { MediaInput } from "./media-input.js"
const PROVIDER = ProviderID.make("google")
const UsageMetadata = Schema.Struct({
promptTokenCount: Schema.optional(Schema.Number),
candidatesTokenCount: Schema.optional(Schema.Number),
totalTokenCount: Schema.optional(Schema.Number),
})
type UsageMetadata = Schema.Schema.Type<typeof UsageMetadata>
export const chunk = <const Part extends Schema.Top>(part: Part) =>
Schema.Struct({
candidates: Schema.optional(
Schema.Array(
Schema.Struct({
content: Schema.optional(Schema.Struct({ parts: Schema.optional(Schema.Array(part)) })),
finishReason: Schema.optional(Schema.String),
}),
),
),
promptFeedback: Schema.optional(
Schema.Struct({
blockReason: Schema.optional(Schema.String),
blockReasonMessage: Schema.optional(Schema.String),
}),
),
usageMetadata: Schema.optional(UsageMetadata),
modelVersion: Schema.optional(Schema.String),
responseId: Schema.optional(Schema.String),
})
interface Chunk {
readonly candidates?: ReadonlyArray<{ readonly finishReason?: string }>
readonly promptFeedback?: { readonly blockReason?: string; readonly blockReasonMessage?: string }
readonly usageMetadata?: UsageMetadata
readonly modelVersion?: string
readonly responseId?: string
}
export interface Metadata {
readonly usage?: UsageMetadata
readonly finishReason?: string
readonly modelVersion?: string
readonly responseId?: string
}
export const track = <State extends Metadata>(state: State, chunk: Chunk): State => ({
...state,
usage: chunk.usageMetadata ?? state.usage,
finishReason: chunk.candidates?.[0]?.finishReason ?? state.finishReason,
modelVersion: chunk.modelVersion ?? state.modelVersion,
responseId: chunk.responseId ?? state.responseId,
})
export const blocked = (name: string, chunk: Chunk, frame: string) => {
const feedback = chunk.promptFeedback
if (feedback?.blockReason === undefined) return undefined
return new AIError({
reason: new ContentPolicyError({
message: `${name} blocked the request (${feedback.blockReason})${
feedback.blockReasonMessage === undefined ? "" : `: ${feedback.blockReasonMessage}`
}`,
body: frame,
}),
})
}
export const usage = (usage: UsageMetadata | undefined): MediaUsage | undefined =>
usage === undefined
? undefined
: {
type: "tokens",
input: usage.promptTokenCount,
output: usage.candidatesTokenCount,
total: ProviderShared.totalTokens(usage.promptTokenCount, usage.candidatesTokenCount, usage.totalTokenCount),
details: { google: usage },
}
export const providerMetadata = (state: Metadata): ProviderMetadata => ({
google: { finishReason: state.finishReason, modelVersion: state.modelVersion, responseId: state.responseId },
})
export const path = (model: string, mode: MediaProtocol.Mode) =>
mode === "stream" ? `/models/${model}:streamGenerateContent?alt=sse` : `/models/${model}:generateContent`
// `generateContent` answers with one document shaped exactly like a streamed chunk, so it is a single frame.
export const frames = (bytes: Stream.Stream<Uint8Array, AIError>, mode: MediaProtocol.Mode) =>
mode === "stream" ? Framing.sse.frame(bytes) : Framing.document.frame(bytes)
// Gemini does not fetch public URLs; inline payloads and Gemini Files references are the accepted inputs.
export const mediaPart = (
route: string,
asset: Media.Asset,
): Effect.Effect<
| { readonly fileData: { readonly mimeType: string; readonly fileUri: string } }
| { readonly inlineData: { readonly mimeType: string; readonly data: string } },
AIError
> => {
const fileUri = MediaInput.refID(asset, PROVIDER)
if (fileUri !== undefined) return Effect.succeed({ fileData: { mimeType: asset.mediaType, fileUri } })
return ProviderShared.requireInlineMedia(route, asset).pipe(
Effect.map((media) => ({ inlineData: { mimeType: media.mime, data: media.base64 } })),
)
}
export * as GeminiGenerateContent from "./gemini-generate-content.js"
@@ -1,5 +1,6 @@
import { Effect, Encoding } from "effect"
import { Media } from "../../media.js"
import type { MediaProtocol } from "../../route/media-protocol.js"
import type { AIError, ProviderID } from "../../schema/index.js"
import { ProviderShared } from "../shared.js"
@@ -13,6 +14,26 @@ export const inlineBytes = (route: string, asset: Media.Asset): Effect.Effect<Ui
)
}
/** Copied because `BlobPart` requires a plain `ArrayBuffer`. */
export const blob = (data: Uint8Array, mediaType: string) => {
const buffer = new ArrayBuffer(data.byteLength)
new Uint8Array(buffer).set(data)
return new Blob([buffer], { type: mediaType })
}
const isScalar = (value: unknown): value is string | number | boolean =>
typeof value === "string" || typeof value === "number" || typeof value === "boolean"
export const query = (route: string, values: Record<string, unknown>): Effect.Effect<MediaProtocol.Query, AIError> => {
const entries = Object.entries(values).filter(([, value]) => value !== undefined)
const invalid = entries.find(([, value]) => !isScalar(value) && !(Array.isArray(value) && value.every(isScalar)))
if (invalid !== undefined)
return Effect.fail(ProviderShared.invalidRequest(`${route} cannot send "${invalid[0]}" as a query parameter`))
return Effect.succeed(
Object.fromEntries(entries.map(([key, value]) => [key, Array.isArray(value) ? value.map(String) : String(value)])),
)
}
/** Provider file handle when the ref belongs to this provider; refs from other providers are never forwarded. */
export const refID = (asset: Media.Asset, provider: ProviderID) =>
asset.source.type === "ref" && asset.source.provider === provider ? asset.source.id : undefined
+43
View File
@@ -0,0 +1,43 @@
import { Auth } from "../route/auth.js"
import type { ProviderAuthOption } from "../route/auth-options.js"
import { HttpOptions, ProviderID, type ModelID } from "../schema/index.js"
import { AssemblyAITranscription, DEFAULT_BASE_URL } from "../protocols/assemblyai-transcription.js"
export type { AssemblyAITranscriptionOptions } from "../protocols/assemblyai-transcription.js"
export const id = ProviderID.make("assemblyai")
const baseURL = DEFAULT_BASE_URL
export type Config = ProviderAuthOption<"optional"> & {
/** `https://api.eu.assemblyai.com` for the EU region. */
readonly baseURL?: string
readonly headers?: Record<string, string>
readonly http?: HttpOptions.Input
}
// The key is the whole `authorization` value, without a scheme.
const auth = (options: ProviderAuthOption<"optional">) => {
if ("auth" in options && options.auth) return options.auth
return Auth.optional("apiKey" in options ? options.apiKey : undefined, "apiKey")
.orElse(Auth.config("ASSEMBLYAI_API_KEY"))
.pipe(Auth.header("authorization"))
}
export const configure = (input: Config = {}) => {
const transcription = (modelID: string | ModelID) =>
AssemblyAITranscription.model({
id: modelID,
auth: auth(input),
baseURL: input.baseURL ?? baseURL,
headers: input.headers,
http: input.http === undefined ? undefined : HttpOptions.make(input.http),
})
return {
id,
transcription,
configure,
}
}
export const provider = configure()
export const transcription = provider.transcription
+12 -9
View File
@@ -2,8 +2,10 @@ import { Auth } from "../route/auth.js"
import type { ProviderAuthOption } from "../route/auth-options.js"
import { HttpOptions, ProviderID, type ModelID } from "../schema/index.js"
import { DEFAULT_BASE_URL, DeepgramSpeech } from "../protocols/deepgram-speech.js"
import { DeepgramTranscription } from "../protocols/deepgram-transcription.js"
export type { DeepgramEncoding, DeepgramSpeechOptions } from "../protocols/deepgram-speech.js"
export type { DeepgramTranscriptionOptions } from "../protocols/deepgram-transcription.js"
export const id = ProviderID.make("deepgram")
const baseURL = DEFAULT_BASE_URL
@@ -22,20 +24,21 @@ const auth = (options: ProviderAuthOption<"optional">) => {
}
export const configure = (input: Config = {}) => {
const speech = (modelID: string | ModelID) =>
DeepgramSpeech.model({
id: modelID,
auth: auth(input),
baseURL: input.baseURL ?? baseURL,
headers: input.headers,
http: input.http === undefined ? undefined : HttpOptions.make(input.http),
})
const media = (modelID: string | ModelID) => ({
id: modelID,
auth: auth(input),
baseURL: input.baseURL ?? baseURL,
headers: input.headers,
http: input.http === undefined ? undefined : HttpOptions.make(input.http),
})
return {
id,
speech,
speech: (modelID: string | ModelID) => DeepgramSpeech.model(media(modelID)),
transcription: (modelID: string | ModelID) => DeepgramTranscription.model(media(modelID)),
configure,
}
}
export const provider = configure()
export const speech = provider.speech
export const transcription = provider.transcription
+4
View File
@@ -6,10 +6,12 @@ import { HttpOptions, ProviderID, mergeHttpOptions, type ModelID } from "../sche
import { Gemini } from "../protocols/gemini.js"
import { GoogleImages } from "../protocols/google-images.js"
import { GoogleSpeech } from "../protocols/google-speech.js"
import { GoogleTranscription } from "../protocols/google-transcription.js"
import { GoogleVideo } from "../protocols/google-video.js"
export type { GoogleImageOptions } from "../protocols/google-images.js"
export type { GoogleSpeechOptions } from "../protocols/google-speech.js"
export type { GoogleTranscriptionOptions } from "../protocols/google-transcription.js"
export type { GoogleVideoOptions } from "../protocols/google-video.js"
export type GeminiOptionsInput = Gemini.OptionsInput
export type GeminiProviderOptionsInput = Gemini.ProviderOptionsInput
@@ -57,6 +59,7 @@ export const configure = (input: Config = {}) => {
image: (modelID: string | ModelID) => GoogleImages.model(media(modelID)),
video: (modelID: string | ModelID) => GoogleVideo.model(media(modelID)),
speech: (modelID: string | ModelID) => GoogleSpeech.model(media(modelID)),
transcription: (modelID: string | ModelID) => GoogleTranscription.model(media(modelID)),
configure,
}
}
@@ -77,3 +80,4 @@ export const model: ProviderPackage.Definition<Settings, Gemini.ProviderOptionsI
export const image = provider.image
export const video = provider.video
export const speech = provider.speech
export const transcription = provider.transcription
+1
View File
@@ -3,6 +3,7 @@ export * as Anthropic from "./anthropic.js"
export * as AnthropicCompatible from "./anthropic-compatible.js"
export * as AmazonBedrock from "./amazon-bedrock.js"
export * as AmazonBedrockMantle from "./amazon-bedrock-mantle.js"
export * as AssemblyAI from "./assemblyai.js"
export * as Azure from "./azure.js"
export * as Baseten from "./baseten.js"
export * as Cartesia from "./cartesia.js"
+5
View File
@@ -7,10 +7,12 @@ import * as OpenAIResponses from "../protocols/openai-responses.js"
import { withOpenAIOptions, type OpenAIProviderOptionsInput } from "./openai-options.js"
import { OpenAIImages, type OpenAIImageString } from "../protocols/openai-images.js"
import { OpenAISpeech } from "../protocols/openai-speech.js"
import { OpenAITranscription } from "../protocols/openai-transcription.js"
export type { OpenAIOptionsInput, OpenAIResponseIncludable } from "./openai-options.js"
export type { OpenAIImageOptions } from "../protocols/openai-images.js"
export type { OpenAISpeechOptions } from "../protocols/openai-speech.js"
export type { OpenAITranscriptionOptions } from "../protocols/openai-transcription.js"
export const id = ProviderID.make("openai")
@@ -109,6 +111,7 @@ export const configure = (input: Config = {}) => {
})
const image = (modelID: string | ModelID) => OpenAIImages.model(media(modelID))
const speech = (modelID: string | ModelID) => OpenAISpeech.model(media(modelID))
const transcription = (modelID: string | ModelID) => OpenAITranscription.model(media(modelID))
return {
id,
@@ -117,6 +120,7 @@ export const configure = (input: Config = {}) => {
chat,
image,
speech,
transcription,
configure,
}
}
@@ -164,3 +168,4 @@ export const responses = provider.responses
export const chat = provider.chat
export const image = provider.image
export const speech = provider.speech
export const transcription = provider.transcription
+26 -5
View File
@@ -16,17 +16,37 @@ import {
// Bodies
// ---------------------------------------------------------------------------
/** JSON `query` is appended to the endpoint URL before the route and caller `http.query` overlays. */
export type Body =
| { readonly type: "json"; readonly value: Record<string, unknown>; readonly query?: Record<string, string> }
| { readonly type: "multipart"; readonly value: FormData }
/** Array values become repeated parameters (`keyterm=a&keyterm=b`). */
export type Query = Readonly<Record<string, string | ReadonlyArray<string>>>
export const json = (value: Record<string, unknown>, query?: Record<string, string>): Body => ({
/** `query` is appended to the endpoint URL before the route and caller `http.query` overlays. */
export type Body =
| { readonly type: "json"; readonly value: Record<string, unknown>; readonly query?: Query }
| { readonly type: "multipart"; readonly value: FormData }
| {
readonly type: "binary"
readonly value: Uint8Array
readonly contentType: string
readonly query?: Query
}
export const json = (value: Record<string, unknown>, query?: Query): Body => ({
type: "json",
value,
query,
})
export const multipart = (value: FormData): Body => ({ type: "multipart", value })
export const binary = (value: Uint8Array, contentType: string, query?: Query): Body => ({
type: "binary",
value,
contentType,
query,
})
export type Send = (path: string, body: Body) => Effect.Effect<HttpClientResponse.HttpClientResponse, AIError>
/** Runs after unsupported-field rejection and before `body.from`, for providers that need an upload first. */
export type Prepare<Request> = (request: Request, send: Send) => Effect.Effect<Request, AIError>
// ---------------------------------------------------------------------------
// Protocol kinds
@@ -91,6 +111,7 @@ export interface Queued<Request, Response, Token> {
/** Serializable handle. `Generation.token` carries the encoded form so it can be persisted and resumed elsewhere. */
readonly token: Schema.Codec<Token, unknown>
readonly start: {
readonly prepare?: Prepare<Request>
readonly body: { readonly from: (request: Request) => Effect.Effect<Body, AIError> }
readonly decode: (
response: HttpClientResponse.HttpClientResponse,
+82 -52
View File
@@ -40,6 +40,7 @@ export interface ModelInput {
/** One request, one response. */
export interface Route<Request extends MediaRequest, Response> {
readonly kind: "inline"
readonly id: string
readonly provider: ProviderID
readonly protocol: string
@@ -48,6 +49,7 @@ export interface Route<Request extends MediaRequest, Response> {
/** Submit, then poll through the returned `Generation`. */
export interface QueuedRoute<Request extends MediaRequest, Response> {
readonly kind: "queued"
readonly id: string
readonly provider: ProviderID
readonly protocol: string
@@ -62,6 +64,7 @@ export interface QueuedRoute<Request extends MediaRequest, Response> {
/** One request whose response parses into events; `generate` runs the same stream and collects it. */
export interface StreamRoute<Request extends MediaRequest, Event, Response> {
readonly kind: "stream"
readonly id: string
readonly provider: ProviderID
readonly protocol: string
@@ -95,14 +98,15 @@ export interface StreamInput<Request extends MediaRequest, Event, Response, Fram
/**
* Compose an inline media protocol with an endpoint and auth into a runnable route. The route owns the transport
* plumbing every media protocol would otherwise duplicate: option merging, surrogate sanitizing, unsupported-field
* rejection, URL and query rendering, auth headers, JSON vs multipart encoding, and handing responses back to the
* protocol.
* rejection, URL and query rendering, auth headers, JSON, multipart, or binary encoding, and handing responses back
* to the protocol.
*/
export const inline = <Request extends MediaRequest, Response>(
input: InlineInput<Request, Response>,
): Route<Request, Response> => {
const transport = makeTransport(input)
return {
kind: "inline",
id: input.id,
provider: transport.provider,
protocol: input.protocol.id,
@@ -157,7 +161,7 @@ export const queued = <Request extends MediaRequest, Response, Token>(
const start = Effect.fn("MediaRoute.start")(function* (request: Request, execute: Execute) {
const submitted = yield* transport.submit(
request,
{ unsupported: protocol.unsupported, from: protocol.start.body.from },
{ unsupported: protocol.unsupported, prepare: protocol.start.prepare, from: protocol.start.body.from },
execute,
)
const started = yield* protocol.start.decode(submitted.response, submitted.context)
@@ -185,7 +189,7 @@ export const queued = <Request extends MediaRequest, Response, Token>(
return new Generation(route, encodeToken(token), yield* route.status)
})
return { id: input.id, provider: transport.provider, protocol: protocol.id, start, resume }
return { kind: "queued", id: input.id, provider: transport.provider, protocol: protocol.id, start, resume }
}
/** Compose a streaming media protocol; `generate` runs the same stream in `generate` mode and folds it with `collect`. */
@@ -236,6 +240,7 @@ export const stream = <Request extends MediaRequest, Event, Response, Frame, Sta
}),
)
return {
kind: "stream",
id: input.id,
provider: transport.provider,
protocol: protocol.id,
@@ -253,10 +258,32 @@ const makeTransport = <Request extends MediaRequest>(input: Composition<Request>
const provider = ProviderID.make(input.provider)
const routeHttp = input.headers === undefined ? undefined : new HttpOptions({ headers: input.headers })
const authorize = Auth.toEffect(input.auth)
const withQuery = (url: URL, query: Record<string, string> | undefined) => {
for (const [key, value] of Object.entries(query ?? {})) url.searchParams.set(key, value)
return url
}
const baseURL = (path: string) => new URL(`${ProviderShared.trimBaseUrl(input.endpoint.baseURL ?? "")}${path}`)
/** `auth` is only what `Auth` added, never deployment headers. */
const send = Effect.fn("MediaRoute.send")(function* (
call: {
readonly method: AuthInput["method"]
readonly url: URL
readonly headers: Headers.Headers
readonly request: AuthInput["request"]
readonly body?: MediaProtocol.Body
},
execute: Execute,
) {
const encoded = encode(call.body, call.headers)
const url = call.url.toString()
const headers = yield* authorize({
request: call.request,
method: call.method,
url,
body: encoded.text,
headers: encoded.headers,
})
const response = yield* execute(
encoded.apply(HttpClientRequest.make(call.method)(url).pipe(HttpClientRequest.setHeaders(headers))),
)
return { response, auth: Object.fromEntries(Object.entries(headers).filter(([key]) => !(key in call.headers))) }
})
return {
provider,
/** Route and model overlays; `start` additionally merges the request's own `http`. */
@@ -266,67 +293,70 @@ const makeTransport = <Request extends MediaRequest>(input: Composition<Request>
request: Request,
protocol: {
readonly unsupported?: ReadonlyArray<keyof Request & string>
readonly prepare?: MediaProtocol.Prepare<Request>
readonly from: (request: Request) => Effect.Effect<MediaProtocol.Body, AIError>
},
execute: Execute,
) {
yield* rejectUnsupported(input.id, provider, request, protocol.unsupported)
const http = mergeHttpOptions(routeHttp, request.model.http, request.http)
const headers = Headers.fromInput(http?.headers)
const prepared =
protocol.prepare === undefined
? request
: yield* protocol.prepare(request, (path, body) =>
send({ method: "POST", url: baseURL(path), headers, request, body }, execute).pipe(
Effect.map((sent) => sent.response),
),
)
// Sanitize after merging so model-level overlays are covered; the model value is restored, not sanitized.
const resolved: Request = { ...sanitizeSurrogates({ ...request, http }), model: request.model }
const resolved: Request = { ...sanitizeSurrogates({ ...prepared, http }), model: request.model }
const body = yield* protocol.from(resolved)
const url = withQuery(
withQuery(
Endpoint.render(input.endpoint, { request: resolved, body }),
body.type === "json" ? body.query : undefined,
body.type === "multipart" ? undefined : body.query,
),
http?.query,
)
const encoded = body.type === "json" ? ProviderShared.encodeJson(body.value) : "[multipart/form-data]"
const baseHeaders = Headers.fromInput(http?.headers)
const headers = yield* authorize({
request: resolved,
method: "POST",
url: url.toString(),
body: encoded,
// The HTTP client sets the multipart boundary; a caller-supplied content-type would corrupt it.
headers: body.type === "multipart" ? Headers.remove(baseHeaders, "content-type") : baseHeaders,
})
const transport = HttpClientRequest.post(url.toString()).pipe(
HttpClientRequest.setHeaders(headers),
body.type === "json"
? HttpClientRequest.bodyText(encoded, "application/json")
: HttpClientRequest.bodyFormData(body.value),
)
const response = yield* execute(transport)
return { response, context: { request: resolved, body } }
const sent = yield* send({ method: "POST", url, headers, request: resolved, body }, execute)
return { response: sent.response, context: { request: resolved, body } }
}),
/**
* Bodiless follow-up call (status, result, cancel) with the same auth and headers as `submit`. `auth` is only
* what `Auth` added, so protocols can hand download credentials to output assets without deployment headers.
*/
call: Effect.fn("MediaRoute.call")(function* (
method: AuthInput["method"],
path: string,
http: HttpOptions | undefined,
execute: Execute,
) {
/** Bodiless follow-up call (status, result, cancel) with the same auth and headers as `submit`. */
call: (method: AuthInput["method"], path: string, http: HttpOptions | undefined, execute: Execute) => {
// Provider-issued absolute URLs (fal `status_url`) are used as-is; everything else resolves against the base.
const url = withQuery(
/^https?:\/\//.test(path)
? new URL(path)
: new URL(`${ProviderShared.trimBaseUrl(input.endpoint.baseURL ?? "")}${path}`),
http?.query,
)
const url = withQuery(/^https?:\/\//.test(path) ? new URL(path) : baseURL(path), http?.query)
for (const [key, value] of Object.entries(input.endpoint.query ?? {})) url.searchParams.set(key, value)
const base = Headers.fromInput(http?.headers)
const headers = yield* authorize({ request: { http }, method, url: url.toString(), body: "", headers: base })
const response = yield* execute(
HttpClientRequest.make(method)(url.toString()).pipe(HttpClientRequest.setHeaders(headers)),
)
const auth = Object.fromEntries(Object.entries(headers).filter(([key]) => !(key in base)))
return { response, auth }
}),
return send({ method, url, headers: Headers.fromInput(http?.headers), request: { http } }, execute)
},
}
}
const withQuery = (url: URL, query: MediaProtocol.Query | undefined) => {
for (const [key, value] of Object.entries(query ?? {})) {
url.searchParams.delete(key)
for (const item of typeof value === "string" ? [value] : value) url.searchParams.append(key, item)
}
return url
}
const encode = (body: MediaProtocol.Body | undefined, headers: Headers.Headers) => {
if (body === undefined) return { text: "", headers, apply: (request: HttpClientRequest.HttpClientRequest) => request }
if (body.type === "json") {
const text = ProviderShared.encodeJson(body.value)
return { text, headers, apply: HttpClientRequest.bodyText(text, "application/json") }
}
if (body.type === "binary")
return {
text: `[${body.contentType}]`,
headers,
apply: HttpClientRequest.bodyUint8Array(body.value, body.contentType),
}
return {
text: "[multipart/form-data]",
// The HTTP client sets the multipart boundary; a caller-supplied content-type would corrupt it.
headers: Headers.remove(headers, "content-type"),
apply: HttpClientRequest.bodyFormData(body.value),
}
}
+122
View File
@@ -0,0 +1,122 @@
import { Context, Effect, Layer, Stream } from "effect"
import { resultEvents, type AwaitOptions, type Generation } from "./generation.js"
import { ProviderShared } from "./protocols/shared.js"
import { RequestExecutor } from "./route/executor.js"
import type { AIError } from "./schema/index.js"
import {
responseEvents,
type TranscriptionEvent,
type TranscriptionModel,
type TranscriptionOptions,
type TranscriptionRequestFor,
type TranscriptionResponse,
type TranscriptionRoute,
} from "./transcription.js"
export interface Interface {
readonly generate: <Options extends TranscriptionOptions>(
request: TranscriptionRequestFor<Options>,
options?: AwaitOptions,
) => Effect.Effect<TranscriptionResponse, AIError>
readonly stream: <Options extends TranscriptionOptions>(
request: TranscriptionRequestFor<Options>,
options?: AwaitOptions,
) => Stream.Stream<TranscriptionEvent, AIError>
readonly start: <Options extends TranscriptionOptions>(
request: TranscriptionRequestFor<Options>,
) => Effect.Effect<Generation<TranscriptionResponse>, AIError>
readonly resume: <Options extends TranscriptionOptions>(
model: TranscriptionModel<Options>,
token: unknown,
) => Effect.Effect<Generation<TranscriptionResponse>, AIError>
}
export class Service extends Context.Service<Service, Interface>()("@opencode/TranscriptionClient") {}
export const generate = <Options extends TranscriptionOptions>(
request: TranscriptionRequestFor<Options>,
options?: AwaitOptions,
): Effect.Effect<TranscriptionResponse, AIError, Service> =>
Effect.gen(function* () {
const client = yield* Service
return yield* client.generate(request, options)
})
export const stream = <Options extends TranscriptionOptions>(
request: TranscriptionRequestFor<Options>,
options?: AwaitOptions,
): Stream.Stream<TranscriptionEvent, AIError, Service> =>
Stream.unwrap(
Effect.gen(function* () {
const client = yield* Service
return client.stream(request, options)
}),
)
export const start = <Options extends TranscriptionOptions>(
request: TranscriptionRequestFor<Options>,
): Effect.Effect<Generation<TranscriptionResponse>, AIError, Service> =>
Effect.gen(function* () {
const client = yield* Service
return yield* client.start(request)
})
export const resume = <Options extends TranscriptionOptions>(
model: TranscriptionModel<Options>,
token: unknown,
): Effect.Effect<Generation<TranscriptionResponse>, AIError, Service> =>
Effect.gen(function* () {
const client = yield* Service
return yield* client.resume(model, token)
})
const notQueued = <Options extends TranscriptionOptions>(route: TranscriptionRoute<Options>, operation: string) =>
ProviderShared.unsupportedOperation({
operation: `transcription.${operation}`,
provider: route.provider,
route: route.id,
message: `${route.provider}/${route.id} is not a queued route; use Transcription.generate or Transcription.stream`,
})
export const layer: Layer.Layer<Service, never, RequestExecutor.Service> = Layer.effect(
Service,
Effect.gen(function* () {
const executor = yield* RequestExecutor.Service
const start = <Options extends TranscriptionOptions>(request: TranscriptionRequestFor<Options>) => {
const route = request.model.route
if (route.kind !== "queued") return Effect.fail(notQueued(route, "start"))
return route.start(request, executor.execute)
}
return Service.of({
start,
resume: (model, token) => {
const route = model.route
if (route.kind !== "queued") return Effect.fail(notQueued(route, "resume"))
return route.resume(model, token, executor.execute)
},
generate: (request, options) => {
const route = request.model.route
if (route.kind !== "queued") return route.generate(request, executor.execute)
return start(request).pipe(Effect.flatMap((generation) => generation.await(options)))
},
stream: (request, options) => {
const route = request.model.route
if (route.kind === "stream") return route.stream(request, executor.execute)
if (route.kind === "queued")
return Stream.unwrap(
start(request).pipe(Effect.map((generation) => resultEvents(generation, responseEvents, options))),
)
return Stream.fromIterableEffect(Effect.map(route.generate(request, executor.execute), responseEvents))
},
})
}),
)
export const TranscriptionClient = {
Service,
layer,
generate,
stream,
start,
resume,
} as const
+294
View File
@@ -0,0 +1,294 @@
import { Effect, Schema, Stream } from "effect"
import { Generation, ProgressEvent, QueuedEvent, type AwaitOptions } from "./generation.js"
import { Media } from "./media.js"
import { MediaModel, composeRoute, tryRequest } from "./media-model.js"
import { MediaRoute } from "./route/media.js"
import type { MediaProtocol } from "./route/media-protocol.js"
import { AIError, HttpOptions, MediaUsage, ProviderMetadata } from "./schema/index.js"
import { TranscriptionClient, Service } from "./transcription-client.js"
// ---------------------------------------------------------------------------
// Model
// ---------------------------------------------------------------------------
export type TranscriptionOptions = Record<string, unknown>
export type TranscriptionRoute<Options extends TranscriptionOptions = TranscriptionOptions> =
| MediaRoute.Route<TranscriptionRequestFor<Options>, TranscriptionResponse>
| MediaRoute.StreamRoute<TranscriptionRequestFor<Options>, TranscriptionEvent, TranscriptionResponse>
| MediaRoute.QueuedRoute<TranscriptionRequestFor<Options>, TranscriptionResponse>
export class TranscriptionModel<Options extends TranscriptionOptions = TranscriptionOptions> extends MediaModel<
TranscriptionRoute<Options>,
Options
> {
declare protected readonly _TranscriptionModel: void
static make<Options extends TranscriptionOptions = TranscriptionOptions>(
input: MediaModel.Input<TranscriptionRoute<Options>>,
) {
return new TranscriptionModel<Options>(input)
}
/** The number of type arguments selects the kind: `<Options>`, `<Options, Frame, State>`, or `<Options, Token>`. */
static fromRoute<Options extends TranscriptionOptions>(
route: TranscriptionModel.InlineRouteInput<Options>,
input: MediaRoute.ModelInput,
): TranscriptionModel<Options>
static fromRoute<Options extends TranscriptionOptions, Frame, State>(
route: TranscriptionModel.StreamRouteInput<Options, Frame, State>,
input: MediaRoute.ModelInput,
): TranscriptionModel<Options>
static fromRoute<Options extends TranscriptionOptions, Token>(
route: TranscriptionModel.QueuedRouteInput<Options, Token>,
input: MediaRoute.ModelInput,
): TranscriptionModel<Options>
static fromRoute<Options extends TranscriptionOptions, Frame, State, Token>(
route: TranscriptionModel.RouteInput<Options, Frame, State, Token>,
input: MediaRoute.ModelInput,
) {
const composed: TranscriptionRoute<Options> = isStreamInput(route)
? composeRoute((composition) => MediaRoute.stream({ ...composition, collect: collectResponse }), route, input)
: isQueuedInput(route)
? composeRoute(MediaRoute.queued, route, input)
: composeRoute(MediaRoute.inline, route, input)
return new TranscriptionModel<Options>({
id: input.id,
provider: route.provider,
http: input.http,
route: composed,
})
}
}
export namespace TranscriptionModel {
export type InlineRouteInput<Options extends TranscriptionOptions = TranscriptionOptions> = MediaModel.RouteInput<
TranscriptionRequestFor<Options>,
MediaProtocol.Inline<TranscriptionRequestFor<Options>, TranscriptionResponse>
>
export type StreamRouteInput<
Options extends TranscriptionOptions = TranscriptionOptions,
Frame = unknown,
State = unknown,
> = MediaModel.RouteInput<
MediaProtocol.Addressed<TranscriptionRequestFor<Options>>,
MediaProtocol.Streamed<TranscriptionRequestFor<Options>, TranscriptionEvent, Frame, State>
>
export type QueuedRouteInput<
Options extends TranscriptionOptions = TranscriptionOptions,
Token = unknown,
> = MediaModel.RouteInput<
TranscriptionRequestFor<Options>,
MediaProtocol.Queued<TranscriptionRequestFor<Options>, TranscriptionResponse, Token>
>
export type RouteInput<
Options extends TranscriptionOptions = TranscriptionOptions,
Frame = unknown,
State = unknown,
Token = unknown,
> = InlineRouteInput<Options> | StreamRouteInput<Options, Frame, State> | QueuedRouteInput<Options, Token>
}
const isStreamInput = <Options extends TranscriptionOptions, Frame, State, Token>(
route: TranscriptionModel.RouteInput<Options, Frame, State, Token>,
): route is TranscriptionModel.StreamRouteInput<Options, Frame, State> => route.protocol.kind === "stream"
const isQueuedInput = <Options extends TranscriptionOptions, Frame, State, Token>(
route: TranscriptionModel.RouteInput<Options, Frame, State, Token>,
): route is TranscriptionModel.QueuedRouteInput<Options, Token> => route.protocol.kind === "queued"
export const TranscriptionModelSchema = Schema.declare(
(value): value is TranscriptionModel => value instanceof TranscriptionModel,
{ expected: "Transcription.Model" },
)
// ---------------------------------------------------------------------------
// Request
// ---------------------------------------------------------------------------
export const TranscriptionTimestamps = Schema.Literals(["none", "segment", "word"])
export type TranscriptionTimestamps = Schema.Schema.Type<typeof TranscriptionTimestamps>
export class TranscriptionRequest extends Schema.Class<TranscriptionRequest>("Transcription.Request")({
model: TranscriptionModelSchema,
audio: Media.AssetSchema,
language: Schema.optional(Schema.String),
prompt: Schema.optional(Schema.String),
/** Routes that cannot produce the requested granularity fail typed; routes may return more than asked. */
timestamps: Schema.optional(TranscriptionTimestamps),
diarize: Schema.optional(Schema.Boolean),
speakers: Schema.optional(Schema.Int),
providerOptions: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
http: Schema.optional(HttpOptions),
}) {
declare protected readonly _TranscriptionRequest: void
}
export type TranscriptionRequestFor<Options extends TranscriptionOptions = TranscriptionOptions> = Omit<
TranscriptionRequest,
"model" | "providerOptions"
> & {
readonly model: TranscriptionModel<Options>
readonly providerOptions?: Options
}
export type TranscriptionModelOptions<Model> = Model extends TranscriptionModel<infer Options> ? Options : never
export type TranscriptionRequestInput<Model extends TranscriptionModel = TranscriptionModel> = Omit<
ConstructorParameters<typeof TranscriptionRequest>[0],
"model" | "providerOptions" | "http"
> & {
readonly model: Model
readonly providerOptions?: NoInfer<TranscriptionModelOptions<Model>>
readonly http?: HttpOptions.Input
}
// ---------------------------------------------------------------------------
// Response and events
// ---------------------------------------------------------------------------
/** Speaker labels are provider-native (`A`, `0`, `spk:0`, or a known speaker name). */
export const TranscriptionSegment = Schema.Struct({
text: Schema.String,
startSeconds: Schema.Number,
endSeconds: Schema.Number,
speaker: Schema.optional(Schema.String),
}).annotate({ identifier: "Transcription.Segment" })
export type TranscriptionSegment = Schema.Schema.Type<typeof TranscriptionSegment>
export const TranscriptionWord = Schema.Struct({
text: Schema.String,
startSeconds: Schema.Number,
endSeconds: Schema.Number,
speaker: Schema.optional(Schema.String),
confidence: Schema.optional(Schema.Number),
}).annotate({ identifier: "Transcription.Word" })
export type TranscriptionWord = Schema.Schema.Type<typeof TranscriptionWord>
const transcriptFields = {
text: Schema.String,
segments: Schema.optional(Schema.Array(TranscriptionSegment)),
words: Schema.optional(Schema.Array(TranscriptionWord)),
/** Provider-native language as detected or echoed (`en`, `english`, `en_us`), lowercased but not normalized. */
language: Schema.optional(Schema.String),
durationSeconds: Schema.optional(Schema.Number),
usage: Schema.optional(MediaUsage),
notices: Schema.optional(Schema.Array(Media.Notice)),
providerMetadata: Schema.optional(ProviderMetadata),
}
export class TranscriptionResponse extends Schema.Class<TranscriptionResponse>("Transcription.Response")(
transcriptFields,
) {}
export const TranscriptionTextDeltaEvent = Schema.Struct({
type: Schema.tag("text-delta"),
delta: Schema.String,
}).annotate({ identifier: "Transcription.Event.TextDelta" })
export const TranscriptionSegmentEvent = Schema.Struct({
type: Schema.tag("segment"),
segment: TranscriptionSegment,
}).annotate({ identifier: "Transcription.Event.Segment" })
export const TranscriptionFinishEvent = Schema.Struct({
type: Schema.tag("finish"),
...transcriptFields,
}).annotate({ identifier: "Transcription.Event.Finish" })
const transcriptionEventTagged = Schema.Union([
QueuedEvent,
ProgressEvent,
TranscriptionTextDeltaEvent,
TranscriptionSegmentEvent,
TranscriptionFinishEvent,
]).pipe(Schema.toTaggedUnion("type"))
export const TranscriptionEvent = Object.assign(transcriptionEventTagged, {
is: {
generationQueued: transcriptionEventTagged.guards["generation-queued"],
generationProgress: transcriptionEventTagged.guards["generation-progress"],
textDelta: transcriptionEventTagged.guards["text-delta"],
segment: transcriptionEventTagged.guards.segment,
finish: transcriptionEventTagged.guards.finish,
},
})
export type TranscriptionEvent = Schema.Schema.Type<typeof transcriptionEventTagged>
export const responseEvents = (response: TranscriptionResponse): ReadonlyArray<TranscriptionEvent> => [
TranscriptionFinishEvent.make({ ...response }),
]
const collectResponse = (events: ReadonlyArray<TranscriptionEvent>): Effect.Effect<TranscriptionResponse> => {
const finish = events.find(TranscriptionEvent.is.finish)
// Every transcription protocol's `finish` emits the terminal event or fails, so a completed stream always has one.
if (finish === undefined) return Effect.die(new Error("The transcription stream completed without a finish event"))
const { type: _type, ...transcript } = finish
return Effect.succeed(new TranscriptionResponse(transcript))
}
// ---------------------------------------------------------------------------
// Request-shaped call API
// ---------------------------------------------------------------------------
export function request<const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model>,
): TranscriptionRequestFor<TranscriptionModelOptions<Model>>
export function request(input: TranscriptionRequest): TranscriptionRequest
export function request(input: TranscriptionRequest | TranscriptionRequestInput) {
if (input instanceof TranscriptionRequest) return input
return new TranscriptionRequest({
...input,
http: input.http === undefined ? undefined : HttpOptions.make(input.http),
})
}
const requestEffect = (input: TranscriptionRequest | TranscriptionRequestInput) => tryRequest(() => request(input))
export function generate<const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model>,
options?: AwaitOptions,
): Effect.Effect<TranscriptionResponse, AIError, Service>
export function generate(
input: TranscriptionRequest,
options?: AwaitOptions,
): Effect.Effect<TranscriptionResponse, AIError, Service>
export function generate(input: TranscriptionRequest | TranscriptionRequestInput, options?: AwaitOptions) {
return requestEffect(input).pipe(Effect.flatMap((request) => TranscriptionClient.generate(request, options)))
}
export function stream<const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model>,
options?: AwaitOptions,
): Stream.Stream<TranscriptionEvent, AIError, Service>
export function stream(
input: TranscriptionRequest,
options?: AwaitOptions,
): Stream.Stream<TranscriptionEvent, AIError, Service>
export function stream(input: TranscriptionRequest | TranscriptionRequestInput, options?: AwaitOptions) {
return Stream.unwrap(requestEffect(input).pipe(Effect.map((request) => TranscriptionClient.stream(request, options))))
}
/** Inline and streaming routes fail with `UnsupportedOperation`. */
export function start<const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model>,
): Effect.Effect<Generation<TranscriptionResponse>, AIError, Service>
export function start(input: TranscriptionRequest): Effect.Effect<Generation<TranscriptionResponse>, AIError, Service>
export function start(input: TranscriptionRequest | TranscriptionRequestInput) {
return requestEffect(input).pipe(Effect.flatMap((request) => TranscriptionClient.start(request)))
}
export const resume = <Options extends TranscriptionOptions>(
model: TranscriptionModel<Options>,
token: unknown,
): Effect.Effect<Generation<TranscriptionResponse>, AIError, Service> => TranscriptionClient.resume(model, token)
export const Transcription = {
request,
generate,
stream,
start,
resume,
} as const
+16
View File
@@ -38,6 +38,7 @@ const EXTENSIONS: Readonly<Record<string, string>> = {
mp4: "video/mp4",
webm: "video/webm",
mp3: "audio/mpeg",
m4a: "audio/mp4",
wav: "audio/wav",
ogg: "audio/ogg",
flac: "audio/flac",
@@ -49,3 +50,18 @@ const EXTENSIONS: Readonly<Record<string, string>> = {
export const extensionMediaType = (path: string): string | undefined =>
EXTENSIONS[path.slice(path.lastIndexOf(".") + 1).toLowerCase()]
const EXTENSION_ALIASES: Readonly<Record<string, string>> = {
"audio/mp3": "mp3",
"audio/m4a": "m4a",
"audio/x-m4a": "m4a",
"audio/webm": "webm",
"audio/wave": "wav",
"audio/x-wav": "wav",
"audio/x-flac": "flac",
}
export const mediaTypeExtension = (mediaType: string): string | undefined => {
const type = mediaType.split(";", 1)[0].trim().toLowerCase()
return EXTENSION_ALIASES[type] ?? Object.entries(EXTENSIONS).find(([, known]) => known === type)?.[0]
}
+4 -12
View File
@@ -1,9 +1,8 @@
import { Context, Effect, Layer, Stream } from "effect"
import type { AwaitOptions, Generation } from "./generation.js"
import { resultEvents, type AwaitOptions, type Generation } from "./generation.js"
import { RequestExecutor } from "./route/executor.js"
import type { AIError } from "./schema/index.js"
import {
isObservation,
responseEvents,
type VideoEvent,
type VideoModel,
@@ -69,15 +68,6 @@ export const stream = <Options extends VideoOptions>(
}),
)
/** Status observations until the first terminal one, then the fetched result expanded into events. */
const events = (generation: Generation<VideoResponse>, options: AwaitOptions | undefined) =>
generation
.events(options)
.pipe(
Stream.filter(isObservation),
Stream.concat(Stream.fromIterableEffect(Effect.map(generation.result(), responseEvents))),
)
export const layer: Layer.Layer<Service, never, RequestExecutor.Service> = Layer.effect(
Service,
Effect.gen(function* () {
@@ -89,7 +79,9 @@ export const layer: Layer.Layer<Service, never, RequestExecutor.Service> = Layer
resume: (model, token) => model.route.resume(model, token, executor.execute),
generate: (request, options) => start(request).pipe(Effect.flatMap((generation) => generation.await(options))),
stream: (request, options) =>
Stream.unwrap(start(request).pipe(Effect.map((generation) => events(generation, options)))),
Stream.unwrap(
start(request).pipe(Effect.map((generation) => resultEvents(generation, responseEvents, options))),
),
})
}),
)
+2 -18
View File
@@ -1,5 +1,5 @@
import { Effect, Schema, Stream } from "effect"
import { Generation, type AwaitOptions, type Event as GenerationEvent } from "./generation.js"
import { Generation, ProgressEvent, QueuedEvent, type AwaitOptions } from "./generation.js"
import { Media } from "./media.js"
import { MediaModel, composeRoute, tryRequest } from "./media-model.js"
import { MediaRoute } from "./route/media.js"
@@ -123,18 +123,6 @@ export class VideoResponse extends Schema.Class<VideoResponse>("Video.Response")
}
}
export const VideoQueuedEvent = Schema.Struct({
type: Schema.tag("generation-queued"),
id: Schema.String,
position: Schema.optional(Schema.Number),
}).annotate({ identifier: "Video.Event.Queued" })
export const VideoProgressEvent = Schema.Struct({
type: Schema.tag("generation-progress"),
id: Schema.String,
progress: Schema.optional(Schema.Number),
}).annotate({ identifier: "Video.Event.Progress" })
export const VideoOutputEvent = Schema.Struct({
type: Schema.tag("video"),
index: Schema.Number,
@@ -148,7 +136,7 @@ export const VideoFinishEvent = Schema.Struct({
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "Video.Event.Finish" })
const videoEventTagged = Schema.Union([VideoQueuedEvent, VideoProgressEvent, VideoOutputEvent, VideoFinishEvent]).pipe(
const videoEventTagged = Schema.Union([QueuedEvent, ProgressEvent, VideoOutputEvent, VideoFinishEvent]).pipe(
Schema.toTaggedUnion("type"),
)
export const VideoEvent = Object.assign(videoEventTagged, {
@@ -171,10 +159,6 @@ export const responseEvents = (response: VideoResponse): ReadonlyArray<VideoEven
}),
]
/** A status observation as a video event; the terminal observation is replaced by the result events, so `none`. */
export const isObservation = (event: GenerationEvent): event is Extract<VideoEvent, { type: "generation-queued" | "generation-progress" }> =>
event.type !== "generation-finished"
// ---------------------------------------------------------------------------
// Request-shaped call API
// ---------------------------------------------------------------------------
+5
View File
@@ -17,6 +17,7 @@ import {
import { Route, Protocol, WebSocketTransport } from "@opencode/ai/route"
import { Provider as ProviderSubpath } from "@opencode/ai/provider"
import {
AssemblyAI,
Baseten,
Cartesia,
CloudflareAIGateway,
@@ -122,6 +123,10 @@ describe("public exports", () => {
expect(ElevenLabs.configure({ apiKey: "fixture" }).speech("eleven_flash_v2_5").route.id).toBe("elevenlabs-speech")
expect(Cartesia.configure({ apiKey: "fixture" }).speech("sonic-3").route.id).toBe("cartesia-speech")
expect(Deepgram.configure({ apiKey: "fixture" }).speech("aura-2-thalia-en").route.id).toBe("deepgram-speech")
expect(OpenAI.configure({ apiKey: "fixture" }).transcription("gpt-transcribe").route.kind).toBe("stream")
expect(Google.configure({ apiKey: "fixture" }).transcription("gemini-3.5-transcribe").route.kind).toBe("stream")
expect(Deepgram.configure({ apiKey: "fixture" }).transcription("nova-3").route.kind).toBe("inline")
expect(AssemblyAI.configure({ apiKey: "fixture" }).transcription("universal-3-5-pro").route.kind).toBe("queued")
})
test("protocol barrels expose supported low-level routes", () => {
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,98 @@
{
"version": 1,
"metadata": {
"tags": [
"prefix:assemblyai-transcription",
"provider:assemblyai",
"protocol:assemblyai-transcription"
],
"name": "assemblyai-transcription/uploads-submits-and-polls-a-transcript",
"recordedAt": "2026-09-23T14:03:07.504Z"
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.eu.assemblyai.com/v2/upload",
"headers": {
"content-type": "application/octet-stream"
},
"body": "[audio]"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json"
},
"body": "{\"upload_url\":\"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\"}\n"
}
},
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.eu.assemblyai.com/v2/transcript",
"headers": {
"content-type": "application/json"
},
"body": "{\"audio_url\":\"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\",\"speech_models\":[\"universal-3-5-pro\"],\"language_detection\":true,\"speaker_labels\":true}"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json; charset=UTF-8"
},
"body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": null, \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"processing\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": null, \"words\": null, \"utterances\": null, \"confidence\": null, \"audio_duration\": null, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": null, \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {}, \"iab_categories_result\": {}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": null, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": null, \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}"
}
},
{
"transport": "http",
"request": {
"method": "GET",
"url": "https://api.eu.assemblyai.com/v2/transcript/a1de21ad-a178-4c28-b8f0-4b0334b72126",
"headers": {},
"body": ""
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json; charset=UTF-8"
},
"body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": null, \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"processing\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": null, \"words\": null, \"utterances\": null, \"confidence\": null, \"audio_duration\": null, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": [], \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {}, \"iab_categories_result\": {}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": null, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": null, \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}"
}
},
{
"transport": "http",
"request": {
"method": "GET",
"url": "https://api.eu.assemblyai.com/v2/transcript/a1de21ad-a178-4c28-b8f0-4b0334b72126",
"headers": {},
"body": ""
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json; charset=UTF-8"
},
"body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": \"en\", \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"completed\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": \"Hello from Open Code.\", \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}], \"utterances\": [{\"speaker\": \"A\", \"text\": \"Hello from Open Code.\", \"confidence\": 0.85656375, \"start\": 445, \"end\": 1455, \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}]}], \"confidence\": 0.85656375, \"audio_duration\": 2, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": [], \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"iab_categories_result\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": 0.9934, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": \"universal-3-5-pro\", \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}"
}
},
{
"transport": "http",
"request": {
"method": "GET",
"url": "https://api.eu.assemblyai.com/v2/transcript/a1de21ad-a178-4c28-b8f0-4b0334b72126",
"headers": {},
"body": ""
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json; charset=UTF-8"
},
"body": "{\"id\": \"a1de21ad-a178-4c28-b8f0-4b0334b72126\", \"language_model\": \"assemblyai_default\", \"acoustic_model\": \"assemblyai_default\", \"language_code\": \"en\", \"speech_understanding\": null, \"translated_texts\": null, \"status\": \"completed\", \"audio_url\": \"https://cdn.assemblyai.com/upload/f057ab4f09d5f28527be4b830b3d92f8bf6a97e5f854837cf22efbf2d09b5182/ed6a04e2-e34c-4a70-89da-b1ec34d2fb5f\", \"text\": \"Hello from Open Code.\", \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}], \"utterances\": [{\"speaker\": \"A\", \"text\": \"Hello from Open Code.\", \"confidence\": 0.85656375, \"start\": 445, \"end\": 1455, \"words\": [{\"text\": \"Hello\", \"start\": 445, \"end\": 633, \"confidence\": 0.8883743, \"speaker\": \"A\"}, {\"text\": \"from\", \"start\": 719, \"end\": 890, \"confidence\": 0.9978282, \"speaker\": \"A\"}, {\"text\": \"Open\", \"start\": 959, \"end\": 1147, \"confidence\": 0.9486656, \"speaker\": \"A\"}, {\"text\": \"Code.\", \"start\": 1215, \"end\": 1455, \"confidence\": 0.5913869, \"speaker\": \"A\"}]}], \"confidence\": 0.85656375, \"audio_duration\": 2, \"punctuate\": true, \"format_text\": true, \"dual_channel\": false, \"webhook_url\": null, \"webhook_status_code\": null, \"webhook_auth\": false, \"webhook_auth_header_name\": null, \"speed_boost\": false, \"auto_highlights_result\": null, \"auto_highlights\": false, \"audio_start_from\": null, \"audio_end_at\": null, \"word_boost\": [], \"boost_param\": null, \"prompt\": null, \"keyterms_prompt\": [], \"filter_profanity\": false, \"redact_pii\": false, \"redact_pii_audio\": false, \"redact_pii_audio_quality\": null, \"redact_pii_audio_options\": null, \"redact_pii_policies\": null, \"redact_pii_sub\": null, \"redact_static_entities\": null, \"speaker_labels\": true, \"speaker_options\": {\"min_speakers_expected\": null, \"max_speakers_expected\": null, \"advanced_speaker_segmentation\": true}, \"content_safety\": false, \"iab_categories\": false, \"content_safety_labels\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"iab_categories_result\": {\"status\": \"unavailable\", \"results\": [], \"summary\": {}}, \"language_detection\": true, \"language_detection_options\": {\"expected_languages\": null, \"fallback_language\": null, \"code_switching\": null, \"code_switching_confidence_threshold\": null, \"on_low_language_confidence\": null, \"on_no_speech_detected\": \"fallback\", \"swiss_german\": null, \"language_detection_model\": null, \"localization\": null}, \"language_detection_results\": null, \"language_confidence_threshold\": null, \"language_confidence\": 0.9934, \"custom_spelling\": null, \"throttled\": false, \"auto_chapters\": false, \"summarization\": false, \"summary_type\": null, \"summary_model\": null, \"custom_topics\": false, \"topics\": [], \"speech_threshold\": null, \"speech_model\": null, \"speech_models\": [\"universal-3-5-pro\"], \"speech_model_used\": \"universal-3-5-pro\", \"temperature\": null, \"remove_audio_tags\": \"all\", \"chapters\": null, \"disfluencies\": false, \"entity_detection\": false, \"sentiment_analysis\": false, \"sentiment_analysis_results\": null, \"entities\": null, \"speakers_expected\": null, \"summary\": null, \"custom_topics_results\": null, \"is_deleted\": null, \"multichannel\": null, \"project_id\": 2040069, \"token_id\": 2077829}"
}
}
]
}
@@ -0,0 +1,32 @@
{
"version": 1,
"metadata": {
"tags": [
"prefix:deepgram-transcription",
"provider:deepgram",
"protocol:deepgram-transcription"
],
"name": "deepgram-transcription/splits-utterances-at-speaker-changes",
"recordedAt": "2026-09-23T13:46:35.886Z"
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.deepgram.com/v1/listen?model=nova-3&smart_format=true&detect_language=true&diarize_model=latest&utterances=true",
"headers": {
"content-type": "audio/mpeg"
},
"body": "[audio]"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json"
},
"body": "{\"metadata\":{\"transaction_key\":\"deprecated\",\"request_id\":\"01a0ce84-a81d-7161-8505-c08d5e0c25bb\",\"sha256\":\"12e9ccb9d346c28b071c2fab7ce48556d0cc115af6850aacb338e7548807cb80\",\"created\":\"2026-09-23T13:46:35.566Z\",\"duration\":3.552625,\"channels\":1,\"models\":[\"2187e11a-3532-4498-b076-81fa530bdd49\"],\"model_info\":{\"2187e11a-3532-4498-b076-81fa530bdd49\":{\"name\":\"general-nova-3\",\"version\":\"2025-07-31.0\",\"arch\":\"nova-3\"}},\"diarize_info\":{\"model_uuid\":\"6946b038-8264-4505-acf7-9822c009fecb\",\"arch\":\"v2\"}},\"results\":{\"channels\":[{\"alternatives\":[{\"transcript\":\"Did the release ship? Yes. It shipped this morning.\",\"confidence\":1.0,\"words\":[{\"word\":\"did\",\"start\":0.08,\"end\":0.48,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Did\"},{\"word\":\"the\",\"start\":0.48,\"end\":0.64,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"the\"},{\"word\":\"release\",\"start\":0.64,\"end\":0.96,\"confidence\":0.99902344,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"release\"},{\"word\":\"ship\",\"start\":0.96,\"end\":1.52,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"ship?\"},{\"word\":\"yes\",\"start\":1.5999999,\"end\":2.32,\"confidence\":0.9980469,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"Yes.\"},{\"word\":\"it\",\"start\":2.32,\"end\":2.56,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"It\"},{\"word\":\"shipped\",\"start\":2.56,\"end\":2.8799999,\"confidence\":0.9707031,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"shipped\"},{\"word\":\"this\",\"start\":2.8799999,\"end\":3.1999998,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"this\"},{\"word\":\"morning\",\"start\":3.1999998,\"end\":3.52,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"morning.\"}],\"paragraphs\":{\"transcript\":\"\\nSpeaker 0: Did the release ship?\\n\\nSpeaker 1: Yes. It shipped this morning.\",\"paragraphs\":[{\"sentences\":[{\"text\":\"Did the release ship?\",\"start\":0.08,\"end\":1.52}],\"speaker\":0,\"num_words\":4,\"start\":0.08,\"end\":1.52},{\"sentences\":[{\"text\":\"Yes.\",\"start\":1.5999999,\"end\":2.32},{\"text\":\"It shipped this morning.\",\"start\":2.32,\"end\":3.52}],\"speaker\":1,\"num_words\":5,\"start\":1.5999999,\"end\":3.52}]}}],\"detected_language\":\"en\",\"language_confidence\":0.917855}],\"utterances\":[{\"start\":0.08,\"end\":3.52,\"confidence\":0.99641925,\"channel\":0,\"transcript\":\"Did the release ship? Yes. It shipped this morning.\",\"words\":[{\"word\":\"did\",\"start\":0.08,\"end\":0.48,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Did\"},{\"word\":\"the\",\"start\":0.48,\"end\":0.64,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"the\"},{\"word\":\"release\",\"start\":0.64,\"end\":0.96,\"confidence\":0.99902344,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"release\"},{\"word\":\"ship\",\"start\":0.96,\"end\":1.52,\"confidence\":1.0,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"ship?\"},{\"word\":\"yes\",\"start\":1.5999999,\"end\":2.32,\"confidence\":0.9980469,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"Yes.\"},{\"word\":\"it\",\"start\":2.32,\"end\":2.56,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"It\"},{\"word\":\"shipped\",\"start\":2.56,\"end\":2.8799999,\"confidence\":0.9707031,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"shipped\"},{\"word\":\"this\",\"start\":2.8799999,\"end\":3.1999998,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"this\"},{\"word\":\"morning\",\"start\":3.1999998,\"end\":3.52,\"confidence\":1.0,\"speaker\":1,\"speaker_confidence\":0.98,\"punctuated_word\":\"morning.\"}],\"speaker\":1,\"id\":\"67ba8055-e541-4089-abfa-0163558083cc\"}]}}"
}
}
]
}
@@ -0,0 +1,50 @@
{
"version": 1,
"metadata": {
"tags": [
"prefix:deepgram-transcription",
"provider:deepgram",
"protocol:deepgram-transcription"
],
"name": "deepgram-transcription/transcribes-raw-audio-with-speakers-and-word-timestamps",
"recordedAt": "2026-09-23T12:32:18.578Z"
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.deepgram.com/v1/listen?model=nova-3&smart_format=true&detect_language=true&diarize_model=latest&utterances=true",
"headers": {
"content-type": "audio/mpeg"
},
"body": "[audio]"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json"
},
"body": "{\"metadata\":{\"transaction_key\":\"deprecated\",\"request_id\":\"01a0ce40-a44f-73b0-9d99-cdfc4f184268\",\"sha256\":\"95495edaebf6b10b2877de8c7b9eb7372de315c105fa8e73ed656ef2ec28ca02\",\"created\":\"2026-09-23T12:32:18.036Z\",\"duration\":1.5411875,\"channels\":1,\"models\":[\"2187e11a-3532-4498-b076-81fa530bdd49\"],\"model_info\":{\"2187e11a-3532-4498-b076-81fa530bdd49\":{\"name\":\"general-nova-3\",\"version\":\"2025-07-31.0\",\"arch\":\"nova-3\"}},\"diarize_info\":{\"model_uuid\":\"6946b038-8264-4505-acf7-9822c009fecb\",\"arch\":\"v2\"}},\"results\":{\"channels\":[{\"alternatives\":[{\"transcript\":\"Hello from OpenCode.\",\"confidence\":0.96972656,\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"paragraphs\":{\"transcript\":\"\\nSpeaker 0: Hello from OpenCode.\",\"paragraphs\":[{\"sentences\":[{\"text\":\"Hello from OpenCode.\",\"start\":0.16,\"end\":1.4399999}],\"speaker\":0,\"num_words\":3,\"start\":0.16,\"end\":1.4399999}]}}],\"detected_language\":\"en\",\"language_confidence\":0.9998003}],\"utterances\":[{\"start\":0.16,\"end\":1.4399999,\"confidence\":0.96875,\"channel\":0,\"transcript\":\"Hello from OpenCode.\",\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"speaker\":0,\"id\":\"43d717d5-ca00-48c1-a39c-381be0b1dad2\"}]}}"
}
},
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.deepgram.com/v1/listen?model=nova-3&smart_format=true&detect_language=true&diarize_model=latest&utterances=true",
"headers": {
"content-type": "audio/mpeg"
},
"body": "[audio]"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json"
},
"body": "{\"metadata\":{\"transaction_key\":\"deprecated\",\"request_id\":\"01a0ce40-a609-78c2-8a9d-806e3ff8db0b\",\"sha256\":\"95495edaebf6b10b2877de8c7b9eb7372de315c105fa8e73ed656ef2ec28ca02\",\"created\":\"2026-09-23T12:32:18.478Z\",\"duration\":1.5411875,\"channels\":1,\"models\":[\"2187e11a-3532-4498-b076-81fa530bdd49\"],\"model_info\":{\"2187e11a-3532-4498-b076-81fa530bdd49\":{\"name\":\"general-nova-3\",\"version\":\"2025-07-31.0\",\"arch\":\"nova-3\"}},\"diarize_info\":{\"model_uuid\":\"6946b038-8264-4505-acf7-9822c009fecb\",\"arch\":\"v2\"}},\"results\":{\"channels\":[{\"alternatives\":[{\"transcript\":\"Hello from OpenCode.\",\"confidence\":0.96972656,\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"paragraphs\":{\"transcript\":\"\\nSpeaker 0: Hello from OpenCode.\",\"paragraphs\":[{\"sentences\":[{\"text\":\"Hello from OpenCode.\",\"start\":0.16,\"end\":1.4399999}],\"speaker\":0,\"num_words\":3,\"start\":0.16,\"end\":1.4399999}]}}],\"detected_language\":\"en\",\"language_confidence\":0.9998003}],\"utterances\":[{\"start\":0.16,\"end\":1.4399999,\"confidence\":0.96875,\"channel\":0,\"transcript\":\"Hello from OpenCode.\",\"words\":[{\"word\":\"hello\",\"start\":0.16,\"end\":0.64,\"confidence\":0.9892578,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"Hello\"},{\"word\":\"from\",\"start\":0.64,\"end\":0.88,\"confidence\":0.96972656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"from\"},{\"word\":\"opencode\",\"start\":0.88,\"end\":1.4399999,\"confidence\":0.9472656,\"speaker\":0,\"speaker_confidence\":0.98,\"punctuated_word\":\"OpenCode.\"}],\"speaker\":0,\"id\":\"6f46ea75-616b-4209-b644-aae3e8f1e3c0\"}]}}"
}
}
]
}
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,32 @@
{
"version": 1,
"metadata": {
"tags": [
"prefix:openai-transcription",
"provider:openai",
"protocol:openai-transcription"
],
"name": "openai-transcription/diarizes-speakers-into-segments",
"recordedAt": "2026-09-23T12:32:01.274Z"
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/audio/transcriptions",
"headers": {
"content-type": "multipart/form-data; boundary=----WebKitFormBoundaryf476753a62264fa88f2d753a7e85c3a4"
},
"body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\ngpt-4o-transcribe-diarize\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"response_format\"\r\n\r\ndiarized_json\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"chunking_strategy\"\r\n\r\nauto\r\n--BOUNDARY--\r\n"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json"
},
"body": "{\"text\":\"Hello from OpenCode!\",\"task\":\"transcribe\",\"duration\":1.54125,\"segments\":[{\"type\":\"transcript.text.segment\",\"text\":\" Hello from OpenCode!\",\"speaker\":\"A\",\"start\":0.30000000000000004,\"end\":1.3499999999999999,\"id\":\"seg_0\"}],\"usage\":{\"type\":\"tokens\",\"total_tokens\":107,\"input_tokens\":15,\"input_token_details\":{\"text_tokens\":0,\"audio_tokens\":15},\"output_tokens\":92}}"
}
}
]
}
@@ -0,0 +1,32 @@
{
"version": 1,
"metadata": {
"tags": [
"prefix:openai-transcription",
"provider:openai",
"protocol:openai-transcription"
],
"name": "openai-transcription/returns-whisper-word-timestamps",
"recordedAt": "2026-09-23T12:32:02.972Z"
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/audio/transcriptions",
"headers": {
"content-type": "multipart/form-data; boundary=----WebKitFormBoundarydfe85869a7d6448994e94f2f41d9ce55"
},
"body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\nwhisper-1\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"response_format\"\r\n\r\nverbose_json\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"timestamp_granularities[]\"\r\n\r\nword\r\n--BOUNDARY--\r\n"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json"
},
"body": "{\"task\":\"transcribe\",\"language\":\"english\",\"duration\":1.5399999618530273,\"text\":\"Hello from OpenCode!\",\"words\":[{\"word\":\"Hello\",\"start\":0.0,\"end\":0.6200000047683716},{\"word\":\"from\",\"start\":0.6200000047683716,\"end\":0.8799999952316284},{\"word\":\"OpenCode\",\"start\":0.8799999952316284,\"end\":1.399999976158142}],\"usage\":{\"type\":\"duration\",\"seconds\":2}}"
}
}
]
}
@@ -0,0 +1,32 @@
{
"version": 1,
"metadata": {
"tags": [
"prefix:openai-transcription",
"provider:openai",
"protocol:openai-transcription"
],
"name": "openai-transcription/streams-text-deltas",
"recordedAt": "2026-09-23T12:31:59.149Z"
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/audio/transcriptions",
"headers": {
"content-type": "multipart/form-data; boundary=----WebKitFormBoundarya44da5204f47452ea1ef8dec145321dc"
},
"body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\ngpt-4o-mini-transcribe\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"stream\"\r\n\r\ntrue\r\n--BOUNDARY--\r\n"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"type\":\"transcript.text.delta\",\"delta\":\"Hello\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\" from\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\" Open\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\"Code\"}\r\n\r\ndata: {\"type\":\"transcript.text.delta\",\"delta\":\".\"}\r\n\r\ndata: {\"type\":\"transcript.text.done\",\"text\":\"Hello from OpenCode.\",\"usage\":{\"type\":\"tokens\",\"total_tokens\":22,\"input_tokens\":15,\"input_token_details\":{\"text_tokens\":0,\"audio_tokens\":15},\"output_tokens\":7}}\r\n\r\ndata: [DONE]\r\n\r\n"
}
}
]
}
@@ -0,0 +1,32 @@
{
"version": 1,
"metadata": {
"tags": [
"prefix:openai-transcription",
"provider:openai",
"protocol:openai-transcription"
],
"name": "openai-transcription/transcribes-with-a-language-hint",
"recordedAt": "2026-09-23T12:31:56.657Z"
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/audio/transcriptions",
"headers": {
"content-type": "multipart/form-data; boundary=----WebKitFormBoundary484bf529b0064c43ac4a965c37433c9e"
},
"body": "--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n[audio]\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\ngpt-transcribe\r\n--BOUNDARY\r\nContent-Disposition: form-data; name=\"languages[]\"\r\n\r\nen\r\n--BOUNDARY--\r\n"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/json"
},
"body": "{\"text\":\"Hello from OpenCode.\",\"languages\":[{\"code\":\"en\"}],\"usage\":{\"type\":\"duration\",\"seconds\":2}}"
}
}
]
}
+51 -3
View File
@@ -1,10 +1,10 @@
import { describe, expect, test } from "bun:test"
import { Effect, Layer } from "effect"
import { HttpClientRequest } from "effect/unstable/http"
import { AIError, LLMEvent, Media, SpeechEvent } from "../src/index.js"
import { AIError, LLMEvent, Media, SpeechEvent, TranscriptionEvent } from "../src/index.js"
import { RequestExecutor } from "../src/route.js"
import { AI } from "../src/promise.js"
import { OpenAI, Runway } from "../src/providers.js"
import { AssemblyAI, OpenAI, Runway } from "../src/providers.js"
import { handlerLayer } from "./lib/http.js"
import { sseEvents } from "./lib/sse.js"
@@ -18,7 +18,8 @@ const chatBody = sseEvents(
/**
* Executor layer that answers chat completions with SSE text, image generations with one base64 PNG, Runway video
* tasks with a queued submission that succeeds on the second poll, and speech with raw audio or SSE audio deltas.
* tasks with a queued submission that succeeds on the second poll, speech with raw audio or SSE audio deltas, OpenAI
* transcription with JSON or SSE text deltas, and AssemblyAI transcripts that complete on the first poll.
*/
const executor = (seen: Array<string>) =>
RequestExecutor.layer.pipe(
@@ -44,6 +45,27 @@ const executor = (seen: Array<string>) =>
{ headers: { "content-type": "text/event-stream" } },
)
: input.respond(Uint8Array.from([1, 2, 3]), { headers: { "content-type": "audio/pcm" } })
if (web.url.endsWith("/audio/transcriptions"))
return input.text.includes('name="stream"')
? input.respond(
sseEvents(
{ type: "transcript.text.delta", delta: "Hello" },
{ type: "transcript.text.delta", delta: " there." },
{ type: "transcript.text.done", text: "Hello there." },
),
{ headers: { "content-type": "text/event-stream" } },
)
: input.respond(JSON.stringify({ text: "Hello there." }), {
headers: { "content-type": "application/json" },
})
if (web.url.endsWith("/v2/transcript"))
return input.respond(JSON.stringify({ id: "tr_1", status: "queued" }), {
headers: { "content-type": "application/json" },
})
if (web.url.endsWith("/v2/transcript/tr_1"))
return input.respond(JSON.stringify({ id: "tr_1", status: "completed", text: "Hello there." }), {
headers: { "content-type": "application/json" },
})
if (web.url.endsWith("/text_to_video"))
return input.respond(JSON.stringify({ id: "task_1" }), { headers: { "content-type": "application/json" } })
if (web.url.endsWith("/tasks/task_1")) {
@@ -134,6 +156,32 @@ describe("AI promise client", () => {
await ai.dispose()
})
test("generates, streams, and starts transcriptions over the same runtime", async () => {
const ai = AI.make({ layer: executor([]) })
const audio = Media.url("https://audio.test/hello.mp3")
const bytes = Media.bytes(Uint8Array.from([0x49, 0x44, 0x33]), "audio/mpeg")
expect(
(await ai.transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio: bytes })).text,
).toBe("Hello there.")
const deltas: Array<string> = []
for await (const event of ai.transcription.stream({
model: openai.transcription("gpt-4o-mini-transcribe"),
audio: bytes,
}))
if (TranscriptionEvent.is.textDelta(event)) deltas.push(event.delta)
expect(deltas).toEqual(["Hello", " there."])
const model = AssemblyAI.configure({ apiKey: "test", baseURL: "https://assemblyai.test" }).transcription(
"universal-2",
)
const generation = await ai.transcription.start({ model, audio })
expect(generation.token).toEqual({ transcriptID: "tr_1" })
expect((await generation.await({ poll: { interval: 10 } })).text).toBe("Hello there.")
await ai.dispose()
})
test("generates and streams speech over the same runtime", async () => {
const ai = AI.make({ layer: executor([]) })
const model = openai.speech("gpt-4o-mini-tts")
@@ -0,0 +1,39 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Transcription } from "../../src/index.js"
import { AssemblyAI } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording } from "./transcription-recording.js"
import { queuedPoll } from "./queued-recording.js"
// Recorded against the EU region: the US endpoint was unreachable from the recording network. Same API and shapes.
const model = AssemblyAI.configure({
apiKey: process.env.ASSEMBLYAI_API_KEY ?? "fixture",
baseURL: "https://api.eu.assemblyai.com",
}).transcription("universal-3-5-pro")
const recorded = recordedTests({
prefix: "assemblyai-transcription",
provider: "assemblyai",
protocol: "assemblyai-transcription",
requires: ["ASSEMBLYAI_API_KEY"],
options: audioRecording,
})
describe("AssemblyAI Transcription recorded", () => {
recorded.effect(
"uploads, submits, and polls a transcript",
() =>
Effect.gen(function* () {
const generation = yield* Transcription.start({ model, audio: yield* audio, diarize: true })
const response = yield* generation.await({ poll: queuedPoll })
expect(response.text).toMatch(TRANSCRIPT)
expect(response.words?.[0]).toMatchObject({ text: "Hello", speaker: "A" })
expect(response.words?.every((word) => word.startSeconds < 5 && word.endSeconds < 5)).toBe(true)
expect(response.segments?.[0]?.speaker).toBe("A")
expect(response.usage).toEqual({ type: "seconds", seconds: expect.any(Number) })
}),
{ timeout: 15 * 60 * 1000 },
)
})
@@ -0,0 +1,47 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Transcription } from "../../src/index.js"
import { Deepgram } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js"
const model = Deepgram.configure({ apiKey: process.env.DEEPGRAM_API_KEY ?? "fixture" }).transcription("nova-3")
const recorded = recordedTests({
prefix: "deepgram-transcription",
provider: "deepgram",
protocol: "deepgram-transcription",
requires: ["DEEPGRAM_API_KEY"],
options: audioRecording,
})
describe("Deepgram Transcription recorded", () => {
recorded.effect("transcribes raw audio with speakers and word timestamps", () =>
Effect.gen(function* () {
const request = Transcription.request({ model, audio: yield* audio, diarize: true, timestamps: "word" })
const response = yield* Transcription.generate(request)
expect(response.text).toMatch(TRANSCRIPT)
expect(response.words?.map((word) => word.text)).toEqual(["Hello", "from", "OpenCode."])
expect(response.words?.every((word) => word.speaker === "0" && word.confidence !== undefined)).toBe(true)
expect(response.segments).toEqual([
{ text: response.text, startSeconds: expect.any(Number), endSeconds: expect.any(Number), speaker: "0" },
])
expect(response.language).toBe("en")
expect(response.usage).toEqual({ type: "seconds", seconds: response.durationSeconds })
const events = Array.from(yield* Stream.runCollect(Transcription.stream(request)))
expect(events.map((event) => event.type)).toEqual(["finish"])
}),
)
recorded.effect("splits utterances at speaker changes", () =>
Effect.gen(function* () {
const response = yield* Transcription.generate({ model, audio: yield* dialog, diarize: true })
expect(response.segments?.map((segment) => segment.speaker)).toEqual(["0", "1"])
expect(response.segments?.[0].text).toMatch(/release ship\?$/)
expect(response.segments?.[1].text).toMatch(/shipped this morning\.$/)
}),
)
})
@@ -3,7 +3,7 @@ import { Effect } from "effect"
import { Video } from "../../src/index.js"
import { Fal } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { videoPoll } from "./video-recording.js"
import { queuedPoll } from "./queued-recording.js"
const model = Fal.configure({
apiKey: process.env.FAL_KEY ?? "fixture",
@@ -29,7 +29,7 @@ describe("fal Video recorded", () => {
resolution: "720p",
providerOptions: { duration: "4s" },
},
{ poll: videoPoll },
{ poll: queuedPoll },
)
expect(response.videos).toHaveLength(1)
@@ -0,0 +1,54 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Transcription, TranscriptionEvent } from "../../src/index.js"
import { Google } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js"
const model = Google.configure({ apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY ?? "fixture" }).transcription(
"gemini-3.5-transcribe",
)
const recorded = recordedTests({
prefix: "google-transcription",
provider: "google",
protocol: "google-transcription",
requires: ["GOOGLE_GENERATIVE_AI_API_KEY"],
options: audioRecording,
})
describe("Google Transcription recorded", () => {
recorded.effect("transcribes with speakers and word timestamps", () =>
Effect.gen(function* () {
const response = yield* Transcription.generate({ model, audio: yield* audio, diarize: true, language: "en-US" })
expect(response.text).toMatch(TRANSCRIPT)
expect(response.words?.length).toBeGreaterThan(2)
expect(response.words?.every((word) => word.endSeconds > word.startSeconds)).toBe(true)
expect(response.segments).toEqual([
{
text: response.text,
startSeconds: response.words?.[0]?.startSeconds,
endSeconds: response.words?.at(-1)?.endSeconds,
speaker: expect.any(String),
},
])
expect(response.usage?.type).toBe("tokens")
}),
)
recorded.effect("streams speaker turns", () =>
Effect.gen(function* () {
const events = Array.from(
yield* Stream.runCollect(Transcription.stream({ model, audio: yield* dialog, diarize: true })),
)
const finish = events.at(-1)
if (finish === undefined || !TranscriptionEvent.is.finish(finish)) throw new Error("Expected a finish event")
expect(finish.text).toMatch(/release ship\? Yes, it shipped this morning\.$/)
const speakers = events.filter(TranscriptionEvent.is.segment).map((event) => event.segment.speaker)
expect(speakers).toHaveLength(2)
expect(new Set(speakers).size).toBe(2)
}),
)
})
@@ -3,7 +3,7 @@ import { Effect } from "effect"
import { Video } from "../../src/index.js"
import { Google } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { videoPoll } from "./video-recording.js"
import { queuedPoll } from "./queued-recording.js"
const model = Google.configure({
apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY ?? "fixture",
@@ -28,7 +28,7 @@ describe("Google Veo recorded", () => {
aspectRatio: "16:9",
durationSeconds: 4,
},
{ poll: videoPoll },
{ poll: queuedPoll },
)
expect(response.videos).toHaveLength(1)
@@ -0,0 +1,86 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Transcription, TranscriptionEvent } from "../../src/index.js"
import { OpenAI } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording } from "./transcription-recording.js"
const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY ?? "fixture" })
const recorded = recordedTests({
prefix: "openai-transcription",
provider: "openai",
protocol: "openai-transcription",
requires: ["OPENAI_API_KEY"],
options: audioRecording,
})
describe("OpenAI Transcription recorded", () => {
recorded.effect("transcribes with a language hint", () =>
Effect.gen(function* () {
const response = yield* Transcription.generate({
model: openai.transcription("gpt-transcribe"),
audio: yield* audio,
language: "en",
})
expect(response.text).toMatch(TRANSCRIPT)
expect(response.language).toBe("en")
expect(response.usage?.type).toBeDefined()
}),
)
recorded.effect("streams text deltas", () =>
Effect.gen(function* () {
const events = Array.from(
yield* Stream.runCollect(
Transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio: yield* audio }),
),
)
const finish = events.at(-1)
if (finish === undefined || !TranscriptionEvent.is.finish(finish)) throw new Error("Expected a finish event")
const deltas = events.filter(TranscriptionEvent.is.textDelta).map((event) => event.delta)
expect(deltas.length).toBeGreaterThan(1)
expect(deltas.join("")).toBe(finish.text)
expect(finish.text).toMatch(TRANSCRIPT)
expect(finish.usage).toMatchObject({ type: "tokens", input: expect.any(Number), output: expect.any(Number) })
}),
)
recorded.effect("diarizes speakers into segments", () =>
Effect.gen(function* () {
const response = yield* Transcription.generate({
model: openai.transcription("gpt-4o-transcribe-diarize"),
audio: yield* audio,
diarize: true,
})
expect(response.text).toMatch(TRANSCRIPT)
expect(response.segments).toEqual([
{
text: expect.stringMatching(TRANSCRIPT),
startSeconds: expect.any(Number),
endSeconds: expect.any(Number),
speaker: "A",
},
])
expect(response.durationSeconds).toBeGreaterThan(1)
}),
)
recorded.effect("returns whisper word timestamps", () =>
Effect.gen(function* () {
const response = yield* Transcription.generate({
model: openai.transcription("whisper-1"),
audio: yield* audio,
timestamps: "word",
})
expect(response.words?.map((word) => word.text.toLowerCase())).toEqual(["hello", "from", "opencode"])
expect(response.words?.every((word) => word.endSeconds > word.startSeconds)).toBe(true)
expect(response.language).toBe("english")
expect(response.usage).toEqual({ type: "seconds", seconds: expect.any(Number) })
}),
)
})
@@ -1,10 +1,10 @@
import type { Poll } from "../../src/index.js"
/**
* Video generations take minutes live, so record with a realistic interval. Replay consumes the recorded polls in
* Queued generations take minutes live, so record with a realistic interval. Replay consumes the recorded polls in
* cassette order regardless of timing, so it runs them back-to-back instead of sleeping between each one.
*/
export const videoPoll: Poll = {
export const queuedPoll: Poll = {
interval: process.env.RECORD === "true" ? "10 seconds" : "10 millis",
timeout: "15 minutes",
}
@@ -3,7 +3,7 @@ import { Effect } from "effect"
import { Video } from "../../src/index.js"
import { Runway } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { videoPoll } from "./video-recording.js"
import { queuedPoll } from "./queued-recording.js"
const model = Runway.configure({
apiKey: process.env.RUNWAYML_API_SECRET ?? "fixture",
@@ -28,7 +28,7 @@ describe("Runway recorded", () => {
aspectRatio: "1280:720",
durationSeconds: 2,
},
{ poll: videoPoll },
{ poll: queuedPoll },
)
expect(response.videos).toHaveLength(1)
@@ -0,0 +1,41 @@
import type { HttpRecorder } from "@opencode/http-recorder"
import { Effect } from "effect"
import { Media } from "../../src/index.js"
const fixture = (name: string) =>
Effect.promise(() => Bun.file(new URL(`../fixtures/audio/${name}`, import.meta.url)).bytes()).pipe(
Effect.map((bytes) => Media.bytes(bytes, "audio/mpeg")),
)
/** "Hello from OpenCode." spoken by Deepgram Aura (`aura-2-thalia-en`, 32 kbps MP3). */
export const audio = fixture("hello.mp3")
/** "Did the release ship?" (`aura-2-thalia-en`), then "Yes, it shipped this morning." (`aura-2-orion-en`). */
export const dialog = fixture("dialog.mp3")
/** Providers disagree on "OpenCode" versus "Open Code" and on the final punctuation. */
export const TRANSCRIPT = /^hello,? from open ?code[.!]?$/i
/**
* Request bodies are snapshotted as text, so the audio would be stored as mangled UTF-8. Replace it with a marker and
* pin the random multipart boundary, which keeps the remaining form fields and query checked on replay.
*/
const redactAudio = (body: string) => {
const boundary = /^--(\S+)\r\n/.exec(body)?.[1]
if (boundary === undefined) return body.includes("\uFFFD") ? "[audio]" : body
return body
.replaceAll(boundary, "BOUNDARY")
.replace(/(filename="[^"]*"\r\nContent-Type: [^\r]*\r\n\r\n)[\s\S]*?(\r\n--BOUNDARY)/, "$1[audio]$2")
}
const contentType = (snapshot: HttpRecorder.RequestSnapshot) =>
snapshot.headers["content-type"]?.replace(/boundary=.*/, "")
export const audioRecording: HttpRecorder.RecorderOptions = {
redact: { body: redactAudio },
match: (incoming, recorded) =>
incoming.method === recorded.method &&
incoming.url === recorded.url &&
contentType(incoming) === contentType(recorded) &&
incoming.body === recorded.body,
}
@@ -3,7 +3,7 @@ import { Effect, Stream } from "effect"
import { Video } from "../../src/index.js"
import { XAI } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { videoPoll } from "./video-recording.js"
import { queuedPoll } from "./queued-recording.js"
const model = XAI.configure({
apiKey: process.env.XAI_API_KEY ?? "fixture",
@@ -28,7 +28,7 @@ describe("xAI Video recorded", () => {
resolution: "480p",
durationSeconds: 2,
})
const observed = yield* generation.events({ poll: videoPoll }).pipe(Stream.runCollect)
const observed = yield* generation.events({ poll: queuedPoll }).pipe(Stream.runCollect)
const progress = Array.from(observed).flatMap((event) =>
event.type === "generation-progress" && event.progress !== undefined ? [event.progress] : [],
)
+4
View File
@@ -8,11 +8,13 @@ import { LLMClient, RequestExecutor } from "../src/route.js"
import { ImageClient } from "../src/image-client.js"
import { VideoClient } from "../src/video-client.js"
import { SpeechClient } from "../src/speech-client.js"
import { TranscriptionClient } from "../src/transcription-client.js"
import { EvaluationClient } from "../src/experimental/evaluation-client.js"
import type { Service as EvaluationClientService } from "../src/experimental/evaluation-client.js"
import type { Service as ImageClientService } from "../src/image-client.js"
import type { Service as VideoClientService } from "../src/video-client.js"
import type { Service as SpeechClientService } from "../src/speech-client.js"
import type { Service as TranscriptionClientService } from "../src/transcription-client.js"
import type { Service as LLMClientService } from "../src/route/client.js"
import type { Service as RequestExecutorService } from "../src/route/executor.js"
import {
@@ -30,6 +32,7 @@ type RecordedEnv =
| ImageClientService
| VideoClientService
| SpeechClientService
| TranscriptionClientService
| EvaluationClientService
| Socket.WebSocketConstructor
@@ -107,6 +110,7 @@ export const recordedTests = (options: RecordedTestsOptions) =>
ImageClient.layer.pipe(Layer.provide(requestExecutor)),
VideoClient.layer.pipe(Layer.provide(requestExecutor)),
SpeechClient.layer.pipe(Layer.provide(requestExecutor)),
TranscriptionClient.layer.pipe(Layer.provide(requestExecutor)),
EvaluationClient.layer.pipe(Layer.provide(requestExecutor)),
webSocket,
)
+174
View File
@@ -0,0 +1,174 @@
import { describe, expect } from "bun:test"
import { Effect, Fiber, Layer, Stream } from "effect"
import * as TestClock from "effect/testing/TestClock"
import { HttpClientRequest } from "effect/unstable/http"
import { Media, Transcription, TranscriptionClient } from "../src/index.js"
import { AssemblyAI, Deepgram, Google, OpenAI } from "../src/providers.js"
import { it } from "./lib/effect.js"
import { dynamicResponse } from "./lib/http.js"
const layer = (handler: Parameters<typeof dynamicResponse>[0]) =>
TranscriptionClient.layer.pipe(Layer.provideMerge(dynamicResponse(handler)))
const audio = Media.bytes(Uint8Array.from([0x49, 0x44, 0x33, 1, 2, 3]), "audio/mpeg")
const openai = OpenAI.configure({ apiKey: "test", baseURL: "https://openai.test/v1" })
const deepgram = Deepgram.configure({ apiKey: "test", baseURL: "https://deepgram.test" }).transcription("nova-3")
const google = Google.configure({ apiKey: "test", baseURL: "https://google.test/v1beta" }).transcription(
"gemini-3.5-transcribe",
)
const assemblyai = AssemblyAI.configure({ apiKey: "aai-key", baseURL: "https://assemblyai.test" }).transcription(
"universal-3-5-pro",
)
describe("Transcription", () => {
it.effect("rejects what a route cannot honor before sending anything", () =>
Effect.gen(function* () {
const errors = yield* Effect.all(
[
Stream.runCollect(Transcription.stream({ model: openai.transcription("whisper-1"), audio })),
Transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio, diarize: true }),
Transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio, timestamps: "word" }),
Transcription.generate({ model: openai.transcription("gpt-4o-transcribe-diarize"), audio, prompt: "Names" }),
Transcription.generate({ model: deepgram, audio, prompt: "OpenCode" }),
Transcription.generate({ model: google, audio, speakers: 2 }),
Transcription.start({ model: deepgram, audio }),
Transcription.generate({ model: deepgram, audio, http: { body: { callback: "https://hook.test" } } }),
Transcription.generate({
model: openai.transcription("gpt-transcribe"),
audio: Media.url("https://a.test/x.mp3"),
}),
Transcription.start({
model: assemblyai,
audio: Media.ref("file_1", { provider: "openai", mediaType: "audio/mpeg" }),
}),
Transcription.generate({
model: openai.transcription("gpt-transcribe"),
audio: Media.bytes(Uint8Array.from([1, 2, 3]), "audio/x-unknown"),
}),
Transcription.generate({
model: Google.configure({ apiKey: "test" }).transcription("gemini-3.6-flash"),
audio,
}),
].map((effect) => Effect.flip(effect)),
)
expect(errors.map((error) => [error.reason._tag, "operation" in error.reason && error.reason.operation])).toEqual(
[
["UnsupportedOperation", "media.stream"],
["UnsupportedOperation", "media.diarize"],
["UnsupportedOperation", "media.timestamps"],
["UnsupportedOperation", "media.prompt"],
["UnsupportedOperation", "media.prompt"],
["UnsupportedOperation", "media.speakers"],
["UnsupportedOperation", "transcription.start"],
["InvalidRequest", false],
["InvalidRequest", false],
["InvalidRequest", false],
["InvalidRequest", false],
["UnsupportedOperation", "transcription.model"],
],
)
}).pipe(Effect.provide(layer(() => Effect.die("an unsupported request reached the network")))),
)
it.effect(
"uploads inline audio to AssemblyAI, resumes polling from a persisted token, and surfaces failed transcripts",
() =>
Effect.gen(function* () {
const calls: Array<{
readonly method: string
readonly url: string
readonly auth: string | null
readonly body: string
}> = []
const json = (value: unknown) => ({
body: JSON.stringify(value),
init: { headers: { "content-type": "application/json" } },
})
const polls = () => calls.filter((call) => call.url.endsWith("/tr_1")).length
const failed = JSON.stringify({ id: "tr_2", status: "error", error: "Audio file could not be decoded" })
const program = Effect.gen(function* () {
const started = yield* Transcription.start({ model: assemblyai, audio, diarize: true, speakers: 2 })
expect(started.status).toBe("queued")
const resumed = yield* Transcription.resume(assemblyai, JSON.parse(JSON.stringify(started.token)))
const response = yield* resumed.await({ poll: { interval: "1 second" } })
const failure = yield* Transcription.resume(assemblyai, { transcriptID: "tr_2" }).pipe(
Effect.flatMap((generation) => generation.await()),
Effect.flip,
)
return { response, failure }
}).pipe(
Effect.provide(
layer((input) =>
Effect.gen(function* () {
const web = yield* HttpClientRequest.toWeb(input.request).pipe(Effect.orDie)
calls.push({
method: web.method,
url: web.url,
auth: web.headers.get("authorization"),
body: input.text,
})
if (web.url.endsWith("/tr_2"))
return input.respond(failed, { headers: { "content-type": "application/json" } })
const reply = web.url.endsWith("/v2/upload")
? json({ upload_url: "https://cdn.assemblyai.test/upload/1" })
: web.method === "POST"
? json({ id: "tr_1", status: "queued" })
: polls() < 3
? json({ id: "tr_1", status: "processing" })
: json({
id: "tr_1",
status: "completed",
text: "Hello there.",
words: [
{ text: "Hello", start: 250, end: 700, confidence: 0.9, speaker: "A" },
{ text: "there.", start: 700, end: 1250, confidence: 0.8, speaker: "B" },
],
utterances: [
{ text: "Hello", start: 250, end: 700, speaker: "A" },
{ text: "there.", start: 700, end: 1250, speaker: "B" },
],
language_code: "en_us",
audio_duration: 2,
})
return input.respond(reply.body, reply.init)
}),
),
),
)
const fiber = yield* Effect.forkChild(program)
yield* TestClock.adjust("5 seconds")
const { response, failure } = yield* Fiber.join(fiber)
expect(calls.slice(0, 2).map((call) => [call.method, call.url, call.auth])).toEqual([
["POST", "https://assemblyai.test/v2/upload", "aai-key"],
["POST", "https://assemblyai.test/v2/transcript", "aai-key"],
])
expect(JSON.parse(calls[1].body)).toEqual({
audio_url: "https://cdn.assemblyai.test/upload/1",
speech_models: ["universal-3-5-pro"],
language_detection: true,
speaker_labels: true,
speakers_expected: 2,
})
expect(calls[2].url).toBe("https://assemblyai.test/v2/transcript/tr_1")
expect(response.segments).toEqual([
{ text: "Hello", startSeconds: 0.25, endSeconds: 0.7, speaker: "A" },
{ text: "there.", startSeconds: 0.7, endSeconds: 1.25, speaker: "B" },
])
expect(response.words?.[1]).toEqual({
text: "there.",
startSeconds: 0.7,
endSeconds: 1.25,
speaker: "B",
confidence: 0.8,
})
expect(response).toMatchObject({
text: "Hello there.",
language: "en_us",
usage: { type: "seconds", seconds: 2 },
})
expect(failure.reason).toMatchObject({ _tag: "ProviderInternal", body: failed })
}),
)
})
+34
View File
@@ -0,0 +1,34 @@
import type { Stream } from "effect"
import { Media, Transcription, type TranscriptionEvent } from "../src/index.js"
import { AssemblyAI, Deepgram, OpenAI } from "../src/providers.js"
type StreamItem<T> = T extends Stream.Stream<infer A, infer _E, infer _R> ? A : never
type Equal<A, B> = [A, B] extends [B, A] ? true : false
type Assert<T extends true> = T
const audio = Media.url("https://example.com/call.mp3")
const deepgram = Deepgram.configure({ apiKey: "test" }).transcription("nova-3")
Transcription.generate({
model: deepgram,
audio,
timestamps: "word",
providerOptions: { keyterm: ["OpenCode"], diarize_model: "future-model", futureOption: true },
})
Transcription.start({ model: AssemblyAI.configure({ apiKey: "test" }).transcription("universal-3-5-pro"), audio })
// @ts-expect-error Known provider options are inferred from the selected model.
Transcription.generate({ model: deepgram, audio, providerOptions: { keyterm: "OpenCode" } })
// @ts-expect-error Timestamp granularity is a closed set.
Transcription.generate({ model: deepgram, audio, timestamps: "character" })
// @ts-expect-error Audio is required.
Transcription.request({ model: deepgram })
// @ts-expect-error Speech models cannot be used for transcription requests.
Transcription.request({ model: OpenAI.configure({ apiKey: "test" }).speech("gpt-4o-mini-tts"), audio })
const streamed = Transcription.stream({
model: OpenAI.configure({ apiKey: "test" }).transcription("gpt-transcribe"),
audio,
})
type StreamEvent = Assert<Equal<StreamItem<typeof streamed>, TranscriptionEvent>>
void (true satisfies StreamEvent)