Providers

@voltro/ai exposes one surface behind a provider abstraction. Pick yours via AI_PROVIDER. The same free functions — generateText({ prompt }), generateObject({ prompt, schema }), streamText({ prompt }) — work against any provider.

There is no ctx.ai. AI lives in the free functions you import from @voltro/ai, not on the request context.

Both catalog readers preserve the evaluation modality and accept it as a filter. Catalog visibility does not imply an execution primitive: executableModalities() continues to describe only supported framework operations.

#Supported providers

Provider	AI_PROVIDER value	Default model	Key env var	Notes
Anthropic	anthropic	claude-opus-4-8	ANTHROPIC_API_KEY	Claude. Best for agentic tool use + long context.
OpenAI	openai	gpt-5.5	OPENAI_API_KEY	GPT models via @ai-sdk/openai.
Gateway	gateway	none (required)	AI_GATEWAY_API_KEY	The Vercel AI Gateway: ANY model the AI SDK can reach through ONE key, addressed by a creator/model id (openai/gpt-5.5, anthropic/claude-opus-4-8, google/gemini-2.5-pro). No per-vendor @ai-sdk/* package needed.
Mock	mock	mock	none	Deterministic responses for tests + CI. Echoes the prompt (or scripted output).

mock is the default — voltro dev runs key-free out of the box, and every smoke test stays deterministic without a network call. Set AI_PROVIDER=anthropic / openai / gateway to use a real model.

Direct provider vs gateway: reach for a direct provider (anthropic / openai) when you want that vendor's own key + native behaviour. Reach for gateway when you want "any model, one key" without adding a new @ai-sdk/* package — the model id carries the vendor (openai/gpt-5.5). All four packages stay pinned to the same @ai-sdk/provider major; mixing a newer-major provider package in would break the shared model type.

#Setting the provider

AI_PROVIDER=anthropic AI_MODEL=claude-opus-4-8 ANTHROPIC_API_KEY=sk-ant-… voltro dev

Env vars providerFromEnv() reads:

Var	Default	Notes
AI_PROVIDER	mock	mock | anthropic | openai | gateway.
AI_MODEL	per provider (see below)	Override the model. Defaults: claude-opus-4-8 (anthropic), gpt-5.5 (openai), mock (mock). REQUIRED for gateway — a creator/model id; boot throws if unset.

Provider API keys are read by the underlying @ai-sdk/* packages from their standard env vars — ANTHROPIC_API_KEY, OPENAI_API_KEY, AI_GATEWAY_API_KEY. The framework doesn't read a separate AI_API_KEY — but it checks the one the provider needs at boot: with AI_PROVIDER set to a real provider and its key unset or empty, voltro serve refuses to start and voltro dev warns, naming the variable. The SDK would otherwise read the key at the first model call, so a rotated key surfaced hours after a deploy that reported healthy. An app that supplies every key in code opts out with ai: { requireProviderKey: false } in app.config.ts. (To give a SINGLE agent its own key from code instead of env, see Per-config key + base URL below.)

#Anthropic

AI_PROVIDER=anthropic
AI_MODEL=claude-opus-4-8
ANTHROPIC_API_KEY=sk-ant-…

Anthropic-specific:

• Prompt caching — automatic for long, repeated prefixes via the AI SDK.
• Tool calling — Claude's native tool format; the framework's defineTool(...) adapts to it.

For long contexts Claude is the practical pick — fewer "context overflow" surprises.

#OpenAI

AI_PROVIDER=openai
AI_MODEL=gpt-5.5
OPENAI_API_KEY=sk-…

Backed by @ai-sdk/openai. AI_MODEL is a plain OpenAI model id (gpt-5.5, gpt-4o-mini, …). For a self-hosted / Azure-style / proxy endpoint, set baseURL on a ProviderConfig (see Per-config key + base URL) rather than an env var.

#Vercel AI Gateway

AI_PROVIDER=gateway
AI_MODEL=openai/gpt-5.5            # creator/model id — REQUIRED, no default
AI_GATEWAY_API_KEY=…

The gateway reaches EVERY model the AI SDK can address through ONE key — you never add a per-vendor @ai-sdk/* package. The model id encodes the vendor: openai/gpt-5.5, anthropic/claude-opus-4-8, google/gemini-2.5-pro. There's no sensible default model (the id IS the choice), so AI_PROVIDER=gateway with no AI_MODEL throws at boot with a pointer to fix it.

Use the gateway when you want to switch models across vendors freely from config; use a direct provider when you want that vendor's own key + native quirks (Anthropic prompt-caching, etc.).

#Live model catalog — getAvailableModels

A model-picker UI shouldn't hard-code a model list that goes stale. getAvailableModels() lists the gateway's models live — each with its modality + pricing — so the picker is always current:

import { getAvailableModels } from '@voltro/ai'

// Reads AI_GATEWAY_API_KEY; pass { apiKey, baseURL } to target a specific gateway.
const models = await getAvailableModels({ modality: 'language' })   // filter optional
// → GatewayModelInfo[]: { id, name, description?, modality, pricing? }
//   id        — the creator/model id you pass as the model ('openai/gpt-5.5')
//   modality  — 'language' | 'embedding' | 'image' | 'video' | 'realtime'
//             | 'reranking' | 'speech' | 'transcription' | 'evaluation' | 'unknown'
//   pricing   — { inputPer1M, outputPer1M, cachedInputPer1M? } (USD per 1M tokens)

Pricing is normalised to USD per 1,000,000 tokens, the same shape as the cost toolkit's ModelPrice — so a catalog entry can feed the cost ledger directly (estimateCostUsd(usage, { model, price })). getAvailableModels is async (a plain Promise, not an Effect) and gatewayProvider is injectable for tests, so a picker query can call it without a live gateway in CI.

#The full catalog — readGatewayCatalog

getAvailableModels asks the AI SDK's gateway provider, whose entry type carries six fields. The HTTP response carries twenty-one. readGatewayCatalog() reads the response:

import { readGatewayCatalog, canExecute } from '@voltro/ai'

const models = await readGatewayCatalog({ modality: 'language' })
const usable = models.filter(canExecute).filter((m) => m.capabilities?.vision)

Each entry adds capabilities, modalities, contextWindow, maxOutputTokens, releasedAt, knowledgeCutoff and dataPolicy.

Plus raw — the entry exactly as it came over the wire.

The normalised fields are the shared answer to the questions every app on the gateway asks. raw is for the ones only one app asks: a chips row built from the vendor's own tags, a pricing display that shows a structure we have not normalised, zdr and no_training kept apart instead of folded into dataPolicy.

const models = await readGatewayCatalog({ modality: 'language' })
const chips = models[0]?.raw?.tags            // vendor's own vocabulary
const audioRate = (models[0]?.raw?.pricing as { audio_per_second?: string })?.audio_per_second

Without it, an app with its own display fetches /v1/models a second time and keeps a second parser of the same response — and then has to decide again, at every field the gateway adds, whether to wait for a framework release. That is the decision raw removes, which is why it is one field rather than three more normalised ones.

It is absent on the injected-provider path (the seam a test uses), for the same reason capabilities is: that path never saw a response, and an empty object would turn "no wire data" into the claim "the wire carried nothing".

capabilities is unioned from three fields that each half-say it — tags, supported_parameters and modalities — because they overlap and regularly disagree: a model tagged reasoning does not always carry the parameter, and the other way round. Taking either alone produces a false negative, and a false "cannot" removes a model from a picker with no way for anyone to find out why. That union is framework knowledge; without it every app writes the same forty lines, differently.

dataPolicy carries zeroRetention and noTraining ('all' | 'some' | 'none'). They decide whether a model may see customer text, and an unreadable value reads as 'none' — the question is "may this", and an unparseable answer is not a yes.

capabilities and modalities are optional, and that is deliberate: they are populated on every HTTP read, and absent only when an entry came from the injected gatewayProvider a test uses. Defaulting them to all-false would turn "not known" into the claim "this model has no vision".

#What the runtime can actually run — executableModalities

The catalog names eight modalities; a build can execute the ones it has primitives for. Offering a model it cannot run produces a control that writes into nothing, so ask rather than keep a list:

import { executableModalities, canExecute } from '@voltro/ai'

executableModalities()      // Set { language, embedding, image, video, speech, reranking, transcription, realtime }
canExecute(model)           // realtime uses defineRealtimeSession, not the RPC socket

A hand-maintained list of runnable kinds goes stale the moment a primitive ships, and it goes stale silently — the app keeps hiding a modality that now works. This set is tied to the exports that justify each entry, so shipping a primitive without adding it turns a test red.

#Per-config key + base URL

Every ProviderConfig accepts an optional apiKey and baseURL. When EITHER is set, the framework builds a dedicated provider instance from it (createOpenAI({ apiKey }) / createAnthropic(...) / createGateway(...)) instead of the env-default singleton. Omit both → it falls back to the standard env var. The mock provider ignores both.

// A one-off call against a specific key + endpoint:
const r = yield* generateText({
  prompt,
  provider: { name: 'openai', model: 'gpt-5.5', apiKey: process.env.TEAM_OPENAI_KEY, baseURL: 'https://my-proxy/v1' },
})

This is the mechanism behind per-agent keys and BYOK (bring-your-own-key): a defineAgentExecutor can carry a static model: { name, model, apiKey }, OR a model: (input) => ({ …, apiKey: input.apiKey }) function that derives the key from the request. The key stays server-side (the executor file never reaches the browser) and is NOT persisted — only the prompt is stored in agent_messages. The full pattern (dynamic model picker, cost/tier routing, BYOK, and the security footguns) lives in Agents → Per-agent model + key.

#Mock (for tests)

AI_PROVIDER=mock

Every call returns deterministic output (it echoes the prompt, or a scripted turn sequence). Useful for:

• CI runs where you don't want real API calls
• Unit tests of agents — assert on the call shape, not the output
• Local dev when you're offline

Configure the mock per-test. useMockAi installs a process-global mock provider that every generateText / generateObject / streamText / runAssistant call resolves to (overriding AI_PROVIDER); call reset() to restore:

import { useMockAi } from '@voltro/ai/test'

let mock: ReturnType<typeof useMockAi>
beforeEach(() => {
  mock = useMockAi({
    // canned generateText / generateObject text
    generate: { text: 'Mocked summary text.' },
    // streamText token sequence (drives runAssistant deltas)
    stream:   ['Hello, ', 'world.'],
    // scripted tool-call → text turns for the tool loop (one per step)
    turns: [
      { toolCalls: [{ name: 'searchDocs', input: { query: 'x' } }] },
      { text: 'Based on the docs.' },
    ],
  })
})
afterEach(() => { mock.reset() })

mockAi({...}) returns a MockAi value with the same fixture shape — handy as a test helper for code that wants a mock to assert against.

throwNTimes(n, value) is a helper for retry tests — a function that throws the first n calls, then returns value.

#The call surface

One-shot text:

import { generateText } from '@voltro/ai'
import { Effect } from 'effect'

export default (input: { prompt: string }) =>
  Effect.gen(function* () {
    const { text, usage } = yield* generateText({ prompt: input.prompt })
    return { text }
  })

GenerateTextOptions is { prompt, system?, provider?, fallbacks?, maxTokens? }. There is no messages/effort shape — the prompt is a single string the SDK wraps as the user turn; system steers it. fallbacks is a provider fallback chain.

Structured output:

import { generateObject } from '@voltro/ai'
import { Schema } from 'effect'

const Summary = Schema.Struct({ title: Schema.String, bullets: Schema.Array(Schema.String) })

const { object } = yield* generateObject({ prompt: input.text, schema: Summary })

#Switching providers per call

Pass provider to override the env default for one call:

import { generateText } from '@voltro/ai'

const r = yield* generateText({
  prompt,
  system,
  provider: { name: 'anthropic', model: 'claude-opus-4-8' },
})

provider is a ProviderConfig — a discriminated union on name, so each provider only accepts the fields that apply to it:

type ProviderConfig =
  | { name: 'mock';      model?: string; mockText?: string; script?: MockScript }  // mock-only fields
  | { name: 'anthropic'; model: AnthropicModel; apiKey?: string; baseURL?: string }
  | { name: 'openai';    model: OpenAIModel;    apiKey?: string; baseURL?: string }
  | { name: 'gateway';   model: `${string}/${string}`; apiKey?: string; baseURL?: string }  // creator/model

The mock-only mockText / script can't appear on a real provider (the type rejects it), the gateway's model is a creator/model-typed string (a bare 'gpt-5.5' is a compile error, not a boot crash), and the per-provider model-id types (AnthropicModel / OpenAIModel) are open unions — known ids autocomplete, but any string the provider ships tomorrow still type-checks. Use provider for per-request model selection (e.g. a cheaper model on a fallback path), or to pin a specific key/endpoint (see Per-config key + base URL).

For runtime provider switching across a whole layer, bind an AiServiceImpl to the AiService Context tag at boot and read it with yield* AiService — defaultAiService (backed by providerFromEnv) is the default.

#Fallback chain — survive a provider outage

Pass an ordered fallbacks list to fall through to another provider/model when the primary FAILS. When the primary call fails — after its own retries — the call re-runs against fallbacks[0], then fallbacks[1], … until one succeeds; exhausting every option surfaces the LAST provider's typed error. So an Anthropic outage transparently drains to OpenAI (or the gateway) when a key is configured:

import { generateText } from '@voltro/ai'

const r = yield* generateText({
  prompt,
  provider:  { name: 'anthropic', model: 'claude-opus-4-8' },
  fallbacks: [
    { name: 'openai',  model: 'gpt-5.5' },
    { name: 'gateway', model: 'google/gemini-2.5-pro' },
  ],
})

• Applies to generateText, generateObject, generateObjectWithTools, and the stream surface. For streaming, use streamTextWithFallback(options) or streamTextWithRetry(options, retry) — the latter retries each provider maxAttempts times BEFORE moving to the next. A plain streamText uses only the primary.
• Only a provider (generation) failure falls through. A decode failure — the model answered but the output didn't satisfy the schema — surfaces immediately, because another provider won't fix a schema/prompt problem.
• Streams fall through only before content flows. Once any token/tool event has streamed the answer is committed and a later error surfaces as-is (re-running elsewhere would duplicate output); a deliberate cancelled never triggers a fallback.
• No fallbacks ⇒ unchanged single-provider behavior.

Typed errors are preserved end-to-end: a fully-exhausted chain fails with the last AiError on the Effect channel (generate) or a terminal error event (stream). Cost/usage is attributed to whichever provider actually served the call (the observability span + metrics stamp its provider/model).

#Middleware — wrap every model call

Language-model middleware wraps every model the framework resolves — for generateText, generateObject, streamText, and agents alike — so you add cross-cutting behavior (logging, reasoning extraction, default settings, caching, guardrails) in ONE place without touching call sites. It's the AI SDK's wrapLanguageModel seam, exposed as a process-global stack you install once at boot:

import { setAiMiddleware, loggingMiddleware } from '@voltro/ai'

// Typically in a *.startup.tsx boot hook (or app.config layers):
setAiMiddleware(loggingMiddleware())

Every subsequent call is wrapped; nothing else changes. setAiMiddleware(...) returns the previous stack (restore it in a test), getAiMiddleware() reads it, clearAiMiddleware() empties it. With an empty stack the model is passed through untouched — zero overhead on the default path.

Built-ins (all re-exported from @voltro/ai):

• loggingMiddleware({ log? }) — logs each call's model + token usage (dev default: console; pass log to route into your logger/metrics).
• extractReasoningMiddleware({ tagName }) — split inline <think>…</think> reasoning out of the answer text into the reasoning channel, for models that don't emit native reasoning parts.
• defaultSettingsMiddleware({ settings }) — pin default call settings (temperature, maxOutputTokens, providerOptions) for every call.
• simulateStreamingMiddleware() — make a generate-only model satisfy the streaming path (emits the full text as one delta).

Custom middleware is any object implementing transformParams / wrapGenerate / wrapStream (typed AiMiddleware):

import { setAiMiddleware, type AiMiddleware } from '@voltro/ai'

const redactPII: AiMiddleware = {
  transformParams: async ({ params }) => params,   // scrub params.prompt before it leaves
}
setAiMiddleware(redactPII, loggingMiddleware())    // applied in order — the first entry is outermost

Middleware is server-only — install it where the app boots, never from a browser-safe descriptor.

#Embeddings — a separate axis

Embeddings use their own provider env, not AI_PROVIDER:

AI_EMBED_PROVIDER=openai          # mock (default) | openai | voyage | cohere | gateway
AI_EMBED_MODEL=text-embedding-3-small

The default is mock (deterministic, key-free). Real embedding providers (openai / voyage / cohere) resolve their AI-SDK package via a lazy, server-only dynamic import — install the package (e.g. @ai-sdk/openai) to use them. gateway needs no extra package: it routes a gateway-hosted embedding model (AI_EMBED_MODEL=openai/text-embedding-3-small, the default for it) through the AI Gateway with the key the text side already uses, and an EmbeddingProviderConfig takes the same apiKey / baseURL override as the text-side gateway provider. providerOptions and expectedDimensions work as for the direct providers:

import { embedMany } from '@voltro/ai'

export const embedChunks = (chunks: ReadonlyArray<string>) =>
  embedMany(chunks, {
    name: 'gateway', model: 'openai/text-embedding-3-small',
    providerOptions: { openai: { dimensions: 1536 } }, expectedDimensions: 1536,
  })

embed / embedMany return the vectors only — no usage or cost, for any provider. See RAG.

#Provider quirks

• Anthropic 429s burst — burst rate limits trip before the monthly quota. Handle retries with Effect.retry in your executor.
• Mock determinism — every test using useMockAi is isolated; fixtures don't bleed across describe blocks.