Cost tracking

Token usage on every call plus the shipped cost toolkit — estimateCostUsd + a price table, the _voltro_ai_usage ledger, spend sums, and per-tenant budget guards.

LLMs are usage-priced, so you need to know how many tokens each call burned. Every @voltro/ai call returns a token usage tally on its result. That's the shipped primitive.

On top of the raw tally, @voltro/ai ships a cost toolkit: a price table + estimateCostUsd, a reactive _voltro_ai_usage ledger (recordAiUsage), spend sums (aiSpendUsd), and a per-tenant budget guard (requireAiBudget → typed AiBudgetExceeded). (@voltro/plugin-audit is separate — it records mutation invocations, not AI calls.)

Token usage on every call

generateText and generateObject return usage alongside the result:

import { generateText } from '@voltro/ai'
import { Effect } from 'effect'

export default (input: { prompt: string }) =>
  Effect.gen(function* () {
    const { text, usage } = yield* generateText({ prompt: input.prompt })
    // usage = { inputTokens, outputTokens, totalTokens }
    //   each is `number | undefined` (provider-reported)
    return { text, tokens: usage.totalTokens }
  })

const { object, usage } = yield* generateObject({ prompt, schema })
// same usage shape

Token usage on a streamed run

A streamText run ends with a terminal done event that carries the same usage shape:

import { streamText } from '@voltro/ai'
import { Stream, Effect } from 'effect'

yield* streamText({ prompt }).pipe(
  Stream.runForEach((event) =>
    Effect.sync(() => {
      if (event._tag === 'done') {
        // event.finishReason — 'stop' | 'tool-calls' | 'error' | …
        // event.usage = { inputTokens, outputTokens, totalTokens }
      }
    }),
  ),
)

The done event's usage is the run total across every LLM↔tool round-trip.

Pricing a call — `estimateCostUsd`

estimateCostUsd(usage, { model }) turns a token tally into a USD CostBreakdown using the built-in price table (MODEL_PRICING_DEFAULTS, USD per 1M tokens). An unknown model — or the mock provider — prices at zero, so cost accounting never breaks a call.

The defaults cover the major providers — Anthropic (claude-*), OpenAI (gpt-*), and Google Gemini (gemini-*) — so a non-Claude call is priced too (a gpt-4o or gemini-2.5-pro call is a real number, not a silent zero). A gateway creator/model id (openai/gpt-4o) prices by its bare model segment.

import { generateText, estimateCostUsd } from '@voltro/ai'

const model = 'claude-opus-4-8'
const { text, usage } = yield* generateText({ prompt, provider: { name: 'anthropic', model } })
const cost = estimateCostUsd(usage, { model })
// cost = { inputTokens, outputTokens, inputCostUsd, outputCostUsd, totalCostUsd, costSource }
//   costSource: 'estimated' (price-table or zero) | 'gateway' (real reported cost)

Override the price for a model the table doesn't know, or for negotiated / volume pricing:

estimateCostUsd(usage, { model, price: { inputPer1M: 2.5, outputPer1M: 10 } })

The built-in prices are point-in-time defaults — override them app-wide

MODEL_PRICING_DEFAULTS are public list prices as of January 2026 and WILL drift as providers re-price. Treat them as a sane default for the budget guard + cost dashboard, not a contract. To encode current or negotiated rates once, at boot, without editing the framework, call setModelPricing — a process-global override map merged OVER the defaults (a user entry for a model id wins):

import { setModelPricing } from '@voltro/ai'

// Wire your ai config's `pricing` map through this at boot.
setModelPricing({
  'gpt-4o':          { inputPer1M: 2.5, outputPer1M: 10 },  // corrected list price
  'my-tuned-model':  { inputPer1M: 0.8, outputPer1M: 2.4 }, // a model the defaults don't know
})

Every estimateCostUsd / recordAiUsage / budget call then reads the merged map. priceForModel(model) returns the effective price (or undefined if unknown). For a gateway-routed model, the gateway's REPORTED per-call cost still wins over any static rate (see below).

Real gateway cost — `gatewayCostUsd` + `actualCostUsd`

The static price table only knows the models it lists. A gateway-routed model the table doesn't carry would otherwise estimate to zero — wrong, not just imprecise. The fix: the Vercel AI Gateway reports the ACTUAL per-call cost in the result's provider metadata, and the toolkit prefers it.

gatewayCostUsd(providerMetadata) pulls providerMetadata.gateway.cost (a USD number or numeric string) out of a generate/stream result, returning undefined for a direct provider / the mock (so you fall back to the table):

import { generateText, gatewayCostUsd, recordAiUsage } from '@voltro/ai'

const model = 'openai/gpt-5.5'   // a gateway id the static table doesn't list
const r = yield* generateText({ prompt, provider: { name: 'gateway', model } })
const actualCostUsd = gatewayCostUsd(r.providerMetadata)   // the gateway's real charge, or undefined

const cost = yield* recordAiUsage(ctx.store, {
  tenantId: ctx.request.subject.tenantId,
  provider: 'gateway',
  model,
  operation: 'generateText',
  usage: r.usage,
  actualCostUsd,          // when set → persisted verbatim, costSource: 'gateway'
})
// cost.costSource === 'gateway' (authoritative) when actualCostUsd was present,
// else 'estimated' (price table or zero).

estimateCostUsd(usage, { model, actualCostUsd }) honours the same rule: a present actualCostUsd wins (split across input/output by token share for the breakdown, costSource: 'gateway'); absent, it uses the price table (costSource: 'estimated'). A cost dashboard can flag estimated rows and un-priced (zero) models so you know which numbers are real vs derived.

The usage ledger — `recordAiUsage` + `aiUsageTable`

recordAiUsage(store, {...}) prices a call and writes one row to the _voltro_ai_usage ledger (aiUsageTable), returning the same CostBreakdown. The table is reactive and auto-migrated whenever the app ships any *.agent.tsx; a non-agent app that wants plain-call tracking imports aiUsageTable into its database/index.ts barrel. Cost is stored as integer micro-USD (costMicroUsd, USD × 1e6) — the same "money = integer minor units" rule the billing plugin uses.

import { generateText, recordAiUsage } from '@voltro/ai'

const model = 'claude-opus-4-8'
const { text, usage } = yield* generateText({ prompt, provider: { name: 'anthropic', model } })
const cost = yield* recordAiUsage(ctx.store, {
  tenantId:  ctx.request.subject.tenantId,
  provider:  'anthropic',
  model,
  operation: 'generateText',     // or 'generateObject' | 'streamText' | 'agent' | your own
  usage,
})
// cost.totalCostUsd — surface it without a re-query

A row carries { tenantId, provider, model, operation, agent, inputTokens, outputTokens, costMicroUsd, costSource, calledAt }. costSource is 'gateway' (the actual reported cost) or 'estimated' (price-table / zero) — pass actualCostUsd (see above) to record the real gateway charge.

Spend + budgets — `aiSpendUsd` / `requireAiBudget`

aiSpendUsd(store, { tenantId?, since? }) sums recorded spend (USD). Because aiUsageTable is reactive, a defineQuery with source: '_voltro_ai_usage' that calls it is a live spend meter — the same reactive-query machinery as everything else.

const spentThisMonth = yield* aiSpendUsd(ctx.store, {
  tenantId: ctx.request.subject.tenantId,
  since:    startOfMonth(),
})

requireAiBudget(store, { tenantId, limitUsd, addUsd?, since? }) gates a call against a per-tenant cap — the precedent is billing's requireEntitlement. It fails with a typed, client-marshalable AiBudgetExceeded when reserved + addUsd would exceed limitUsd. Call it BEFORE the provider call (estimate addUsd from the prompt); record the real cost after.

Atomic across replicas — a hard cap, not a soft one. The guard RESERVES addUsd on a single per-tenant counter row (_voltro_ai_budget) via a bounded compare-and-set loop — the same store-level atomic-consume the storage and billing plugins use. So N concurrent calls (same replica or across replicas) reserve exactly the budgeted amount and the rest fail — no overshoot. (This replaces an older check-then-act sum that concurrent callers could all read under-cap and all pass.) The reservation is optimistic: it does NOT auto-release if the provider call later fails, which for a rolling budget is the correct conservative bound. Pass addUsd: 0 for a check-only UI pre-flight that reserves nothing. A rolling since window rotates to a fresh counter (the prior window's row ages out via retention).

import { generateText, requireAiBudget, recordAiUsage, AiBudgetExceeded } from '@voltro/ai'
import { Effect } from 'effect'

export default (input: { prompt: string }, ctx) =>
  Effect.gen(function* () {
    const tenantId = ctx.request.subject.tenantId
    // Refuse if this tenant is already at/over its monthly cap.
    yield* requireAiBudget(ctx.store, { tenantId, limitUsd: 50, addUsd: 0.25, since: startOfMonth() })

    const model = 'claude-opus-4-8'
    const { text, usage } = yield* generateText({ prompt: input.prompt, provider: { name: 'anthropic', model } })
    yield* recordAiUsage(ctx.store, { tenantId, provider: 'anthropic', model, operation: 'generateText', usage })
    return { text }
  })

Declare error: AiBudgetExceeded on the descriptor so the rpc layer surfaces the rejection typed; the client pattern-matches on { _tag: 'AiBudgetExceeded', limitUsd, spentUsd, attemptedUsd } (spentUsd is the amount already reserved on the counter).

Per-call observability — automatic spans + metrics

Every generateText / generateObject / generateObjectWithTools / streamText call is automatically wrapped in a voltro.ai.call OTel span (attributes ai.provider / ai.model / ai.operation) and records metrics into the framework's global metric registry — the same one @voltro/plugin-prometheus exposes at /metrics and the dashboard reads at /_voltro/inspect/metrics. No wiring needed:

Metric	Type	What
`voltro_ai_calls_total`	counter	Calls, labelled `provider` / `model` / `operation` / `status`.
`voltro_ai_call_errors_total`	counter	Calls that errored.
`voltro_ai_call_duration_seconds`	histogram	Provider call latency.
`voltro_ai_input_tokens_total` / `voltro_ai_output_tokens_total`	counter	Prompt / completion tokens.
`voltro_ai_cost_microusd_total`	counter	Estimated spend (micro-USD), priced off the merged table.

Labels carry only provider / model / operation ids — never prompt or response content, never a key. Cost here is the estimated figure from the price table (for a live-priced budget cap, use requireAiBudget; for the authoritative gateway charge, use recordAiUsage({ actualCostUsd })).

Semantic caching — zero-token hits

The cheapest LLM call is the one you don't make. A plain key/value cache misses on a near-duplicate prompt ("how do I deploy" vs "how to deploy?"); a semantic cache keys on the embedding of the prompt and returns a hit when a stored entry's vector is within a cosine-similarity threshold. @voltro/ai's semanticGenerateText / semanticGenerateObject wrap generateText / generateObject with that lookup: a hit returns the cached answer with zero token usage, a miss generates and stores it.

import { semanticGenerateText } from '@voltro/ai'
import { makeSemanticCache, tableDep } from '@voltro/cache'
import { Effect } from 'effect'

const answer = (prompt: string) =>
  Effect.gen(function* () {
    // `store` is a resolved CacheStore (memory or RESP) — see /docs/caching.
    const cache = yield* makeSemanticCache(store)

    const res = yield* semanticGenerateText(
      { prompt },
      { deps: [tableDep('docs')], threshold: 0.95 },  // 0.95 default — only near-duplicates share
      { cache },
    )
    // res.cached === true on the next near-identical prompt (res.usage all-zero).
    return { text: res.value, cached: res.cached, tokens: res.usage.totalTokens }
  })

Framework-managed — `cacheSemantic: true`

In an app you don't hand-build the cache. Set cacheSemantic: true in app.config.ts and both boot paths (voltro dev, voltro serve) build a SemanticCache over the SAME cache store the query cache uses, provide it as a yield*-able handler service, AND wire row-granular eviction off the runtime's store.onChange automatically. A handler asks for it instead of calling makeSemanticCache:

import { SemanticCache } from '@voltro/cache'
import { recordReads, semanticGenerateText } from '@voltro/ai'
import { Effect } from 'effect'

export default (input: { prompt: string }, ctx) =>
  Effect.gen(function* () {
    const cache = yield* SemanticCache            // provided when `cacheSemantic: true`
    const rec = recordReads(ctx.store)
    const docs = yield* Effect.promise(() => rec.store.query(docsQuery))
    const res = yield* semanticGenerateText(
      { prompt: `${input.prompt}\n\n${JSON.stringify(docs)}` },
      { deps: rec.deps() },
      { cache },
    )
    return { text: res.value, cached: res.cached }
  })

It is off by default — an app that never sets cacheSemantic builds no vector index, no service, and no eviction sink, so it pays nothing; a handler that yield* SemanticCaches without the opt-in gets the ordinary "service not found".

Dependency-driven eviction — it never serves a stale answer

The correctness property a generic "Redis + embeddings" cache lacks: each entry records the source rows/tables the answer read as its dependency set, and evicts when one of them changes. Tag the entry with tableDep(table) / rowDep(table, id), or capture the set automatically from the reads with recordReads:

import { recordReads, semanticGenerateText } from '@voltro/ai'
import { makeSemanticCache } from '@voltro/cache'
import { Effect } from 'effect'

const groundedAnswer = (prompt: string) =>
  Effect.gen(function* () {
    const cache = yield* makeSemanticCache(store)
    const rec = recordReads(ctx.store)                       // wrap the store
    const docs = yield* Effect.promise(() => rec.store.query(docsQuery))
    const res = yield* semanticGenerateText(
      { prompt: `${prompt}\n\n${JSON.stringify(docs)}` },
      { deps: rec.deps() },   // exactly the rows/tables `docs` came from
      { cache },
    )
    return res.value          // evicted the moment any of those rows change
  })

The honest bounds

Eviction on live writes is automatic under cacheSemantic: true. With the opt-in on, both boot paths subscribe the runtime's store.onChange for you: a live DB write to a source row drops every semantic entry that depended on it (onSourceChange), so the cache never serves an answer whose grounding rows have changed — you wire no sink. (If you hand-build a SemanticCache with makeSemanticCache outside the opt-in, you drive onSourceChange(change) / onTableChange(table) yourself; the app-config path is the supported one.)
The vector index is per-process (V1). A RESP-backed store's cached VALUES survive a restart and are shared across replicas, but the embedding index that finds a near-duplicate lives in-process — so a semantic HIT is per-replica and is rebuilt after a restart. Cross-process semantic lookup needs a durable ANN index (not yet shipped).
A cache outage degrades to always-generate. Both the lookup and the store are best-effort — a CacheError reads as a miss (or a swallowed put), never a failed call. The cache is an optimisation, not a dependency.
The object variant stores the DECODED object. With the memory backend it round-trips by reference; with a RESP backend it is JSON, so a schema whose decoded form is not JSON-safe (class instances, non-plain branded carriers) will not survive a cross-process hit — cache the text form or a JSON-safe projection for those.

A model call inside a workflow — `aiStep`

@voltro/ai/workflow wraps a call as a durable step:

import { aiStep, aiObjectStep } from '@voltro/ai/workflow'

const summary = yield* aiStep({
  name: 'summarise-thread',
  prompt: `Summarise:\n${thread}`,
  store: ctx.store,
  tenantId: payload.tenantId,
})

Journaling is not what this adds — every step() is already journaled, so a replay of a plain wrapped generateText returns the recorded completion rather than re-calling the model. Three things are different:

It records what the run cost. A model call inside a workflow was invisible to _voltro_ai_usage unless the app remembered to call recordAiUsage by hand — so the spend ledger was systematically missing exactly the calls that run unattended. Pass store and every call is recorded, attributed to the workflow and the step.
It does not copy the prompt into a second table. step({ input }) is written to _voltro_workflow_run_steps and rendered in the dashboard; for a prompt built from customer data that is a plaintext copy outside whatever boundary you established for the source. The default records a digest plus the length. recordPrompt: 'full' exists and has to be typed out.
Provider failures retry like provider failures. The default policy handles a 429 with a Retry-After and a 5xx, rather than every app rediscovering that a bare call fails the whole durable run on a rate limit.

aiObjectStep is the schema-constrained form; the schema is the step's success schema too, so the journaled value decodes on replay exactly as it did on the first run.

Pass offload: true and the run stops occupying a worker while the model thinks — see the next section.

Offloading the call — `offload: true`

An inline aiStep holds a runner fiber for the length of the model call. At six seconds a call and two hundred concurrent runs, that is two hundred parked workers waiting on a socket, and the cluster's concurrency is spent on latency rather than on work.

const summary = yield* aiStep({
  name: 'summarise-thread',
  prompt: `Summarise:\n${thread}`,
  store: ctx.store,
  offload: true,
})

The run suspends: the worker is released, the wait lives as a row in _voltro_ai_inferences, and a dispatcher owns the socket. Two hundred waiting runs become two hundred rows and (by default) four in-flight requests.

Nothing about this needs a third party to operate an inference tier. It needs something to own the socket while the run sleeps — and a server process is something. The two pieces it is built from already existed: durable suspend/resume (awaitSignalSuspending, built for human-in-the-loop waits) and a leased work queue with a coordinated drainer (the same shape the admission queue has).

The cost, so you can decide per call. A suspend/resume round trip adds the dispatcher's poll interval (250 ms) plus one engine wake. On a six-second call that is under 5%; on a 200 ms classification call it doubles the latency. So it is a mode, not a default: offload the calls that are slow enough for a worker to be worth freeing — which is most of them — and leave the fast ones inline.

What the queue guarantees.

The enqueue is idempotent. The row id is derived from the execution and the step name, so a replay cannot queue — and pay for — the same call twice.
The claim is a conditional update, not a read-then-write. Two dispatchers cannot both perform (and both bill) one call.
The order is perform → resume the run → mark the row. A crash between the resume and the mark leaves a row whose lease expires and is reclaimed, and the second resume of a resolved deferred is a no-op. The other order would leave a run waiting for a signal nobody will send again.
A give-up resumes the run with the failure. A queued call that was abandoned without telling its run is the one unrecoverable outcome here, and the ordering exists to rule it out.
Retries follow the same rules as the inline policy, Retry-After included, so the two modes do not back off differently.

aiObjectStep({ offload: true }) renders your schema to JSON Schema for the dispatcher — a JavaScript Schema cannot be journaled — and still decodes on the awaiting side, where the real schema exists.

The Flow tab shows the queue: what is waiting and for how long, which calls have been waiting more than two minutes, which claims have a lease their dispatcher will never release, and the dispatcher's own last tick. A run parked on an offloaded call reads suspended in the run list with no step row yet, so this is the only view of the wait while it is happening.

Budget SUSPEND — stop spending without destroying the run

requireAiBudget above fails ONE call. That does stop the spend, and it does it by killing a durable run that may be nine steps in — the work is lost, and lost again on every retry until somebody raises the limit. A ceiling whose only expression is destruction gets set high, or turned off. defineCostBudget sits at the other extreme: it is an observability-grade signal over work that already happened, and never blocks anything.

The third answer is the one the workflow engine already knows how to do for a human: suspend. aiStep / aiObjectStep take a budget:

import { aiStep } from '@voltro/ai/workflow'

const summary = yield* aiStep({
  name: 'summarise-thread',
  prompt: `Summarise:\n${thread}`,
  store: ctx.store,
  tenantId,
  budget: {
    limitUsd:    50,
    estimateUsd: 0.25,
    onExceeded:  'suspend',
  },
})

Over the cap, the run parks on a durable _voltro_budget_holds row, frees its worker, and resumes when the budget has headroom — then continues from where it stopped. Nothing is spent while it is held, and nothing is lost.

The ordering is the feature. The gate reads the reservation counter before the journaled step and before an offloaded call is enqueued — not a sum over _voltro_ai_usage, which by definition only knows about money already gone. On the suspend path no provider is contacted and no queue row exists for a dispatcher to pick up. Under the cap, estimateUsd is RESERVED atomically before the call, which is the difference between a ceiling and a speed bump.

A release wakes a run; it does not authorise a spend. Every wake re-reads the budget and parks again if it is still over, so an operator lifting the wrong hold — or a window rolling over for a tenant that immediately spends again — cannot spend through the ceiling. Three things can wake a hold:

its own durable recheck clock (15 minutes by default, recheckEveryMs), so a tumbling window that rolls over on a clock nothing notifies us about is still noticed;
releaseBudgetHolds({ store, budget, tenantId? }), called from your own code — an admin mutation, or a subscriber on defineCostBudget's recovered signal. The framework does not subscribe for you: whether a compute budget recovering should wake AI holds is an app decision, and the recheck clock already guarantees the run is not stranded either way;
its total timeout (holdTimeoutMs, 7 days), after which the run fails having spent nothing.

onExceeded: 'fail' is the default, so an existing budget behaves exactly like requireAiBudget. A compute budget opts in the same way:

import { defineCostBudget } from '@voltro/runtime'

export default defineCostBudget({
  name:       'tenant-recompute-hourly',
  unit:       'recompute',
  limit:      100_000,
  window:     '1h',
  onExceeded: 'suspend',   // default 'observe' — signal only
})

Held runs are visible via pendingBudgetHolds(store).

Deliberately your call

The toolkit prices + records + gates; a few things stay explicit by design:

Recording the LEDGER is opt-in per call. The free functions (generateText etc.) have no store or tenant, so they can't self-write the _voltro_ai_usage row — call recordAiUsage where you have ctx (an agent send handler is the natural spot). The metrics above ARE automatic; the durable per-row ledger is the opt-in part.
The built-in prices are point-in-time list prices. Call setModelPricing to override app-wide, pass price per call for negotiated rates, or actualCostUsd (from gatewayCostUsd) for the gateway's real per-call charge.

For quota tied to billing TIERS (not a raw USD cap), see @voltro/plugin-billing's entitlements — requireEntitlement(ctx, 'aiCalls', n).

Cost tracking

#Token usage on every call

#Token usage on a streamed run

#Pricing a call — estimateCostUsd

#The built-in prices are point-in-time defaults — override them app-wide

#Real gateway cost — gatewayCostUsd + actualCostUsd

#The usage ledger — recordAiUsage + aiUsageTable

#Spend + budgets — aiSpendUsd / requireAiBudget

#Per-call observability — automatic spans + metrics

#Semantic caching — zero-token hits

#Framework-managed — cacheSemantic: true

#Dependency-driven eviction — it never serves a stale answer

#The honest bounds

#A model call inside a workflow — aiStep

#Offloading the call — offload: true

#Budget SUSPEND — stop spending without destroying the run

#Deliberately your call