Transactional outbox

ctx.outbox.enqueue — a reliable external side effect from a mutation, committed in the same transaction as the write that caused it.

A mutation must not do external I/O. It runs in a transaction, and an HTTP call cannot be rolled back — if the request succeeds and the transaction then fails, you have charged a card for an order that does not exist.

So "write this row and sync it to Jira" has no correct one-step form. The usual workaround is to build one: a deliveries table written inside the transaction, a cron that drains it, and a worker with backoff and a dead-letter. That is a real subsystem, and every integration app rebuilds it.

ctx.outbox is that subsystem, as a one-liner.

// apps/api/mutations/ticket.create.server.ts
export default async (input: { title: string }, ctx: AppContext) => {
  const ticket = await ctx.store.insert('tickets', { title: input.title })

  await ctx.outbox.enqueue('jira.sync', { ticketId: ticket.id }, {
    idempotencyKey: `jira.sync:${ticket.id}`,
  })

  return { id: ticket.id }
}

Why this is correct, not just convenient

enqueue writes through ctx.store — and inside a mutation, ctx.store is the transactional view. The outbox row commits in the same transaction as the domain write, or neither does.

That is the whole guarantee. There is no window in which the ticket exists and the intent to sync it was lost, because losing the intent means the ticket was rolled back too.

This is what separates it from reacting to a change after commit. A post-commit tap — including @voltro/plugin-cdc-out, which says so plainly — is at-least-once from enqueue: a crash between the commit and the tap loses the event. Here, enqueue cannot be lost.

Delivery after commit is still at-least-once. That is the strongest guarantee available without a distributed transaction into the target system, so:

Handlers must be idempotent. A process that dies between "the remote accepted it" and "we recorded that" will retry.

On several replicas, that used to be the smaller reason. Every replica runs the drain, and the drain read every pending row — so an effect was dispatched once per replica, on the happy path, every time. It is claimed now: a row moves pending → delivering in one atomic statement stamped with the claiming process, so a racing replica loses the row rather than duplicating it, and a claim whose holder stops responding is returned to the queue after its lease (claimLeaseMs, default 5 minutes — raise it above your slowest handler). The worker on the surviving replicas is armed for the moment that lease runs out, so the row is delivered then even when nothing else writes to the outbox.

That removes the routine duplicate. It does not make delivery exactly-once, and nothing can: the process can still die between the remote accepting and the row being marked. The idempotency requirement stands — it is now about the failure case it was always meant to describe, rather than about every single delivery.

Declaring the handler

One *.outbox.ts file per effect:

// apps/api/outbox/jira.sync.outbox.ts
import { defineOutboxHandler } from '@voltro/runtime'

export default defineOutboxHandler({
  effect: 'jira.sync',
  maxAttempts: 5,
  handler: async ({ payload, attempt, subjectId, traceId }) => {
    await jira.syncIssue(payload['ticketId'] as string)
  },
})

The handler runs after the enqueuing transaction committed, outside it, and may do external I/O — that is the point. It receives the payload, the attempt number, and the subject / tenant / trace of whoever enqueued it.

Two handlers claiming the same effect is refused at boot with both filenames, rather than letting whichever loaded last silently win.

Effects and the service environment

Handlers may return an Effect directly or a Promise. Both voltro dev and voltro serve execute the Effect with the app's plugin services, layers:, cache, KV, analytics and outbound HttpClient. User layers override plugin layers of the same service. The worker starts only after that environment is built. Provide a Delivery layer in the API's app.config.ts for this example:

import { Context, Effect } from 'effect'
import { defineOutboxHandler } from '@voltro/runtime'

export class Delivery extends Context.Tag('app/Delivery')<Delivery, {
  readonly send: (id: string, payload: Readonly<Record<string, unknown>>) => Effect.Effect<void, Error>
}>() {}

export default defineOutboxHandler({
  effect: 'remote.sync',
  handler: ({ outboxId, payload }) => Effect.gen(function* () {
    const delivery = yield* Delivery
    yield* delivery.send(outboxId, payload)
    return { accepted: true }
  }),
})

Each attempt owns an Effect scope; finalizers finish before delivery is marked successful. Typed failures, defects, interruptions and missing services enter the normal retry/dead-letter path — an Effect object is never itself a success. An async handler that returns an Effect is executed too. Avoid a nested Effect.runPromise: return the Effect so its services remain available.

outboxId is the stable intent identity across attempts; forward it as the remote system's idempotency key. idempotencyKey is the original enqueue key or null, and can be reused after a prior intent succeeds, so it is not an equivalent permanent identity. Delivery remains at-least-once.

The queued subjectId and tenantId are attribution, not refreshed authority. There is no request SubjectService, request EffectStore, or automatic aggregate/workflow registry in this environment. Services needing those must explicitly resolve current authorization and tenant-scoped access. The worker does not turn an old queued user ID into an unscoped store permission.

Low-level drainOutbox callers with service-requiring handlers must pass runEffect(effect, context) to provide their environment. Service-free Effects can run without that adapter. Direct calls/tests constructing OutboxHandlerContext must include outboxId and idempotencyKey.

Retries, backoff, dead-letter

Retry schedule exponential — 1s, 2s, 4s … capped at 5 minutes
Default attempts 8 (maxAttempts on the handler, or per-enqueue)
Exhausted row moves to dead, logged at ERROR, stays in the table
Unknown effect left pending, never discarded

That last row matters. The usual cause of an unknown effect is a deploy where the enqueuing code shipped ahead of its handler. Dead-lettering those would turn a rollout ordering detail into permanent loss of a side effect the app believes happened, so they wait instead.

A dead-lettered row is not deleted — it is queryable in _voltro_outbox with its lastError, because a dead letter is a side effect your app thinks occurred and which never will.

Delivery, and why there is both a nudge and a poll

When the transaction commits, the worker is nudged and the effect usually goes out in milliseconds. A poll also runs every 5 seconds.

The nudge is an optimisation. The poll is the contract: it picks up rows whose nudge was lost because the process died between commit and delivery, rows enqueued by another replica, and rows waiting out a backoff. Without it the guarantee degrades to "delivered unless something went wrong" — which is the exact case a durable outbox exists for.

The 5 seconds is a floor, not a rate. Those three reasons are taken one at a time, and only the last needs a clock:

the poll exists for… what brings the loop back
a nudge lost to a dead process the first pass at boot, which is not deferred behind a tick
a row another replica enqueued a change event on _voltro_outbox
a row waiting out a backoff the runner arms for that row's own nextAttemptAt

So on an empty queue the timer stops entirely and the loop waits to be woken. Where no change channel is available it keeps the fixed 5-second tick instead — the poll is then the only thing that can notice another replica's row, and a durable outbox that stops looking is worse than one that polls.

Options

await ctx.outbox.enqueue('mail.welcome', { userId }, {
  idempotencyKey: `welcome:${userId}`,  // drop if an undelivered row has this key
  maxAttempts: 3,                        // override the handler's default
  delayMs: 60_000,                       // don't attempt before then
})

idempotencyKey dedupes against rows that have not yet succeeded, so a retried mutation does not produce a second side effect. A delivered key is deliberately not a blocker — reusing a key later means "do it again", and treating it as permanently consumed would silently swallow a legitimate request.

enqueue returns the outbox row id, which is also the delivery id: persist it alongside your row and a client can watch the effect's progress.

Delivery history

The queue answers "is this still owed". It cannot answer "what did the remote say on attempt 3", "how long did it take", or "who resent it" — which is what a delivery-history screen renders. So every attempt appends a row to _voltro_outbox_attempts:

column
outboxId the entry this attempt belongs to
effect denormalised — the history stays readable after the entry is purged
attempt 1-indexed, monotonic across the entry's whole life
trigger automatic (the drain) or manual (a resend)
triggeredBy / reason who asked for a manual resend, and why
outcome delivered · failed · dead (dead = the attempt that exhausted the budget)
startedAt / finishedAt / durationMs timing
error failure message (clipped)
response whatever the handler returned, as JSON
subjectId / tenantId / traceId carried from the entry

response is how per-attempt transport detail gets recorded without the framework pretending to model HTTP — return { status, body } from the handler and the history has it:

export default defineOutboxHandler({
  effect: 'jira.sync',
  handler: async ({ payload }) => {
    const res = await fetch(url, { method: 'POST', body: JSON.stringify(payload) })
    if (!res.ok) throw new Error(`jira ${res.status}`)
    return { status: res.status, body: (await res.text()).slice(0, 500) }
  },
})

It is a framework table like any other, so reading it is a normal store read — there is no separate history API to learn:

const history = await ctx.store.query({
  table: '_voltro_outbox_attempts',
  predicate: eq('outboxId', deliveryId),
  order: [{ column: 'attempt', direction: 'desc' }],
  take: 20,
})

Attempts are recorded best-effort: if the log write fails, the delivery still succeeds. History must never be able to break the thing it observes.

Resending on demand

await ctx.outbox.resend(deliveryId, { reason: 'customer never received it' })

Re-arms one entry for delivery now — including one that already reached dead. This is legitimate precisely because delivery is already at-least-once and handlers are therefore already required to be idempotent: a resend is the same hazard the contract obliges them to absorb, not a new one.

What it must never be is invisible. The resulting attempt is recorded with trigger: 'manual' plus the requesting subject and reason, so an operator redelivery can never be mistaken for a backoff retry when someone later asks why the remote saw the effect twice. The entry also carries a resendCount.

Three behaviours worth knowing:

  • It re-arms the existing entry rather than enqueuing a copy. A copy would carry the same idempotencyKey, duplicate the payload, and split one entry's history across two ids.
  • It grants a small fresh budget (attempts, default 1). A dead entry has already spent its allowance, so without this it would re-die untried — and "try again now" is what the button means, not "restart the whole schedule".
  • It refuses an entry whose attempt is in flight, where re-arming would race the running attempt into a double delivery.

Inspecting the outbox in dashboards

Open Automation → Outbox in either the Cloud app inspector or local dashboard. Filter waiting, delivering, delivered or exhausted entries; pages hold at most 50 entries. Expand an entry for its tenant, original subject, trace and up to 50 retained delivery attempts, including errors, duration and manual-request reasons. Payloads and handler responses are omitted. The queue is read from the shared store; the list of registered handlers belongs to the responding process.

Request another attempt requires inspect write access, confirmation and a reason of at most 500 characters. It re-arms the existing entry immediately with one additional attempt. A delivered effect can run again. The action refuses an in-flight entry or a handler missing from this process. Its acknowledgment means that another attempt was requested; check the history for the delivery outcome. The local view polls; use Refresh views in Cloud to read the latest result. Cloud owners and admins can recover entries when write credentials are configured; ordinary members cannot read the outbox history or request retries.

Manual requests use a separate resendRequestedAt marker, so an inspect request without an app-user identity is still recorded as trigger: 'manual' with triggeredBy: null. The marker is consumed after that attempt. Re-arming compares the stored status, attempt count and resend count atomically; a concurrent worker claim or another resend causes the request to fail instead of overwriting it.

The shared development/production routes are GET /_voltro/inspect/outbox, GET /_voltro/inspect/outbox/:id/attempts and POST /_voltro/inspect/outbox/:id/resend. The POST body is { "reason": "Destination restored", "confirm": true }.

Retention

Two bounds, because an unbounded attempt log is a slow-motion outage:

  • A per-entry cap (50 attempts) trims the oldest as new ones arrive. The automatic path can't reach it — maxAttempts defaults to 8 — so this bounds the one case that is genuinely unbounded: an entry resent by hand for years. It works on every dialect.
  • A time bound on the boot sweep purges attempts older than 30 days (VOLTRO_OUTBOX_ATTEMPTS_TTL_HOURS), plus delivered queue rows (VOLTRO_OUTBOX_TTL_HOURS). dead and pending entries are never aged out — a dead letter is an unresolved incident and a pending one is still owed.

The time sweep runs on postgres; on other dialects the per-entry cap is the bound.

When NOT to use it

  • Work that must be observable step-by-step, or that suspends → a workflow. The outbox delivers one effect; it is not a durable multi-step process.
  • Reacting to any change on a table, not to one mutation's intent → defineSubscriber or defineReaction.
  • Mirroring a table outward continuously → @voltro/plugin-cdc-out, which is built for reverse-ETL with per-pipe ordering.

There is no generic job queue — take X for Y

Voltro deliberately ships no defineJob primitive (priorities, worker pools, a BullMQ equivalent). The outbox, workflows and schedules cover the cases between them, and a third durability primitive would drift from both. What to reach for instead:

you want… take
a concurrency-limited worker pool workflows + declarative flow control — concurrency / throttle bound how many runs execute at once
true priority scheduling (high-priority work overtakes queued low-priority work) does not exist as a primitive — a workflow draining your own queue table in your priority order is the honest build
a delayed / scheduled message delayMs on enqueue (above) for a one-off delayed effect; a schedule for recurring time-based work; workflow sleep for a pause inside a durable process
exactly-once delivery does not exist — delivery is at-least-once everywhere, which is the strongest guarantee available without a distributed transaction into the target; idempotent handlers are mandatory (see above)

See also

The dashboard resend form ignores repeated submissions while its request is pending. On failure it retains the reason and allows a deliberate retry. A currently delivering entry cannot be resubmitted through the form.