Declarative flow control

debounce, singleton, concurrency, throttle, rateLimit, batch, priority, timeouts and onFailure — declared on the workflow, enforced before the run exists.

Everything on this page is declared on workflow({...}) and enforced at the admission boundary — the moment ctx.workflows.start(...) is called, before a durable run exists.

That timing is the whole point. Once a run is enqueued, the only tools left are cancel and sleep, and neither of them un-spends the durable entity. So "run this at most once per row per fifteen minutes" cannot be a primitive you call inside the body; it has to be a property of the declaration.

Not the same as Flow control. That page covers durableQueue / processQueue / rateLimit — primitives you call inside a running workflow to bound the work it fans out. This page is about whether the run starts at all. They compose: a workflow can declare concurrency here and still use a durable queue in its body.

The shape

import { Schema } from 'effect'
import { workflow } from '@voltro/workflow/define'

export const tourNarration = workflow({
  name: 'tourNarration',
  payload: Schema.Struct({
    rowId: Schema.String,
    tenantId: Schema.String,
    editedAt: Schema.Number,
  }),
  success: Schema.Void,
  idempotencyKey: ({ rowId, editedAt }) => `tour:${rowId}:${editedAt}`,

  debounce: { key: (p) => `tour:${p.rowId}`, period: '15 minutes' },
  concurrency: { limit: 5, key: (p) => p.tenantId },
  timeouts: { start: '1 hour', finish: '10 minutes' },
  onFailure: 'narrationFailed',
})

Every key callback receives the workflow's own decoded payload type. A misspelled field is a compile error, not a key that quietly becomes the string "undefined" and collapses every row in your deployment into one bucket.

idempotencyKey is the execution's identity — read this first

This is the single most expensive misunderstanding in the workflow API, and getting it wrong produces a design that looks right and silently stops working.

idempotencyKey is not a dedupe window. It is the execution's identity, permanently:

const a = yield* wf.execute({ id: 'same' })
const b = yield* wf.execute({ id: 'same' })   // does NOT run — replays a's result

After the run completes the key is spent. A later, genuinely new invocation under that key is a silent no-op that returns the old output. Nothing errors and nothing logs, because from the engine's point of view you asked for a run it already has.

So a key must be unique per unit of work you want to happen:

`tour:${rowId}` wrong if the tour can ever be re-narrated
`tour:${rowId}:${editedAt}` right — every edit is a new unit of work

And a flow-control key is a different thing

Conflating the two is what makes "I need to re-arm a key" feel like a missing feature. It is not missing; it is two fields:

  • idempotencyKey — the execution's identity. Varies per unit of work.
  • debounce.key / singleton.key / concurrency.key — the resource runs compete for. Stable.

"One job, fifteen minutes after the last edit, latest state wins" is then the example at the top of this page: twenty edits mint twenty identities, and exactly one is ever admitted. There is no restart: true in this API because separating the two keys is the mechanism it would have been.

debounce — collapse a burst

debounce: { key: (p) => `tour:${p.rowId}`, period: '15 minutes', timeout: '1 hour' }

Starts sharing a key collapse into one pending row. The timer resets on every arrival, and the latest payload wins — which is what "narrate what settled" means.

timeout is a hard cap measured from the first start in the burst. It is optional and uncapped when unset, which is a real trade: an unbroken stream of starts arriving faster than period defers the run forever. We do not invent a default cap and we do not warn — instead the starvation is a number you can see:

voltro workflows flow

A waiting= climbing past a few multiples of your period is the signal. Set timeout when you see it, or from the start if the burst is user-driven.

singleton — one run per resource

singleton: { key: (p) => p.tenantId, mode: 'skip' }    // newcomer stands down
singleton: { key: (p) => p.tenantId, mode: 'cancel' }  // newcomer evicts the incumbent

There is no default mode: 'skip' discards the incoming request and 'cancel' discards the running one, and choosing for you would silently throw away work either way.

Under 'skip', start() returns the incumbent's handle — a real, pollable run. Under 'cancel', the incumbent is cancelled when the replacement actually starts, not when it is queued. That matters when you also declare debounce: evicting at queue time would leave the whole quiet period with the old run dead and the new one not yet begun.

concurrency — bound what is in flight

concurrency: { limit: 5, key: (p) => p.tenantId }

At most limit runs in flight per key, across every replica — the count rides the shared admissions ledger, so three replicas with limit: 5 are five runs, not fifteen. Excess starts queue and are admitted as slots free.

pool — share one budget across workflows

// embeddings.workflow.tsx
concurrency: { limit: 10, pool: 'openai' }
// summarize.workflow.tsx
concurrency: { limit: 10, pool: 'openai' }

Without pool, the limit bounds this workflow's runs. With it, every workflow declaring the same pool name competes for one budget — the shape a rate-limited provider forces: five workflows that each call OpenAI must share ten slots, not hold ten each.

key still partitions within the pool: give each member key: (p) => p.tenantId and the shared budget applies per tenant.

The admission ledger stores concurrency keys as JSON tuples [pool, key], using null for an unpooled key. Control characters are escaped for SQL text storage; pool names and application keys cannot collide by containing a separator. Treat this stored representation as diagnostic data, not an application-generated ID.

For custom runtime adapters, reserveAdmissionSubmission is a low-level database-only reservation primitive. Supply a stable submission ID and the schema-encoded complete engine payload. It atomically stores that input and one admission lease; retrying the same ID returns the original payload, caller and selected intent IDs before checking capacity again. An ID cannot be reused by another workflow or tenant. Keep this table beside the resident queue. The primitive does not invoke an engine, consume queue rows or acknowledge an execution; those remain dispatcher responsibilities. It is not an exactly-once execution guarantee, and the framework drainer is not yet wired to this primitive.

After observing engine acceptance, acknowledgeAdmissionSubmission atomically links the submission and its lease to the execution ID and optional run ID. Repeated acknowledgements are safe; a different execution or run is refused. A failed write rolls back both links. This records observed acceptance, not successful workflow completion or proof that an unacknowledged call never ran.

Low-level dispatchers must distinguish a reusable debounce slot from a caller's receipt. recordWorkflowStartReceipt stores a separate stable caller ID in the same transaction as the pending write. Allocate that ID before transaction retries. bindWorkflowStartReceipts binds the waiting receipts to a persisted submission in the transaction that takes the queue rows. The binding is paged and records its completion: retrying an old submission cannot collect receipts from a later burst occupying the same slot. The framework start adapters now persist these caller receipts; dispatcher binding and the waiting protocol are not yet wired through these helpers.

takePendingAdmissionSubmission combines reservation, receipt binding and queue removal in one resident database transaction. Supply a stable submission ID and the selected pending IDs. Its synchronous prepare callback receives the current encoded rows read inside that transaction; decode and assemble the engine input there, without external I/O. A missing, postponed or dead-lettered selected row returns selection-unavailable without taking a partial batch. A previously transferred submission is returned before reading reusable queue slots, so its retry cannot consume a later burst. A failed binding or deletion rolls back the entire transfer. This helper does not dispatch the engine. The gate now recovers completed, unacknowledged transfers before processing its pending queue, even when that queue is empty. It retries the persisted input through the admitted start path and acknowledges observed acceptance. Failed starts stay recoverable; the bounded recovery cursor advances past failures so a full failing page cannot starve later submissions. pendingAdmissionSubmissions(store, limit, afterId) exposes the same recovery window to custom adapters. Workflow idempotency remains the engine's execution identity; this is not proof of exactly-once effects. Ordinary controlled drains and admitted delayed arrivals now produce these submissions before engine I/O. The submission failure lifecycle, operator-discard and timeout receipt outcomes, and receipt waiting remain unfinished. Custom AdmissionDrainDeps adapters must provide transaction(work) using withAdmissionTransaction on the resident store, preserving payload codecs and serializing with direct arrivals. The callback is database-only and may replay; calling work(store) without a transaction does not satisfy this contract.

Submission retries now use the same recovery path in standalone drainTick and the CLI gate. Attempt count, last error and the next attempt time live in the submission's retry field, not in its consumed queue row. An acknowledged execution wins over a late failure report. Missing acknowledgement does not prove non-execution, so this path must not report abandonment using the old intent. Terminal failure handling remains unfinished. If a queue read fails after completed recovery, the outcome retains that recovery's counters, adds a failure and requests another tick; examined: false marks the incomplete pass.

At workflows.deadLetterAfterAttempts, an unacknowledged submission stores retry.stoppedAt and stops automatic retry. This is unknown acceptance, not a dead letter: it does not invoke onFailure, release its concurrency slot or erase caller receipts. The normal 24-hour lease backstop cannot free unknown acceptance either. A later proven engine acknowledgement can still bind the execution; changing the retry limit does not restart an already stopped record. false leaves automatic retry unlimited. Inspect/reconciliation commands for these stopped submissions are still unfinished; the pending-intent retry command does not operate on them.

Recovery isolates failures to persist retry metadata per submission, reports both the dispatch and recording error, and continues with the rest of its page. Already acknowledged starts remain counted. A full page requests the next page promptly; at the end of the scan only actual retry deadlines remain. Thus a full page of stopped records does not continuously schedule itself, and future retry deadlines found on an earlier page are not lost.

Malformed submission metadata is validated per record during recovery. It counts as a failure, remains stored and is not sent to the engine; valid neighbors and later pages still run. The warning identifies the submission without printing malformed values. The public typed pendingAdmissionSubmissions reader remains strict and rejects a malformed page instead of silently omitting records. Recovery isolation is not automatic repair of damaged data.

A singleton reservation with unknown acceptance also blocks new starts under that key, including after the lease backstop. Both skip and cancel return a queued receipt with deferral mode singleton until the incumbent's execution is confirmed. Voltro cannot supply a skipped execution or cancel an invented ID. Once confirmed, the declared skip/cancel behavior applies. The queue keeps the actual singleton key; unrelated keys remain independent.

Every member of a pool must declare the same limit — the boot fails otherwise. Two limits for one budget is a contradiction, and silently picking either would enforce a number somebody did not write.

throttle vs rateLimit — late, or gone

They are mutually exclusive, and declaring both is a boot error.

throttle:  { limit: 100, period: '1 minute', key: (p) => p.tenantId }  // QUEUES the excess
rateLimit: { limit: 100, period: '1 minute', key: (p) => p.tenantId }  // DROPS the excess

Reach for throttle when every start must eventually run. Reach for rateLimit when the excess is genuinely surplus and running it late is worse than not running it.

A dropped start is never silent: start() resolves to a handle with status: 'dropped' and a retryAfterMs, and the drop is a row in the admissions ledger with its key and reason.

throttle has no burst knob. The window is sliding, so its maximum instantaneous burst is already exactly limit; a separate knob could only duplicate it.

batch — many starts, one run

export const refreshIssues = workflow({
  name: 'refreshIssues',
  payload: Schema.Struct({ items: Schema.Array(IssueRef) }),
  success: Schema.Void,
  idempotencyKey: ({ items }) => `refresh:${items.length}:${items[0]?.issueKey ?? ''}`,
  batch: { item: IssueRef, key: (i) => i.tenantId, maxSize: 100, timeout: '30 seconds' },
})

await ctx.workflows.start('refreshIssues', { tenantId, issueKey: 'ABC-1' })

Callers start it with a single item; the workflow's own payload is the batch shape. That mismatch is checked at declaration time — a decode failure on the batching replica is a failure nobody is watching.

batch.item is therefore also what an arriving start is validated against. A caller's payload is judged by the item schema; the run the drainer eventually starts is judged by the workflow's own payload. Two schemas, because there are genuinely two shapes — and a WorkflowPayloadError naming items on a start() call would be the framework asking the caller for the batch it is supposed to be building.

The timeout is a deadline, not a quiet period: it does not reset per item, or a steady trickle would never flush. batch and debounce cannot both be declared for exactly that reason.

priority

priority: (p) => (p.urgent ? 100 : 0)

Higher runs first out of the pending queue. Ties break by arrival, so an all-default deployment is FIFO rather than dialect-dependent.

Scope — read before porting BullMQ priority lanes. priority orders the admission queue only: the pending rows a deferring control (debounce, batch, throttle, concurrency), a pause, or a delayed { at } start has parked. A start that is admitted immediately never competes with anything — it goes straight to the engine, whatever any other start's priority says. There are no cross-workflow priority lanes, no preemption of running work, and no ordering between two starts that both found a free slot. If urgent starts must overtake normal ones, put both classes behind the same concurrency limit (or a shared pool), so every start passes through the queue that priority orders.

timeouts

timeouts: { start: '1 hour', finish: '10 minutes' }
  • start bounds how long a start may sit in the admission queue, measured from the first arrival in its group. A debounced run that never gets a quiet moment is a job that silently did not happen.
  • finish bounds the run itself once admitted, and also tightens the crash backstop on its concurrency slot.

Both reach the same onFailure signal as an exhausted retry. timeouts.start removes the queued intent before a run exists. timeouts.finish durably changes the running row to cancelled; the cancel request carries its cause, and the transaction that makes the cancellation terminal reads it back and writes the onFailure report in the same commit. Seeing the same old row again does not emit another report, and a run that finished on its own before the cancel landed gets none. Declaring timeouts.start on a workflow that cannot defer is a boot error — it would never fire.

Each admission gate scans running rows in its own physical store for finish timeouts, across all result pages. A deployment-wide run listing is not the authority for a local gate. Dev and serve discover physical queues from the durable namespace directory, including after a restart without the writer's post-commit callback. Admission visits at most 100 namespace entries per source per tick and continues on the next tick. New arrivals use their caller's current resident store; changing tenant ownership does not authorize consuming its old queue. Every queue uses the same deployment-wide pool coordinator. Pool release recovery requires retained resident release evidence, not a timeout or missing run history. Operator pauses apply across all these queues.

If a later namespace page remains unreadable and its cursor stops advancing, discovery restarts from the roots. Healthy queues are revisited rather than starved behind the failed page. This restart does not turn incomplete discovery into a successful report; the unavailable source remains reported.

Custom low-level admission hosts can provide AdmissionDrainDeps.readPauses to combine their coordinator and resident pause sets. It is read outside resident transactions; a local transactional recheck can add a pause, not remove a host-wide one. Normal workflow declarations need no additional configuration.

Physical admission keeps polling because the primary store's change channel cannot prove that it sees every regional or namespace write. Partial discovery, release or drain failures produce one workflow admission: physical recovery incomplete warning; accessible placements continue. The warning carries the same numbers the verdict is made from — discoveryProblems, placementFailures (each with its placement, phase and the database's own reason), releaseRowFailures and drainFailures — and it is written when the picture changes, not on every tick: a condition that stays the same second after second is logged once, a different one is logged again, and workflow admission: physical recovery healthy again closes it. The last admission report stays readable at any time and distinguishes discovery completion from successful processing.

A recovery failure is not idle success, and it is not work either. A tick that found no admissible row but could not open a placement, read a release census page or complete a namespace page reports idle: true beside its placementFailures, discoveryProblems and releaseFailures counts, and the scheduler backs off to the same five-second reconciliation bound a healthy idle process reaches. The failing placement is retried there, never disarmed, and never at the base interval for as long as the condition lasts: a condition that does not clear by itself used to hold the admission task at one tick per second and write one claim row per second with it. Incomplete discovery is different — a further page is known work, so the next tick follows at the base interval.

The release census that frees a pooled workflow's global permit pages the resident submissions in the store's own order and cursor semantics; a page the store returns is walked as given, and only a page that repeats its cursor row or an identity within itself is refused as a stalled cursor, naming both identities. A census page that fails restarts that placement's census from the start on the next tick instead of re-reading the failing page: an owner released behind the cursor since is visited again, and the failure stays reported.

Cancellation and staleness use independent paginated visits to the same physical placements. A cancellation event is matched against its resident runs and pending inputs, not against a deployment-wide run list. Each journal keeps its own watermark; the aggregate cancellation report therefore has no single watermark. Placement failures appear in the sweep's failures and do not stop reachable stores. Stall events are recorded in the same store as the stalled run, allowing the next visit to deduplicate them. The engine-wide unread-message diagnostic still runs once per scheduled tick, not once per namespace. These changes do not yet make the queue inspect listing a complete cross-placement inventory.

Within each store, staleness scans advance in (startedAt, runId) order, bounded by runPage. Already reported old runs cannot permanently hide later pages. Failed candidate reads retain the cursor; exhaustion restarts the census so runs can be examined again. The built-in host manages these cursors. Low-level sweepStalledRuns callers pass the returned nextRun as afterRun on the next visit to the same store; retain their previous cursor on candidate-read failure and omit it after successful exhaustion. A full page is not proof that another run exists: an empty final page may be needed.

For a bounded view across physical stores, use the authenticated endpoint in both voltro dev and voltro serve:

curl -G "$API_URL/_voltro/inspect/workflows/flow-control/physical" \
  -H "Authorization: Bearer $VOLTRO_INSPECT_TOKEN" \
  --data-urlencode 'limit=100'

Read data.placements: each entry identifies its placementId and contains its own flowControl sample, without queued payloads or caller contexts. rowWindowMayBeLimited warns that an intent or admission window filled. To continue directory discovery, send the non-null data.next string using --data-urlencode "after=$NEXT". A null cursor ends discovery, not necessarily the row inventory. Check data.failures independently of discoveryComplete: a visited placement may be unreadable even when discovery finished. The inspect envelope remains explicitly incomplete because these are registered-placement samples, not an exhaustive queue census.

Copy an entry's placementId into Physical queue placement in either dashboard and choose Load placement. Empty selects the primary store. Queue reads, input validation, retry and discard use that selection; workflow pause/resume remains deployment-wide. Changing placement clears open action confirmations. The server resolves current placement ownership again for each operation and refuses an unavailable or moved placement instead of falling back to the primary store. The panel refuses actions if the returned placement does not match its selection.

For direct calls, add placementId to the query string of /workflows/flow-control and /workflows/flow-control/validate, or to the JSON body of /workflows/flow-control/retry and /workflows/flow-control/discard under /_voltro/inspect. Writes require both the read bearer token and the separate x-voltro-inspect-write factor. The body still names intentId. A rejected placement returns HTTP 409. Omitting placementId selects the primary store; it never searches for a matching intent in other stores. Bulk run operations and exhaustive cross-placement row pagination remain separate work; selecting a queue placement does not change their scope.

Run-history HTTP reads accept the same placementId query parameter: /workflows/runs, /workflows/stats, /workflows/runs/:id/steps, /workflows/runs/:id/events, and /workflows/children?parentExecutionId=.... The selected identity is echoed in data.placementId. Each read revalidates ownership; an unavailable or moved placement returns HTTP 409, never a primary row with the same ID. These are direct HTTP options; the admission panel's selection does not automatically select a placement in the separate run-history UI.

Discard is durable: the resident admission transaction records the decision, settles unbound caller receipts as cancelled and deletes the pending input together. A failed audit or receipt write rolls back the discard, rather than deleting work without evidence. Low-level discardIntent requires a store with transactional and insertIgnoreWithOutcome. Inside an existing withAdmissionTransaction, call discardIntentInTransaction on its transaction store; never nest admission transactions. Neither operation calls the engine.

When finish lands. The bound is enforced by the admission drainer's tick, not by a timer on the run. Each visited placement contributes its earliest known finish deadline. A placement beyond the current discovery page is checked on a later tick; with many namespaces, allow for a full discovery cycle rather than assuming a one-tick maximum. Pending work is not required to keep this check running.

onFailure — the signal that replaces the sweep

onFailure: 'narrationFailed'

A workflow name, not a function. A closure cannot be journaled: the failure may be noticed by a different replica, minutes later, after the process that held it is gone.

It fires for every way a run fails to deliver — not only an exhausted retry:

  • the body failed and retries are spent — raised from the run's terminal transition, when its row turns failed
  • timeouts.finish cancelled an overrunning run
  • timeouts.start expired a start that never got a slot
  • the workflow was renamed away while starts were queued

The last two produce no run row at all, which is exactly why polling listRuns({ status: 'failed' }) could never see them.

The named workflow's payload is WorkflowFailureReport:

export const narrationFailed = workflow({
  name: 'narrationFailed',
  payload: Schema.Struct({
    workflow: Schema.String,
    payload: Schema.Unknown,
    errorTag: Schema.NullOr(Schema.String),
    errorMessage: Schema.NullOr(Schema.String),
    runId: Schema.NullOr(Schema.String),
    executionId: Schema.NullOr(Schema.String),
    reason: Schema.String,
    failedAt: Schema.Number,
  }),
  success: Schema.Void,
  idempotencyKey: ({ workflow, runId, failedAt }) =>
    runId === null ? `failed-before-run:${workflow}:${failedAt}` : `failed-run:${runId}`,
})

For failures of an existing run — including timeouts.finish — runId, executionId, and the original payload are populated. Use runId as the stable idempotency identity. Pre-run failures (timeouts.start and a queued start whose workflow was removed) have both IDs set to null; they are emitted once when their durable pending row is consumed.

A report is a row in _voltro_workflow_failure_reports, written in the transaction that decides the failure — a run's terminal transition, a queued start's expiry or dead-letter, a submission's rejection — and handed to the handler's arrival queue in that same commit. The failure recovery on every drain tick delivers it from the row: a process that dies anywhere between deciding and delivering leaves nothing to redo but the delivery. There is no process-local callback on this path, on purpose; a callback a dead process never calls is the one way a report could still be lost.

There is no onFailure for an onFailure — a handler that fails is logged and not re-notified, because the alternative is one run per failure per level with no floor.

encrypted

encrypted: true

Encrypts every durable copy of this workflow's data with the key behind .encrypted() columns — one key, one rotation story:

VOLTRO_FIELD_ENCRYPTION_KEY=…   # voltro dev mints it; `voltro secret generate field-encryption` prints one

No plugin is needed; under governancePlugin({ fieldEncryption: { secretKey } }) the key is read from the secret named there. Rotating it keeps every sleeping run resumable — see rotating the key.

A workflow's data is not stored once. The durable engine journals the payload, the result and every step's result so a run can resume after a crash; the run row keeps the payload and the result for the dashboard and listRuns; the step journal keeps each step's input and output; a queued start keeps its payload until it is admitted; a failure report keeps the payload for its handler. For a workflow over personal data or free text, each of those is a copy outside the .encrypted() boundary of the table the value came from. With encrypted: true, all of them hold ciphertext:

Where What is sealed
engine journal — cluster_messages, cluster_replies each payload field, the result, a typed failure and a thrown error (a defect), the same for every step, and the value awaitSignal / awaitUpdate / awaitEvent / awaitSignalSuspending received
_voltro_workflow_runs each payload field, the result, the error message
_voltro_workflow_run_steps each step's input, output, error message and error cause
_voltro_workflow_run_events a signal's or update's value, an update's result, an error message — ids, names and tags stay readable
queued starts, submissions, failure reports, control results the payload, result or error message they copy, including a queued start's lastError

The payload stays an object: its field names (which the schema already publishes) remain readable, each value is a ciphertext. It works on every SQL dialect the engine runs on — PostgreSQL, MySQL, MariaDB, SQL Server, SQLite and Turso — because the engine stores what the workflow's schemas encode, and the declaration seals those encodings.

What the body and your readers see is unchanged. The body receives the values it was started with, a step's result replays as it was returned, a thrown error reaches the caller as it was thrown, and the readers the access rules already let in — the live run, step and event rows your app subscribes to, the client's workflow hooks, ctx.workflows.listRuns(...), the dashboard — receive the values, opened. The protection is at rest.

Error text is data. A schema error prints the value it rejected (Expected number, actual "…"), an HTTP client quotes the body it was refused with, a database error names the row. So an error's message is sealed wherever a value is: on the run, the step, the event, the failure report, the queued start. Its tag — the class name, or a typed failure's _tag — stays readable, because grouping, filtering and alerting run on it.

Nothing leaves the process in plaintext. Logs and error reporters are outside the database, and usually outside your infrastructure: a log line about an encrypted workflow's failure carries the error's class and says the full cause is recorded sealed on its row. Every server error is published with contentSensitive: true for an encrypted workflow, and the Sentry plugin then sends the class, the stack frames and the run's identity, not the message, the cause or a typed failure's fields — tagged voltro.contentWithheld, so the issue still groups, with a pointer to the run. A reporter of your own reads the same flag and can use withheldServerError(event) from @voltro/protocol.

Declaring it without a cipher is a boot refusal, not a warning, and an encode that finds no cipher fails rather than writing plaintext: a fallback would leave the declaration reading as protection while the rows are readable.

What it costs, measured with the real AES-256-GCM cipher: about 90 µs to seal and open a 1 KB value; a run of 20 steps with 1 KB results showed no difference beyond the noise of the machine (plain 127–173 ms, encrypted 108–191 ms per run, SQLite). A run event carrying a value costs one extra lookup of its run.

Turning it on is safe for runs already in flight. A value written in plaintext before the declaration — a run started earlier, a queued start — is still read. Turning it off again keeps sealed values readable as long as the cipher is configured.

What it hands on is encrypted too. An encrypted workflow's onFailure handler must be declared encrypted, or the boot refuses and names both: the failure report carries the payload and the error. The handler receives both opened and stores them sealed. A durable queue it enqueues into must be declared durableQueue({ …, encrypted: true }) — a queue journals its payload, result and failure like a workflow — or processQueue fails, naming the queue.

What it does not cover, so it is not assumed:

  • keys your code derives — debounce.key, batch.key, concurrency.key, rateLimit.key. They are stored as your functions return them. Derive them from ids. The engine stores only a hash of the idempotencyKey;
  • field NAMES of the payload — they are what your schema already publishes;
  • what your own code does with a value: a table a step writes to (use .encrypted() there), a log line your code writes, an HTTP call it makes;
  • a payload field declared with a property-signature transformation (a default, a renamed key) — refused at definition time, naming the field, rather than left plaintext. Apply the default in the body.

Before 0.82.0 this option was encryptSteps and covered the step journal only; voltro update renames it.

cancelOn — stop live work when a correlated event arrives

workflow({
  name: 'tourNarration',
  payload: { rowId: Schema.String, issueKey: Schema.String },
  idempotencyKey: (p) => `tour:${p.rowId}`,
  cancelOn: [{
    event: 'jira.issue.deleted',
    schema: JiraIssueDeleted,
    match: (event, payload) => event.issueKey === payload.issueKey,
  }],
})

Both sides are typed: event from the entry's own schema, payload from the workflow's.

Why it is a declaration and not a race inside the body. You can express "stop when the issue is deleted" with awaitEvent and an interrupt. That works while the body is running. It does not work while the run is sleeping for six hours, suspended on a signal, or still sitting in the admission queue — which is the case you wanted cancellation for. The event has to reach a run whose fiber is not executing anything, and only something outside the body can do that.

So it is swept: a coordinated tick reads events published since a durable watermark, resolves each declaring workflow's live runs, and cancels the ones that correlate.

The sweep holds this watermark when listing, cancellation or queued-discard delivery fails. It continues evaluating the current window and reports individual failures, then retries that window on the next tick; newer events cannot move a failed operation behind the cursor. The first tick persists its lookback boundary before reading the journal, so a failed cold start does not slide past old work. A persistently failing operation can delay later windows: inspect the reported failures rather than treating a held cursor as healthy idleness. This protects retryable failures; an adapter's resolved reply still must accurately distinguish durable acceptance from confirmed execution. The CLI also treats a false cancellation result as unconfirmed: it appears in cancelHealth().failures, does not increment cancelled, and does not advance the event window.

Cancellation and queued-discard callbacks receive the complete planner decision, including eventId, not only the event name. The CLI carries this ID in the cancellation detail and the queued-discard audit reason. Repeated evaluation of one publication retains that ID; a name alone cannot distinguish two publications. This causal identity does not itself prove control acceptance or completion.

It also discards queued starts. Cancelling only the running one leaves a debounced or concurrency-queued duplicate to start seconds later, against the row that was just deleted — the exact outcome the declaration was meant to prevent, arriving late enough that nobody connects the two.

Field Meaning
event The name, exactly as ctx.events.publish writes it
schema Decoded before match runs. An event whose shape does not decode is reported and never matched — cancelling on an event you could not read is cancelling blind
match Required. Write match: () => true if you really mean "every live run"
within Only cancel runs started within this window before the event
reason Stored in the resident control request and its control-settled event; defaults to cancelOn:<event>

match has no default for the same reason singleton.mode has none: the omitted case is "cancel every live run of this workflow", which is a legitimate thing to want and a catastrophic thing to acquire by forgetting a line.

A run that started after the event is never cancelled. Without that rule, a sweep catching up after a deployment gap reads an hour of history and kills runs that started in the meantime — and the symptom (fresh work cancelled for no visible reason) looks nothing like its cause (a restart).

Bulk cancel and bulk replay

A bad deploy leaves four thousand runs that must all stop, or four thousand that must all be re-driven once the downstream is fixed.

voltro workflows cancel-many --workflow tourNarration --reason "bad deploy"
voltro workflows cancel-many --workflow tourNarration --reason "bad deploy" --commit
voltro workflows replay-many --status failed --mode redrive --limit 200 --commit

Three things are deliberately stricter than the obvious design:

  • --limit is required and there is no "all". The cap is the blast radius. truncated in the result says whether more matched, so "did I get all of them" stays answerable without an unbounded verb ever existing.
  • It is a dry run unless you pass --commit. The default for a verb that can stop a thousand runs is the one that stops none. The dashboard panel enforces the same order: the apply button does not exist until a preview has returned a number.
  • --reason is required for a cancel. The current bulk path logs it. Its resident request-identity and durable settlement integration is still incomplete; do not treat that log as a persisted cancellation cause.

The result is per-run, not a count: succeeded, failed (with the reason for each) and skipped (with what made each ineligible) are three different outcomes. A bulk op that reports "4000 cancelled" while forty failed is how people learn not to trust bulk ops.

Eligibility is fixed by the verb: a cancel acts on running and suspended runs; replay --mode redrive on failed only (redrive resumes from the step that died, which only exists for a failure); replay --mode retry on failed and cancelled. --mode has no default because the two cost very different amounts.

Delayed one-off starts — start(..., { at })

Run this once, later — without a schedule and without a sleep at the top of the body:

await ctx.workflows.start('orders.remind', { orderId }, {
  at: new Date(Date.now() + 24 * 60 * 60_000),   // tomorrow, this time
})

The start is parked as a durable row in the same pending queue the controls above use (mode: 'delayed'), and the coordinated drainer fires it when at arrives — it survives restarts and fires on whichever replica drains, never from an in-process timer. The handle comes back status: 'queued' with deferral: { mode: 'delayed', dueAt }.

Semantics worth knowing:

  • at is an absolute instant, deliberately — not a delay duration. A delay is measured "from when?" (enqueue? admission? retry?) and every queueing system answers differently; an instant has no such ambiguity and composes with the schedule/backfill surfaces, which are also instant-based. A relative delay is one line: at: new Date(Date.now() + ms).
  • At at, the start becomes an ordinary ARRIVAL. Declared controls judge it as of that moment — a debounce collapses it into whatever window is open then, a rate cap can drop it (recorded, as always). { at } delays the arrival; it never outranks a control.
  • An at in the past starts immediately — "no earlier than" is already satisfied.
  • { at, wait: true } follows the durable caller receipt through its delayed arrival and any further admission deferral, then waits for the accepted execution. It has no implicit deadline; use start() without wait followed by wait(handle, { timeoutMs }) for a bounded wait. Waiting inside a mutation transaction remains refused.
  • Each { at } start is its own row. Two delayed starts never collapse into one — unlike debounce, nothing about { at } says the second supersedes the first. Want collapsing? That is debounce, and they compose.

Prefer this over sleep as the first step of the body when the wait precedes the work: a parked row costs one row, a sleeping run costs a durable execution the whole time.

What a deferred start returns

start() no longer always returns a running handle:

const handle = await ctx.workflows.start('tourNarration', payload)

if (handle.status === 'queued')  { /* handle.deferral.dueAt tells you when */ }
if (handle.status === 'dropped') { /* over a rateLimit cap; it will NOT run */ }
if (handle.status === 'skipped') { /* handle.executionId is the incumbent */ }

executionId is null for queued and dropped, because there is no execution and there may never be one. Inventing an id there would produce a handle that polls status: 'unknown' forever.

For queued, handle.id is a durable caller receipt; handle.deferral.intentId is the queue slot used by voltro workflows flow. They are distinct: several debounced callers can share a slot, and a later burst can reuse that slot after the earlier run starts. Mutation receipts are provisional until commit and roll back with their pending work. Custom enqueue/admission adapters return receiptId alongside intentId and dueAt and persist both atomically.

A dispatch without an answer is queued, not an error. A controlled start is dispatched to the engine after admission. When that dispatch fails without a definite answer — the database pool was exhausted, or the acknowledgement was lost after the engine took the run — the run may already exist, and recovery retries the dispatch. start() returns status: 'queued' with deferral.mode: 'unacknowledged', and handle.id is the receipt recovery binds to the resulting run. Do not tell a user the start failed; follow the receipt (start(…, { wait: true }) and run(…) do). A definite rejection still throws.

Blocking callers

Every controlled direct arrival now also persists a resident caller receipt, including the arrival that fills a batch. Its input, receipt and submission transfer share one transaction; no extra runnable queue row survives commit. Until engine acknowledgement the receipt resolves as pending, never accepted merely because capacity was reserved. This adds a receipt row per direct caller; the ordinary successful start() still returns its execution handle.

ctx.workflows.run(...), start({ wait: true }) and child({ wait: true }) follow a queued admission through its durable caller receipt. They wait for acceptance before polling that exact execution; they do not resubmit the arrival payload or bypass debounce / batch / throttle / concurrency. A batching receipt follows its batch's execution and result, not a separately started item. A terminal non-execution receipt throws WorkflowStartNotExecuted; the accepted execution preserves its original success or failure Cause. An immediately accepted blocking call also only observes the accepted execution; it does not submit the input a second time to join its result. Successful start({ wait: true }) and child({ wait: true }) return handles with status: 'succeeded', not running. This also applies to delayed and replayed waits. A failed waited execution rejects with its original Cause. Update exhaustive handle-status checks and custom wire schemas, and rebuild generated clients together with the server. rateLimit still throws WorkflowRateLimitedError for run(); singleton: 'skip' with a confirmed incumbent throws WorkflowSingletonHeldError. A singleton with unknown acceptance remains queued until its receipt resolves. Blocking calls have no implicit deadline; use start() then wait(handle, { timeoutMs }) when the caller needs a bounded wait. Waiting inside a mutation transaction remains refused; wait only after commit.

Upgrade: WorkflowDeferredResultError has been removed. Remove its imports and catch branches; do not mechanically rename them to WorkflowStartNotExecuted. The latter means admission has settled without execution, not that a caller is still queued. voltro update identifies references needing this manual change.

ctx.workflows.executionId(name, payload) validates the execution payload. For a batching workflow this means the assembled { items: [...] }, not an individual arrival item. An item has a caller receipt; its batch execution identity only becomes known after admission assembles the batch. Run-history replay lookup happens only after an arrival passes through admission without being queued or dispatched by a control. It must neither derive a batch identity from an item nor replace a current admission decision with an old run.

Pausing a workflow

voltro workflows pause tourNarration --reason "deploying a fix"
voltro workflows unpause tourNarration

A pause makes starts collect, never discard — so you come back to a backlog rather than a hole in the data. unpause drains it.

The pause is a row, so it applies fleet-wide; each replica picks it up on its next drain tick (~1 s).

Seeing what happened

voltro workflows flow
voltro workflows flow --workflow tourNarration --format json
voltro workflows flow validate
voltro workflows flow retry <intentId>
voltro workflows flow discard <intentId>

or GET /_voltro/inspect/workflows/flow-control.

This is not optional colour. A debounce that collapses nineteen starts into one is indisputably correct behaviour and indistinguishable from nineteen starts vanishing — unless something writes down that it happened. So every decision is a row in _voltro_workflow_admissions, with its key, its reason, how many starts folded into it, and how long it waited.

Three questions it answers:

Question Where
"Twenty edits, one run — did that work, or did I lose nineteen?" collapsed
"Nothing has run for an hour. Stuck, or quiet?" waiting= on the queued row
"Why did my run not start?" the ledger's outcome + reason

Permanently invalid queued payloads

Workflow apps also declare _voltro_workflow_control_requests in their schema. It is the resident storage foundation for stable control-request identities and attempt history, not a public cancellation API or proof of cancellation. Engine delivery, generation-bound confirmation and recovery are still being integrated; an empty table must not be interpreted as evidence that no control operation was requested through the current APIs.

Low-level hosts can call acceptWorkflowControlInTransaction(tx, request) and beginWorkflowControlDispatchInTransaction(tx, requestId) from @voltro/workflow inside withAdmissionTransaction. Allocate the request ID before the transaction; its run, workflow, execution, generation and operation cannot change on replay. These database-only functions neither invoke the engine nor confirm cancellation. A dispatch result is not permission to call an unfenced engine interrupt: the engine boundary must enforce the target generation too.

Acceptance also captures owner: { tenantId } from the original run generation. owner: null means ownership was not proven; it is not equivalent to owner: { tenantId: null }. Replaying a request does not replace its owner, even if the run's subject changes or the run row is removed. Caller-supplied metadata cannot supply this authority. Public receipt authorization must use this stored owner; the request ID alone is not a credential.

readWorkflowControlInTransaction(tx, requestId) reads the accepted request and its nullable result. The owning body's terminal transaction stores that result alongside the run and capacity release for every existing request of the same generation. Cancellation confirms cancel/terminate; success or failure retains the actual outcome as already-terminal, including output and errors. A resume request is never confirmed by cancellation. This stored evidence survives a later body generation. Dispatch returns settled without another attempt when a result exists. A result-write failure rolls back the terminal transaction. Acceptance after completion stores the matching generation's observed outcome immediately as already-terminal, even if it was cancelled: the new request did not cause that earlier exit. A missing/invalid completion timestamp refuses that observation; a different generation's outcome is never substituted. Public control adapters and suspended-run/resume delivery remain under integration; absence of a result is not proof that an engine did nothing.

completeWorkflowRunGenerationInTransaction also accepts outcome: { status: 'cancelled' } for a confirmed exit of the owning body generation. It atomically records cancellation and releases linked and dispatched pre-acknowledgement reservations, without creating a failure report. An interrupt promise resolving does not satisfy that precondition. This low-level transaction does not itself interrupt the engine or confirm delivery.

The run recorder now records cancelled when the engine has requested interruption and the body exits with an interruption-only cause. Capacity remains held while that body is still active. A real handler failure remains failed, even if cancellation was requested at the same time. Confirmed cancellation emits run-cancelled, not a failure report; low-level onTerminal and recordRun callbacks must handle the cancelled status. This recorder boundary does not yet replace all control API and ParentClose paths.

The workflow facade's cancel and resume now return { operation, kind, requestId, snapshot }. Read the observed run state from snapshot, not from the requested operation. already-terminal preserves the original completed result. confirmed requires an observed cancellation exit; an unresolved command or observation yields unknown. A running snapshot alone does not prove a particular resume was accepted. The engine-only facade has no resident request ID (requestId: null) and therefore never claims accepted. The durable control-request adapter is still pending. poll, query and wait continue returning snapshots and classify completed interruption-only exits as cancelled. Custom facade implementations must adopt WorkflowControlResult; voltro update reports this migration for affected source.

Custom makeWorkflowFacade hosts can supply control({ operation, workflowName, executionId, requestId }) as the resident owner. It returns WorkflowControlResult and owns acceptance, dispatch and observation together. With this adapter configured, the facade performs no engine poll, interrupt or resume around it. Its rejection propagates; it never triggers a second, engine-only attempt. The built-in host integration is still pending. Do not claim accepted before the resident request commit, or confirmed from a void engine reply.

For these hosts, cancel(name, executionId, { requestId }) and resume(name, executionId, { requestId }) forward the caller's non-empty identity unchanged, including through lazy and caller-context facades. Reuse it for retries of one intent, not for a new operation or target. The resident adapter reads the original receipt before considering the current run generation; a retry cannot bind that identity to a replacement body. Without a resident adapter, supplying requestId refuses before any engine command instead of promising durable replay.

Use readControl({ operation, workflowName, executionId, requestId }) to observe that receipt without submitting another request. Custom hosts provide the makeWorkflowFacade option readControl; unsupported hosts refuse explicitly. This lookup does not require the workflow to remain in the current declaration registry. The resident adapter must validate receipt ownership and authorization; the request ID is not permission. A missing receipt is unknown, and a failed read propagates rather than silently resubmitting the operation. A saved terminal result remains the original result even after a replacement generation starts.

Executing bodies and suspended-entry cancellation use the same validated, generation-scoped pending-request scan. Settled receipts do not trigger another interruption, including when they fill a complete page before a pending request. A malformed record causes a diagnostic read failure and a retry, not an abort of the body. Repair of malformed control data is not automatic.

The internal resident control request can persist an optional cause with a reason and structured detail in the acceptance transaction. Replays retain the first cause alongside the original request timestamp, including after dispatch and completion. Settlement writes one control-settled run event per request in the same transaction: its payload carries requestId, operation, generation, cause, kind and status (plus enteredGeneration for resume). An already-terminal observation does not claim the request caused that outcome. Retries preserve one event; event-write failure rolls back settlement. Inspect the run-event timeline/table to retrieve this evidence. Terminal run events (run-succeeded, run-failed, run-cancelled) also commit with their owned transition, receipts and capacity release, once per generation, including a version-incompatible entry refusal before the body runs. An event-write failure rolls that transition back. Custom recording hosts must include the event table in the resident schema; terminal events no longer pass through custom recorder.recordEvent. Use emit/onTerminal for post-commit observation, not a second writer. Live push, wakeup cleanup and parent-close callbacks remain best-effort; their recovery integration is still unfinished. Not every control producer is connected yet, and a diagnostic log is not a durable cause.

Every terminal generation also stores a _voltro_workflow_parent_closures intent in that transaction. It retains the parent run/execution/generation, terminal status and timestamp, with an initially empty child cursor and completion timestamp. It contains no workflow payload or caller credentials and stays in the resident store. Repeated completion cannot create another intent. Custom resident schemas must include this table. The resident admission drain visits these intents even when it skips pending arrivals. An intent is not evidence that children have stopped, and existing parent-close callbacks remain best-effort.

The low-level stageWorkflowParentClosureInTransaction(tx, closureId) stages at most 100 children per resident transaction, storing cancel/terminate receipts and cursor together without engine I/O or child status writes. abandon remains untouched. Replay retains the first child generation and original parent cause. scanHasOpenChildren retains open-child evidence across pages. End-of-scan wraps for another pass when any bound child was open; only a complete pass observing none sets completedAt. Malformed child status is not terminal evidence. Completion also requires every accepted parent-close control to be settled. The indexed parentClosureId is derived from the original control cause; a missing child row or malformed receipt cannot substitute for its result. An unresolved receipt retains the closure for recovery even when no child rows remain. Recovery isolates malformed intents without logging their contents. The shared CLI worker requests another recovery probe within five seconds even with an idle pending queue (the configured base interval remains the floor); this costs periodic database checks instead of depending on a wake after commit. Neither built-in boot path installs an interrupt-and-write parent-close callback: child finalizers must finish before resident recording confirms cancellation. Physical store enumeration remains unfinished.

closeWorkflowChildrenForParent and its options/decision/store/emitter/logger types have been removed. Custom hosts must run drainTick against the same resident store as terminal recording. Never replace this with direct child status writes after an engine interrupt. WorkflowParentCloseStatus remains the recording observer's terminal status type.

Recorded children carry parentGeneration beside parentExecutionId. Ambient recorded nesting captures the parent body's generation; explicit recording hosts can supply parentGeneration with their parent execution ID. An explicit parent never borrows an ambient generation. Staging skips children of another parent generation and refuses cancellable children with no binding. Explicit parentGeneration now survives child caller metadata, queue decoding, durable start-context storage and both boot paths into the recorder. Retrying the same execution with another parent generation is refused; a fresh host reads the original binding. The recording wrapper now supplies its committed body generation before the application context is constructed; both boot paths bind this identity to that context's child facade, including Promise-based calls. execution.generation is the current body's identity; execution.parentGeneration belongs to its parent and must not be substituted. Event-triggered starts from that context preserve the same parent generation through caller metadata and start-context storage. Child entry checks that exact parent generation under the resident admission lock. A recorded closure, even with completedAt set, makes cancel/terminate children terminal before their body is constructed; the run, control receipt, terminal event and capacity release commit together. Missing parent identity or generation is refused for these policies. abandon is exempt. Queued acceptance can still precede this check. Resident drain recovery stages controls for children that entered earlier; an accepted control alone does not complete their parent closure.

Server callers can pass { requestId, cause: { reason, detail } } as the third argument to facade cancel/resume. Lazy and caller-context wrappers preserve it, and the resident adapter stores it on first acceptance. A retry on a fresh host with the same request ID does not replace either the original generation or cause. Engine-only hosts refuse these options instead of dropping the cause. The shared canceller reads an explicit receipt even after the run became terminal, preserving a confirmed cancellation without submitting another request. Repeated confirmation is not proof that a producer's follow-up ran exactly once.

cancelOn derives that identity from the workflow, run and triggering event, so a repeated sweep reuses the same control intent. The shared canceller forwards the original reason and event details. Built-in resident boot/recovery integration is still incomplete: an engine-only host refuses this persistent control path; the sweep retains its watermark and reports the failure rather than claiming cancellation.

@voltro/protocol exports the browser-safe WorkflowRunSnapshotSchema and WorkflowControlResultSchema; runtime result types derive from this shared contract. The facade validates resident adapter responses on submission and observation. Contradictory results, such as confirmed cancellation with a running snapshot or acceptance without a receipt ID, fail explicitly without falling back to an engine command. Responses must also match the requested operation, workflow, execution and any caller-supplied receipt ID. This validates response consistency, not authorization or durability. Exception: an unrecorded submission returns unknown with a null receipt ID, even when the caller proposed an ID. That is not acceptance. A receipt lookup must retain the requested identity, including when it cannot find it. The built-in cancel/resume RPCs and useWorkflow().cancel/resume return this complete result too. Rebuild generated clients with the server; inspect kind and snapshot before reporting completion. The RPC path does not write its own terminal run state or event after a cancel command. Inspect response migration is still in progress; its boolean responses are not equivalent receipts.

To retry the same intent after an uncertain browser transport result, retain an ID before the first call and pass it as the second argument:

const control = { requestId: crypto.randomUUID(), cause: { reason: 'operator' } }
await workflow.cancel(run, control)
// If the transport outcome is unknown, reuse control for this same intent.

The workflow must declare controls: for any of this to reach it from a client; without it the built-ins refuse every caller but the system and admin:full.

resume(run, control) accepts the same shape. The built-in RPC input carries it as control: { requestId, cause? }; both boot paths authorize the run before forwarding it to the resident owner. An explicit ID requires a resident control adapter; unsupported hosts refuse. Generating another ID for each retry creates another intent. Omitting control does not establish caller-owned retry identity. Use workflow.readControl({ workflowName, executionId, operation: 'cancel', requestId: control.requestId }) to observe the same receipt without submitting another control. The built-in __voltro.workflow.control query authorizes the stored receipt owner, not the current run row; it can therefore read retained evidence after run retention. Missing, foreign and unowned receipts refuse with the same authorization error, preserved by the hook as a typed rejection. The hook reads one snapshot and closes the stream; call it again when another observation is needed. It does not automatically poll or resume anything. Host readControl wiring and physical resident-store routing are still required.

A configured onFailure handler must accept its report: a dropped start or a singleton-skipped incumbent does not prove that this report was received, and the failure recovery retries the handoff with a new receipt generation. A queued notification is accepted work, not proof its body has completed. An absent onFailure declaration is an explicit absence, not a delivery to retry. The internal _voltro_workflow_failure_reports table provides immutable incident storage beside the resident queue: replaying a report ID retains its original encoded payload, recipient and failedAt. Every source writes there in its own transaction — a run's terminal transition (an exhausted body, a timeouts.finish cancellation), a queued start's expiry or dead-letter, a submission's rejection — and the recovery delivers from the row. Recording a report does not mean its handler has received it; the report's acceptedExecutionId says when it did.

Dispatchers persist dispatchAttempts before engine I/O. Local validation before the first call, or a guaranteed pre-engine input rejection of that first call, can prove non-acceptance. A prior or competing unacknowledged call cannot: a later schema rejection leaves acceptance unknown and retains capacity. Cancellation or acknowledgement failures after a successful start are not input rejection. Low-level isPermanentStartError classifiers must guarantee this pre-engine boundary, not merely classify errors as non-retryable.

Low-level hosts can use recordWorkflowFailureInTransaction(tx, record) from @voltro/workflow inside withAdmissionTransaction, in the same resident transaction as the source transition. Allocate the incident ID before that transaction and retain it for recovery; a reusable debounce slot is not an incident identity. readWorkflowFailureRecord(store, id) returns the stored record and refuses malformed data. Neither function invokes a handler or acknowledges delivery. Never put engine or notification I/O in the transaction.

handoffWorkflowFailureInTransaction(tx, reportId, now) transfers the stored report to the existing durable arrival queue and records its caller receipt in the same resident admission transaction. Retries reuse that receipt and the original recipient, payload and failedAt; an explicitly absent handler stays unconfigured. A missing or contradictory linked receipt is refused, not repaired by creating another start. kind: 'queued' describes the handoff, not the handler's acceptance or live execution status. Handler flow control still applies when draining the arrival. Proven submission rejection now invokes this handoff automatically, in the transaction that records the report and releases the lease. A failed receipt write rolls back all three; retry keeps the original report and handler. AdmissionDataStore adapters must forward the resident store's native insertIgnoreWithOutcome, including inside transaction(work); do not emulate it with a pre-read or upsert.

The normal drainer now discovers resident reports after restart, even when its local pending-queue hint says idle. reconcileWorkflowFailureInTransaction checks their receipts: a recorded drop or skip creates a new receipt/arrival for the original report, preserving the previous terminal receipt. Retries use the shared bounded exponential backoff (1 s to 5 min, deterministic upper bound) and never precede a rate-limit retry deadline. Concurrent reconcilers commit one new attempt, not one per replica. Unknown engine acceptance retains the existing submission and prepared execution ID; it never creates another arrival. Only a matching acknowledged submission records acceptedExecutionId on the report. This confirms handler acceptance, not completion or external delivery.

Report recovery is bounded and paged; malformed reports remain stored, are reported by ID without their payload, and do not block neighboring records. scanned includes examined failure reports as well as submissions and arrivals. When a failure-delivery submission is rejected before engine acceptance, its own failure is recorded without another onFailure notification. The resident report/receipt relationship identifies that call; an ordinary invocation of the same workflow still uses its declared handler. This prevents recursive start- rejection notifications, without suppressing the original report's retry.

Proven handler-input rejection also retries the original report with backoff, without mutating the rejected submission or its receipt. The rejection must have a matching failure record and no earlier ambiguous dispatch; a rejection marker alone does not prove non-acceptance. A repaired handler can then accept the report. Missing, foreign-owned or dead-lettered arrivals now fail reconciliation instead of returning queued. The warning includes failureId and a safe reason (arrival-missing, arrival-owner-mismatch, arrival-dead-lettered or rejection-proof-invalid); it never creates another arrival from absence alone. Replacing a debounce payload settles its still-pending caller receipts with terminal: { kind: 'superseded', at } in the replacement transaction. Those receipts cannot be bound to the later execution. Failure recovery retries each superseded report with backoff and a new receipt; the original reports remain unchanged. No payload comparison or application-schema guess decides ownership. Other source hooks and recovery policy for operator-discarded, missing or dead-lettered arrivals remain unfinished. Batch handlers retain all selected reports in items; both size-triggered and timeout-triggered batches preserve their submission and payload through a lost engine acknowledgement. Every included report is acknowledged only after the matching submission ACK. Receipt acceptance alone is not proof of body completion or external delivery.

When a delayed arrival is deferred into another queue slot, the drainer now creates that slot, forwards every unbound caller receipt and consumes the old arrival in one resident transaction. Bound receipts do not move. Concurrent drainers re-read the arrival under the lock; a foreign receipt rolls back the whole transfer. An admitted delayed arrival binds its receipts and consumes all selected inputs together with an immutable submission in that same transaction, using the admission already recorded rather than reserving a second slot. It then uses the prepared execution and acknowledgement path described above. A lost engine reply leaves that submission recoverable; recovery retains its original identity and input. This is not a claim of exactly-once external effects or complete coverage of every failure source.

For a recoverable start outside a mutation transaction, pass requestId and an explicit at to ctx.workflows.start. Create and retain both values before the first attempt; retries must reuse the same identity, instant, workflow, tenant and payload. requestId must be non-blank and at most 64 characters. Identified starts always use a durable receipt, even when at is already in the past; normal admission still runs when the queue drains. Concurrent matching attempts reuse the original receipt instead of creating another pending input. A different payload or instant with the same identity is a conflict. Reuse also requires the same caller and parent generation; a fresh trace ID on a retry does not change the request identity. A new intentional start needs a new identity, even if its payload is identical. An identified start cannot be combined with a mutation's transactional enqueue or internal admitted dispatch. Without requestId, existing immediate and delayed-start behavior is unchanged. Keep the request identity outside a transient response if recovery after a lost reply is needed; this option does not persist a browser journal for you.

Low-level admission adapters can bind a caller-owned start identity by passing an optional requestFingerprint to recordWorkflowStartReceipt. The fingerprint is a lowercase SHA-256 digest of the immutable encoded request; it is retained after the pending row is consumed. The receipt separately binds the workflow and tenant. readWorkflowStartRequest(store, { id, workflowName, tenantId, requestFingerprint }) reads that binding without admitting or executing work: it returns the original receipt, returns undefined for a missing identity, or rejects conflicting reuse. A legacy receipt without a fingerprint cannot be adopted as a new request. Perform lookup and the first receipt/intent writes in the same resident admission transaction. Matching a binding is not evidence that the workflow completed; resolve the receipt for its actual outcome. This read helper is a low-level adapter contract; requestFingerprint is not a ctx.workflows.start option and is computed by the admission adapter.

readWorkflowStartReceipt(store, id) now exposes terminal, initially null. A queued caller later refused by rate limiting gets { kind: 'dropped', at, retryAfterMs }; a singleton skip gets { kind: 'skipped', at, executionId, runId } naming the incumbent, not a newly started run. Both delayed and controlled queue drains persist the decision, settle all affected receipts and delete their inputs atomically. A failed write keeps the inputs and receipts available for retry. A terminal receipt is never rebound or forwarded when the queue slot is reused; it cannot simultaneously claim a submission. This records the admission outcome, not workflow completion. Use recordWorkflowStartReceipt for new receipts rather than hand-writing a row that omits terminal: null.

WorkflowStartTerminalOutcomeSchema and WorkflowStartTerminalOutcome from @voltro/protocol define these browser-safe admission outcomes. The persisted receipt uses the same schema. A cancelled start needs no execution ID; a skipped start's execution/run IDs identify the incumbent. These values do not by themselves prove an execution was interrupted.

WorkflowQueuedCancellationResultSchema and WorkflowQueuedCancellationResult from @voltro/protocol describe cancellation evidence addressed by receiptId. kind distinguishes accepted, confirmed, unknown and already-terminal. Its observation is separate: pending before execution evidence, admission with a settled start outcome (or proven rejection), execution with a real run snapshot, or unavailable with the missing-evidence reason. Only the execution branch carries a snapshot. Missing requests have requestId: null; loss of run evidence retains the known request ID. Contradictory combinations refuse decoding. The runtime facade exposes cancelQueued(target, options) and readQueuedCancellation(target, options), where target contains workflowName, receiptId and a stable requestId. Custom hosts supply separate resident adapters on makeWorkflowFacade; missing adapters refuse. The caller-bound facade supplies identity, and these operations must run after a mutation commits. Custom WorkflowsAppContext implementations must provide both methods or explicitly refuse them. Both voltro dev and voltro serve install the resident adapters. They locate an existing cancellation request before its source receipt, so retries and reads retain the original result even after receipt removal. Missing requests remain unknown; reads never create a cancellation request. Both operations require an authenticated caller before physical discovery; an anonymous identity cannot control a tenant-less request. The generated RPC group includes the __voltro.workflow.queued.cancel action and __voltro.workflow.queued.control query. Both take workflowName, receiptId and requestId; cancellation additionally accepts cause. They return the shared queued cancellation result. The server supplies caller identity from the authenticated request, never from input fields. Regenerate the client group when adopting these procedures. useWorkflow exposes cancelQueued(input) and readQueuedCancellation(target). The latter reads one snapshot and closes its subscription; a typed query denial stays the original error. Custom WorkflowState implementations must provide both methods. Automatic queued control recovery is not yet fully connected: an accepted response is not confirmation that a running execution has stopped.

Custom hosts can use cancelQueuedWorkflowStartInTransaction and readQueuedWorkflowCancellationInTransaction from @voltro/workflow inside their resident admission transaction. Supply the authorized tenantId with the target identities. These adapters perform database work, not engine I/O; authorization and selection of the original physical store remain host duties.

Low-level adapters can call resolveWorkflowStartReceipt inside the resident admission transaction, supplying receiptId, workflowName and the authorized tenantId. Its WorkflowStartResolution distinguishes pending, accepted with a confirmed executionId, terminal with the queue outcome, and rejected with a proven failureId. A prepared ID alone stays pending. Missing or contradictory owner records, dead-lettered inputs and stopped unacknowledged submissions fail visibly instead of waiting indefinitely. ctx.workflows.wait(queuedHandle, options) follows that receipt until confirmed acceptance, then polls the accepted execution. Its timeoutMs budget starts before receipt resolution and is not restarted at acceptance. This bounds the polling loop; it does not cancel a database or engine read already in progress. The current caller is bound by the context facade; a receipt is not authority. Custom runtime hosts must supply resolveQueuedStart on makeWorkflowFacade.

The dev and production hosts locate the receipt across declared physical stores and registered tenant namespaces, then reread it under the resident admission lock. They require explicit caller identity and matching workflow/tenant ownership. A duplicate receipt ID, incomplete discovery or unavailable store refuses the observation; a root-store match does not hide another unsearched placement. This lookup does not move receipts or dispatch queued work.

No execution means no run snapshot: WorkflowStartNotExecuted from @voltro/runtime reports reason as dropped, skipped, superseded, cancelled, abandoned or rejected, plus handleId, workflowName, nullable failureId and nullable incumbentExecutionId. This also applies to directly dropped or skipped handles. To observe a skipped incumbent deliberately, use the explicit workflow-name/execution-ID wait overload. Inside a mutation, both wait overloads refuse: return the provisional receipt and wait only after the transaction commits.

If a controlled or batched queue declaration is removed, the drainer records each original input and caller in a resident failure report and settles its receipts with { kind: 'abandoned', at, failureId } in the same transaction as queue removal. It does not call an ephemeral failure callback or invent a current handler for a removed declaration. The report therefore has no handler; inspect and reconcile it explicitly. The same outcome is preserved when a delayed start's workflow disappears. This does not yet cover every dead-letter path, nor does it make a queued handle waitable by itself.

Queued start-timeouts persist the failure report, admission history, terminal receipts, input removal and the configured handler's durable arrival together. A batch keeps its full ordered item list in one report; no execution is claimed before the handler accepts it. A notification timing out does not recursively invoke that handler's own onFailure. A batch mixing notification and ordinary callers is retained with a reconciliation error because it has no single safe recipient decision. These are resident transactions, not atomic transactions with the workflow engine or an external provider.

After a transient batch drain failure, all still-matching inputs receive their retry counters and one shared, jittered retry deadline atomically. Updating only the first item—or drawing a separate deadline for each—would let a later tick process an unintended subset of the failed batch. A changed selection is not overwritten by this retry update.

executePreparedWorkflow(definition, decodedPayload, executionId) from @voltro/workflow is the low-level submission primitive for an identity already chosen and durably bound to that input. It works with Voltro workflow() definitions and does not evaluate their idempotencyKey again. Validation, engine annotations and suspended retry settings remain in effect. It returns the accepted execution ID, not completion of the body. An explicit fourth argument { discard: false } waits for the typed body result under the same ID, preserving declared failures without deriving the key or executing a second body. Record acceptance before joining the result: a body failure is not a refused start. The caller must commit admission, identity/input binding and caller context before invoking it outside the transaction; this primitive performs none of those persistence steps. The runtime facade accepts an internal preparedExecutionId only alongside admitted: true, without wait, at or transaction enqueue. It records caller context under that same ID before submission. This is a dispatcher contract, not an application admission bypass. Controlled queued submissions, direct controlled arrivals and recovery prepare and persist this identity before dispatch. A size-closing batch transfers all earlier caller receipts with its inputs in the admission transaction. run() and start({ wait: true }) confirm acceptance before joining the body result; a later body failure does not reject that admission. Uncontrolled immediate starts do not use this submission journal. Failure-report delivery uses the queue; terminal body-failure reporting still needs its separate durable completion boundary. Memory-engine replay tests prove lost-ACK retry through a fresh facade, not engine restart survival.

Custom WorkflowAdmissionOutcome adapters return accept(): Promise<string> for kind: 'start', not a post-execution commit callback. Persist the reservation, encoded input and receipt transfer first with takeAdmittedSubmissionInTransaction; dispatch outside the transaction through acceptAdmissionSubmission. Return only the durably acknowledged execution ID. Do not reserve the same admission twice or execute a second time in the facade. Codec adapters encode each batch item separately. Recovery can win before the original caller dispatches. The dispatcher rereads the resident binding and returns its existing acknowledgement; a stale in-process submission object is not evidence that preparation or acceptance is still pending.

Low-level terminal generation boundary

The framework inventory includes _voltro_workflow_queued_cancellations beside the resident admission tables. The recorded entry binds retained cancellation requests for dispatched submissions to its own generation before application code, across all request pages. A matching request closes the run, settles its control receipts and releases its local reservation in that same transaction, even before the submission ACK is recorded. Internal acceptance after entry captures that resident generation under the same lock; its control monitor observes the committed request, and only actual exit confirms cancellation. If that owner is already cleanly suspended, acceptance settles cancellation in the same transaction without an engine wake. Retrying acceptance may settle that same original owner after it suspends; a read-only status query never does. Public queued-cancel acceptance and post-entry recovery are still unfinished; this table is not a direct-write API. An unresolved submission cancellation also stops a replacement entry for the same run and execution before its body starts. It does not transfer or confirm the original generation's control: that observation remains unknown because preventing a replacement body does not prove that the old body exited. Only the replacement generation receives the new terminal event. A clean suspension is different: its body has exited, so entry can settle the original generation before any replacement body starts. Settled controls do not become new stop requests. Process-restart recovery and fencing external work from an old runner are not proved by this entry boundary.

_voltro_workflow_runs.runnerGeneration stores an opaque body-entry identity. It is independent of executionId, which survives replay, and reclaimCount, which resets after a clean exit. Allocate the identity before the transaction; use enterWorkflowRunGenerationInTransaction for both initial insertion and re-entry inside the resident transaction. It preserves the original run ID/input, returns the prior snapshot for crash/version checks and distinguishes created, entered, replayed, cancelled and already-cancelled. A pending resident cancel/terminate for a cleanly suspended generation (runnerEnteredAt: null) completes that owner before replacement: run, control results and capacity release share the entry transaction. Neither cancellation decision may invoke application code; only cancelled represents a new transition to observe. already-cancelled also prevents a repeated engine wake from reviving the execution. A clean suspended entry also confirms pending resume requests for the previous generation in the same transaction. Their immutable result contains status: 'running', generation and enteredAt, not a terminal completedAt. A running replacement is not resume evidence; cancellation takes precedence. Later completion does not erase the entry receipt. Automatic wakeup and resume delivery remain unfinished. Replaying the same entry identity never increments the reclaim count or resets an already completed run. A missing or unreadable write must prevent body execution, not silently disable recording. completeWorkflowRunGenerationInTransaction accepts that owner and an outcome. Under withAdmissionTransaction, it checks ownership, records the terminal row, failure report and handler queue arrival, and releases all admission reservations. Any write failure rolls the transaction back. superseded and not-running mean no new completion; do not emit a terminal hook for either result. Failure identity includes the generation that actually completed. Retrying report delivery preserves that identity; it does not reopen a failed execution.

The production recorder now commits entry before invoking the body. Custom wrapWorkflowExecuteWithRunRecording hosts must supply transaction using withAdmissionTransaction(store, work) against the same resident store as store. The callback is database-only and may be replayed. An entry commit failure fails execution; it is not a best-effort observation failure. Execution recording stores must provide query: the active body observes resident cancel/terminate requests matching its run, workflow, execution and generation. It interrupts its own fiber, not a replacement addressed only by execution ID. The monitor performs one sequential read followed by a one-second pause while idle and stops when the body completes or suspends. Read errors retry without claiming cancellation. Control producers, suspended-run delivery and durable result receipts are still being integrated; this monitor alone does not complete the control lifecycle. Custom recording stores may implement query as an instance method: the replay-shape reader preserves its receiver when loading prior steps. Stored run inputs and durable failure reports retain the encoded full workflow payload. Dev and serve provide the declared input encoder, including the full batch envelope rather than one item. Custom recorder hosts with transformed inputs supply encodePayload: Schema.encodeUnknownSync(definition.payloadSchema); omitting it promises JSON-only input. Encoding failure prevents entry and body execution. The recorder uses this boundary for body success and failure. Its onTerminal callback is an observation after commit, not a lease-release or handler-start callback. Custom hosts configure durable delivery with failureContext: { handler, callerContext, tenantId }. Framework boot paths provide the bound identity with cached authority removed. Failure reports retain the original start identity and correlation from the durable start-context binding. A repeated delivery keeps its incident's first recipient and timestamp. A distinct later incident may use the then-current onFailure declaration; the handler is not frozen at the run's first start. Execution permissions are still resolved anew, never copied into the delivery context. Notification-body failures are recorded without a secondary handler, including completion before dispatcher ACK. It does not fence application side effects or establish cluster runner ownership. Suspension and crash-loop parking also check the current running generation inside the resident transaction. A late old suspension neither changes the row nor reports a suspension event or failure. A failed commit propagates; successful suspension retains admission capacity and the reclaim counter. Version rejection uses the same generation-checked terminal transaction as body failure, including the original stored payload, durable failure arrival and admission release. Its application body does not run; callbacks and metrics follow only a committed outcome. Resident parent-close recovery stages controls in bounded, committed pages; a failed later page preserves earlier cursor progress. An unresolved control keeps its closure open even when the child row disappears. For a clean suspended body, cancelSuspendedWorkflowRunInTransaction(tx, owner, now) settles an accepted cancellation only for the same recorded generation with a cleared entry marker. It commits run state, receipt, terminal event and admission release together. The resident adapter and parent-close recovery use this boundary without an engine wake. A later engine entry observes cancelled and does not run the body. Active bodies still confirm through their own monitor and finalizers. Resume delivery and physical-store enumeration remain unfinished. An ACK that arrives after body completion also releases that admission's lease in its transaction; a newer running generation retains capacity. Body completion also releases reservations bound through dispatched prepared submissions when ACK remains absent. It does not fabricate an acknowledgement or release an undispatched proposal. Recovery still confirms the same prepared execution. Do not infer that a complete process-crash protocol follows from these body-completion tests.

For a resident admission submission, prepareAdmissionSubmissionExecutionInTransaction(tx, submissionId, proposedExecutionId) binds the application-derived ID before dispatch. Compute the proposal outside the database transaction. Competing proposals return the first committed preparedExecutionId; always use the returned value. Preparation neither acknowledges execution nor releases admission capacity. An ACK with another ID is refused. A submission already dispatched without an ID cannot safely choose one afterward. Both controlled submission producers and recovery invoke the same preparation step before the dispatch marker. Custom AdmissionDrainDeps must implement resolveExecutionId(name, decodedPayload) without executing work and honor the fourth preparedExecutionId argument to startAdmitted unchanged. The shared boot builder passes that ID to the runtime facade. A contradictory returned ID fails before singleton eviction, not merely at acknowledgement.

Low-level admission hosts may provide acquireSubmissionCapacity(submission). It runs after resident commit, outside SQL, before engine preparation in both queue transfer and submission recovery. Return false for ordinary capacity waiting: no dispatch attempt or failure retry is consumed; recovery revisits after one second. Replays must reuse a durable permit for the same submission. The direct CLI gate also checks before returning its start decision and retains a queued receipt on false. This adapter does not by itself enable deployment pool wiring or terminal permit release; those integrations remain unfinished.

A retained succeeded or failed run refuses body re-entry for the same execution ID. An engine may replay its own complete journal result without entering the body; if it lost that result, the framework refuses rather than rerunning side effects. The run-history output may be truncated and is not used as a substitute result. Reconcile the engine journal; do not submit a new ID merely to bypass this refusal. After run-history retention, its durable parent-closure record still prevents a new body entry, including for cancelled executions. These records are not TTL history, even after child-close recovery finishes; do not delete them to clear a queue. A late wakeup of a suspended generation still needs separate fencing.

Admission rows persist concurrencyReservation (pool, resolved key and limit) with their local reservation. Cross-store capacity adapters read this committed metadata, not the current workflow declaration. A changed deployment cannot silently move an existing submission to a new pool. A concurrency admission without matching reservation metadata requires reconciliation, not a guessed current limit. The low-level readAdmissionSubmission reader validates stored metadata and distinguishes an absent row from malformed data.

A queued payload is decoded against the workflow's current input schema when the drainer starts it. While the pending intent still owns the work, a WorkflowPayloadError dead-letters the intent and records its onFailure report in one transaction, logging workflow flow control: intent dead-lettered; the report reaches the handler through the failure recovery. After queue transfer, the submission owns the error instead: proven rejection sets its failureId and removes it from dispatch recovery. Its original input and retained report remain stored. Do not use the pending-intent retry command to re-arm this state; submission reconciliation and report delivery are not finished yet.

Any other failure — the engine unavailable, a store error, an unknown workflow name after a rename — is retried, but not on every tick: each failed attempt backs the intent off (one second, doubling, capped at five minutes, jittered), so one undrainable intent no longer produces a warning every two seconds on every pod. After workflows.deadLetterAfterAttempts failed attempts (default 50 — several hours under that backoff, which is no longer transient in any sense a retry can help) the intent is dead-lettered exactly like an invalid payload: one onFailure report written with the dead-letter, the row retained, voltro workflows flow retry to re-arm it. false disables the cutoff.

The payload, error, attempt count, and dead-letter time remain visible in voltro workflows flow and the dashboard. Remove that retained row explicitly after inspection:

voltro workflows flow discard <intentId>

Each queued row also records the app version and a fingerprint of the workflow input schema that existed when it was queued. Before deploying a schema change, validate the backlog against the release currently running the command:

voltro workflows flow validate
voltro workflows flow validate --limit 1000 --format json

The result says valid, invalid, workflow-missing or schema-unavailable per intent and compares queued/current schema fingerprints. It never returns the stored payload. This is a diagnostic and does not mutate the queue.

After deploying a workflow/schema fix, explicitly re-arm one retained dead-letter:

voltro workflows flow retry <intentId>

Only a dead-letter can be retried. The command retains its payload and provenance, clears terminal/error/attempt state, and lets the coordinated drainer validate it again. Use discard when the work is obsolete; do not retry unchanged invalid bytes merely to make the warning recur.

A later debounce arrival for the same key replaces the obsolete payload and clears its dead-letter state, so a valid new edit can run without an operator first deleting the old row. If onFailure itself cannot be signalled, the intent stays live and visible for another attempt; Voltro does not mark work as handled before its failure path ran.

Cost

A workflow that declares no control takes exactly the code path it took before this feature existed — no query, no branch beyond one map lookup.

A workflow that declares one pays only for that one: an undeclared control costs zero round trips. The rate/throttle window reads at most limit rows, which is why limit is a throughput knob and not somewhere to put 10⁶.

The drainer runs on one replica per tick through the same claim arbiter the cron scheduler uses. N replicas draining at once would each see a free slot and each take it.

Measured admission throughput

Measured, not estimated — node packages/cli/scripts/admission-throughput.mjs in the framework repo drives the real gate, facade and drainer against the in-memory reference store (slope across N=500/1000/2000 starts, median of 3 sequential repeats; Apple-silicon dev machine, 2026-08):

  • no controls (passthrough): ~1 µs/start (~800,000 starts/s) — the do-nothing path really does nothing.
  • through a concurrency gate: ~110 µs/start (~9,000 starts/s) — dominated by the reference store's unindexed admission-state scan, which grows with the ledger; a real dialect serves that read from an index, but also adds its round-trips. Read this as the machinery's worst-case CPU floor, not a database benchmark.
  • durable park → drain → start: ~4 µs/row (~240,000 rows/s) of pure machinery per queued row.

The deployed ceiling is min(these numbers, what your database serves for the admission reads/writes) — on any SQL dialect the database is the bound long before the gate is. That is also why there is no key-group batching in the admission path: at ~9k gated starts/s worst-case CPU, batching would add a flush boundary to a path whose bound is elsewhere.

Per step, the recorder adds exactly 2 fire-and-forget store writes (insert at step start, update at settle), off the step's critical path — plus the cluster engine's own journal write, which is the durability you asked for. Hot high-step workflows can turn the introspection copy off: workflows: { recording: 'coarse' } — see Debugging.

Composing admission transactions

Custom server-side hosts can use reserveAdmissionSubmissionInTransaction from @voltro/workflow inside an existing withAdmissionTransaction callback. This database-only operation lets the host reserve admission and bind its durable start receipt and prepared execution identity in one transaction. A failed binding must roll back the capacity reservation too. Do not open a nested transaction or call the workflow engine inside that callback; dispatch only after commit. Ordinary workflow calls keep using the framework admission path.