Declarative flow control
debounce, singleton, concurrency, throttle, rateLimit, batch, priority, timeouts and onFailure — declared on the workflow, enforced before the run exists.
Everything on this page is declared on workflow({...}) and enforced at the admission boundary — the moment ctx.workflows.start(...) is called, before a durable run exists.
That timing is the whole point. Once a run is enqueued, the only tools left are cancel and sleep, and neither of them un-spends the durable entity. So "run this at most once per row per fifteen minutes" cannot be a primitive you call inside the body; it has to be a property of the declaration.
Not the same as Flow control. That page covers
durableQueue/processQueue/rateLimit— primitives you call inside a running workflow to bound the work it fans out. This page is about whether the run starts at all. They compose: a workflow can declareconcurrencyhere and still use a durable queue in its body.
The shape
import { Schema } from 'effect'
import { workflow } from '@voltro/workflow/define'
export const tourNarration = workflow({
name: 'tourNarration',
payload: Schema.Struct({
rowId: Schema.String,
tenantId: Schema.String,
editedAt: Schema.Number,
}),
success: Schema.Void,
idempotencyKey: ({ rowId, editedAt }) => `tour:${rowId}:${editedAt}`,
debounce: { key: (p) => `tour:${p.rowId}`, period: '15 minutes' },
concurrency: { limit: 5, key: (p) => p.tenantId },
timeouts: { start: '1 hour', finish: '10 minutes' },
onFailure: 'narrationFailed',
})Every key callback receives the workflow's own decoded payload type. A misspelled field is a compile error, not a key that quietly becomes the string "undefined" and collapses every row in your deployment into one bucket.
idempotencyKey is the execution's identity — read this first
This is the single most expensive misunderstanding in the workflow API, and getting it wrong produces a design that looks right and silently stops working.
idempotencyKey is not a dedupe window. It is the execution's identity, permanently:
const a = yield* wf.execute({ id: 'same' })
const b = yield* wf.execute({ id: 'same' }) // does NOT run — replays a's resultAfter the run completes the key is spent. A later, genuinely new invocation under that key is a silent no-op that returns the old output. Nothing errors and nothing logs, because from the engine's point of view you asked for a run it already has.
So a key must be unique per unit of work you want to happen:
`tour:${rowId}` |
wrong if the tour can ever be re-narrated |
`tour:${rowId}:${editedAt}` |
right — every edit is a new unit of work |
And a flow-control key is a different thing
Conflating the two is what makes "I need to re-arm a key" feel like a missing feature. It is not missing; it is two fields:
idempotencyKey— the execution's identity. Varies per unit of work.debounce.key/singleton.key/concurrency.key— the resource runs compete for. Stable.
"One job, fifteen minutes after the last edit, latest state wins" is then the example at the top of this page: twenty edits mint twenty identities, and exactly one is ever admitted. There is no restart: true in this API because separating the two keys is the mechanism it would have been.
debounce — collapse a burst
debounce: { key: (p) => `tour:${p.rowId}`, period: '15 minutes', timeout: '1 hour' }Starts sharing a key collapse into one pending row. The timer resets on every arrival, and the latest payload wins — which is what "narrate what settled" means.
timeout is a hard cap measured from the first start in the burst. It is optional and uncapped when unset, which is a real trade: an unbroken stream of starts arriving faster than period defers the run forever. We do not invent a default cap and we do not warn — instead the starvation is a number you can see:
voltro workflows flow
A waiting= climbing past a few multiples of your period is the signal. Set timeout when you see it, or from the start if the burst is user-driven.
singleton — one run per resource
singleton: { key: (p) => p.tenantId, mode: 'skip' } // newcomer stands down
singleton: { key: (p) => p.tenantId, mode: 'cancel' } // newcomer evicts the incumbentThere is no default mode: 'skip' discards the incoming request and 'cancel' discards the running one, and choosing for you would silently throw away work either way.
Under 'skip', start() returns the incumbent's handle — a real, pollable run. Under 'cancel', the incumbent is cancelled when the replacement actually starts, not when it is queued. That matters when you also declare debounce: evicting at queue time would leave the whole quiet period with the old run dead and the new one not yet begun.
concurrency — bound what is in flight
concurrency: { limit: 5, key: (p) => p.tenantId }At most limit runs in flight per key, across every replica — the count rides the shared admissions ledger, so three replicas with limit: 5 are five runs, not fifteen. Excess starts queue and are admitted as slots free.
pool — share one budget across workflows
// embeddings.workflow.tsx
concurrency: { limit: 10, pool: 'openai' }
// summarize.workflow.tsx
concurrency: { limit: 10, pool: 'openai' }Without pool, the limit bounds this workflow's runs. With it, every workflow declaring the same pool name competes for one budget — the shape a rate-limited provider forces: five workflows that each call OpenAI must share ten slots, not hold ten each.
key still partitions within the pool: give each member key: (p) => p.tenantId and the shared budget applies per tenant.
The admission ledger stores concurrency keys as JSON tuples [pool, key], using
null for an unpooled key. Control characters are escaped for SQL text storage;
pool names and application keys cannot collide by containing a separator. Treat
this stored representation as diagnostic data, not an application-generated ID.
For custom runtime adapters, reserveAdmissionSubmission is a low-level
database-only reservation primitive. Supply a stable submission ID and the
schema-encoded complete engine payload. It atomically stores that input and
one admission lease; retrying the same ID returns the original payload, caller
and selected intent IDs before checking capacity again. An ID cannot be reused
by another workflow or tenant. Keep this table beside the resident queue.
The primitive does not invoke an engine, consume queue rows or acknowledge an
execution; those remain dispatcher responsibilities. It is not an exactly-once
execution guarantee, and the framework drainer is not yet wired to this primitive.
After observing engine acceptance, acknowledgeAdmissionSubmission atomically
links the submission and its lease to the execution ID and optional run ID.
Repeated acknowledgements are safe; a different execution or run is refused.
A failed write rolls back both links. This records observed acceptance, not
successful workflow completion or proof that an unacknowledged call never ran.
Low-level dispatchers must distinguish a reusable debounce slot from a caller's
receipt. recordWorkflowStartReceipt stores a separate stable caller ID in the
same transaction as the pending write. Allocate that ID before transaction
retries. bindWorkflowStartReceipts binds the waiting receipts to a persisted
submission in the transaction that takes the queue rows. The binding is paged
and records its completion: retrying an old submission cannot collect receipts
from a later burst occupying the same slot. The framework start adapters now
persist these caller receipts; dispatcher binding and the waiting protocol are
not yet wired through these helpers.
takePendingAdmissionSubmission combines reservation, receipt binding and queue
removal in one resident database transaction. Supply a stable submission ID and
the selected pending IDs. Its synchronous prepare callback receives the current
encoded rows read inside that transaction; decode and assemble the engine input
there, without external I/O. A missing, postponed or dead-lettered selected row
returns selection-unavailable without taking a partial batch. A previously
transferred submission is returned before reading reusable queue slots, so its
retry cannot consume a later burst. A failed binding or deletion rolls back the
entire transfer. This helper does not dispatch the engine. The gate now recovers
completed, unacknowledged transfers before processing its pending queue, even
when that queue is empty. It retries the persisted input through the admitted
start path and acknowledges observed acceptance. Failed starts stay recoverable;
the bounded recovery cursor advances past failures so a full failing page cannot
starve later submissions. pendingAdmissionSubmissions(store, limit, afterId)
exposes the same recovery window to custom adapters. Workflow idempotency remains
the engine's execution identity; this is not proof of exactly-once effects.
Ordinary controlled drains and admitted delayed arrivals now produce these
submissions before engine I/O. The submission failure lifecycle, operator-discard
and timeout receipt outcomes, and receipt waiting remain unfinished. Custom AdmissionDrainDeps adapters must provide
transaction(work) using withAdmissionTransaction on the resident store,
preserving payload codecs and serializing with direct arrivals. The callback
is database-only and may replay; calling work(store) without a transaction
does not satisfy this contract.
Submission retries now use the same recovery path in standalone drainTick
and the CLI gate. Attempt count, last error and the next attempt time live in
the submission's retry field, not in its consumed queue row. An acknowledged
execution wins over a late failure report. Missing acknowledgement does not
prove non-execution, so this path must not report abandonment using the old
intent. Terminal failure handling remains unfinished. If a queue read fails
after completed recovery, the outcome retains that recovery's counters, adds
a failure and requests another tick; examined: false marks the incomplete pass.
At workflows.deadLetterAfterAttempts, an unacknowledged submission stores
retry.stoppedAt and stops automatic retry. This is unknown acceptance, not
a dead letter: it does not invoke onFailure, release its concurrency slot or
erase caller receipts. The normal 24-hour lease backstop cannot free unknown
acceptance either. A later proven engine acknowledgement can still bind the
execution; changing the retry limit does not restart an already stopped record.
false leaves automatic retry unlimited. Inspect/reconciliation commands for
these stopped submissions are still unfinished; the pending-intent retry
command does not operate on them.
Recovery isolates failures to persist retry metadata per submission, reports both the dispatch and recording error, and continues with the rest of its page. Already acknowledged starts remain counted. A full page requests the next page promptly; at the end of the scan only actual retry deadlines remain. Thus a full page of stopped records does not continuously schedule itself, and future retry deadlines found on an earlier page are not lost.
Malformed submission metadata is validated per record during recovery. It
counts as a failure, remains stored and is not sent to the engine; valid
neighbors and later pages still run. The warning identifies the submission
without printing malformed values. The public typed
pendingAdmissionSubmissions reader remains strict and rejects a malformed
page instead of silently omitting records. Recovery isolation is not automatic
repair of damaged data.
A singleton reservation with unknown acceptance also blocks new starts under
that key, including after the lease backstop. Both skip and cancel return
a queued receipt with deferral mode singleton until the incumbent's execution
is confirmed. Voltro cannot supply a skipped execution or cancel an invented
ID. Once confirmed, the declared skip/cancel behavior applies. The queue keeps
the actual singleton key; unrelated keys remain independent.
Every member of a pool must declare the same limit — the boot fails otherwise. Two limits for one budget is a contradiction, and silently picking either would enforce a number somebody did not write.
throttle vs rateLimit — late, or gone
They are mutually exclusive, and declaring both is a boot error.
throttle: { limit: 100, period: '1 minute', key: (p) => p.tenantId } // QUEUES the excess
rateLimit: { limit: 100, period: '1 minute', key: (p) => p.tenantId } // DROPS the excessReach for throttle when every start must eventually run. Reach for rateLimit when the excess is genuinely surplus and running it late is worse than not running it.
A dropped start is never silent: start() resolves to a handle with status: 'dropped' and a retryAfterMs, and the drop is a row in the admissions ledger with its key and reason.
throttle has no burst knob. The window is sliding, so its maximum instantaneous burst is already exactly limit; a separate knob could only duplicate it.
batch — many starts, one run
export const refreshIssues = workflow({
name: 'refreshIssues',
payload: Schema.Struct({ items: Schema.Array(IssueRef) }),
success: Schema.Void,
idempotencyKey: ({ items }) => `refresh:${items.length}:${items[0]?.issueKey ?? ''}`,
batch: { item: IssueRef, key: (i) => i.tenantId, maxSize: 100, timeout: '30 seconds' },
})
await ctx.workflows.start('refreshIssues', { tenantId, issueKey: 'ABC-1' })Callers start it with a single item; the workflow's own payload is the batch shape. That mismatch is checked at declaration time — a decode failure on the batching replica is a failure nobody is watching.
batch.item is therefore also what an arriving start is validated against. A caller's payload is judged by the item schema; the run the drainer eventually starts is judged by the workflow's own payload. Two schemas, because there are genuinely two shapes — and a WorkflowPayloadError naming items on a start() call would be the framework asking the caller for the batch it is supposed to be building.
The timeout is a deadline, not a quiet period: it does not reset per item, or a steady trickle would never flush. batch and debounce cannot both be declared for exactly that reason.
priority
priority: (p) => (p.urgent ? 100 : 0)Higher runs first out of the pending queue. Ties break by arrival, so an all-default deployment is FIFO rather than dialect-dependent.
Scope — read before porting BullMQ priority lanes.
priorityorders the admission queue only: the pending rows a deferring control (debounce, batch, throttle, concurrency), a pause, or a delayed{ at }start has parked. A start that is admitted immediately never competes with anything — it goes straight to the engine, whatever any other start's priority says. There are no cross-workflow priority lanes, no preemption of running work, and no ordering between two starts that both found a free slot. If urgent starts must overtake normal ones, put both classes behind the sameconcurrencylimit (or a sharedpool), so every start passes through the queue thatpriorityorders.
timeouts
timeouts: { start: '1 hour', finish: '10 minutes' }startbounds how long a start may sit in the admission queue, measured from the first arrival in its group. A debounced run that never gets a quiet moment is a job that silently did not happen.finishbounds the run itself once admitted, and also tightens the crash backstop on its concurrency slot.
Both reach the same onFailure signal as an exhausted retry. timeouts.start removes the queued intent before a run exists. timeouts.finish durably changes the running row to cancelled; the cancel request carries its cause, and the transaction that makes the cancellation terminal reads it back and writes the onFailure report in the same commit. Seeing the same old row again does not emit another report, and a run that finished on its own before the cancel landed gets none. Declaring timeouts.start on a workflow that cannot defer is a boot error — it would never fire.
Each admission gate scans running rows in its own physical store for finish timeouts, across all result pages. A deployment-wide run listing is not the authority for a local gate. Dev and serve discover physical queues from the durable namespace directory, including after a restart without the writer's post-commit callback. Admission visits at most 100 namespace entries per source per tick and continues on the next tick. New arrivals use their caller's current resident store; changing tenant ownership does not authorize consuming its old queue. Every queue uses the same deployment-wide pool coordinator. Pool release recovery requires retained resident release evidence, not a timeout or missing run history. Operator pauses apply across all these queues.
If a later namespace page remains unreadable and its cursor stops advancing, discovery restarts from the roots. Healthy queues are revisited rather than starved behind the failed page. This restart does not turn incomplete discovery into a successful report; the unavailable source remains reported.
Custom low-level admission hosts can provide AdmissionDrainDeps.readPauses to
combine their coordinator and resident pause sets. It is read outside resident
transactions; a local transactional recheck can add a pause, not remove a
host-wide one. Normal workflow declarations need no additional configuration.
Physical admission keeps polling because the primary store's change channel
cannot prove that it sees every regional or namespace write. Partial discovery,
release or drain failures produce one workflow admission: physical recovery incomplete
warning; accessible placements continue. The warning carries the same numbers
the verdict is made from — discoveryProblems, placementFailures (each with
its placement, phase and the database's own reason), releaseRowFailures and
drainFailures — and it is written when the picture changes, not on every
tick: a condition that stays the same second after second is logged once, a
different one is logged again, and workflow admission: physical recovery healthy again closes it. The last admission report stays readable at any time
and distinguishes discovery completion from successful processing.
A recovery failure is not idle success, and it is not work either. A tick that
found no admissible row but could not open a placement, read a release census
page or complete a namespace page reports idle: true beside its
placementFailures, discoveryProblems and releaseFailures counts, and the
scheduler backs off to the same five-second reconciliation bound a healthy idle
process reaches. The failing placement is retried there, never disarmed, and
never at the base interval for as long as the condition lasts: a condition that
does not clear by itself used to hold the admission task at one tick per second
and write one claim row per second with it. Incomplete discovery is different —
a further page is known work, so the next tick follows at the base interval.
The release census that frees a pooled workflow's global permit pages the resident submissions in the store's own order and cursor semantics; a page the store returns is walked as given, and only a page that repeats its cursor row or an identity within itself is refused as a stalled cursor, naming both identities. A census page that fails restarts that placement's census from the start on the next tick instead of re-reading the failing page: an owner released behind the cursor since is visited again, and the failure stays reported.
Cancellation and staleness use independent paginated visits to the same physical placements. A cancellation event is matched against its resident runs and pending inputs, not against a deployment-wide run list. Each journal keeps its own watermark; the aggregate cancellation report therefore has no single watermark. Placement failures appear in the sweep's failures and do not stop reachable stores. Stall events are recorded in the same store as the stalled run, allowing the next visit to deduplicate them. The engine-wide unread-message diagnostic still runs once per scheduled tick, not once per namespace. These changes do not yet make the queue inspect listing a complete cross-placement inventory.
Within each store, staleness scans advance in (startedAt, runId) order, bounded
by runPage. Already reported old runs cannot permanently hide later pages.
Failed candidate reads retain the cursor; exhaustion restarts the census so
runs can be examined again. The built-in host manages these cursors. Low-level
sweepStalledRuns callers pass the returned nextRun as afterRun on the next
visit to the same store; retain their previous cursor on candidate-read
failure and omit it after successful exhaustion. A full page is not proof that
another run exists: an empty final page may be needed.
For a bounded view across physical stores, use the authenticated endpoint in
both voltro dev and voltro serve:
curl -G "$API_URL/_voltro/inspect/workflows/flow-control/physical" \
-H "Authorization: Bearer $VOLTRO_INSPECT_TOKEN" \
--data-urlencode 'limit=100'Read data.placements: each entry identifies its placementId and contains
its own flowControl sample, without queued payloads or caller contexts.
rowWindowMayBeLimited warns that an intent or admission window filled.
To continue directory discovery, send the non-null data.next string using
--data-urlencode "after=$NEXT". A null cursor ends discovery, not necessarily
the row inventory. Check data.failures independently of discoveryComplete:
a visited placement may be unreadable even when discovery finished. The inspect
envelope remains explicitly incomplete because these are registered-placement
samples, not an exhaustive queue census.
Copy an entry's placementId into Physical queue placement in either
dashboard and choose Load placement. Empty selects the primary store.
Queue reads, input validation, retry and discard use that selection; workflow
pause/resume remains deployment-wide. Changing placement clears open action
confirmations. The server resolves current placement ownership again for each
operation and refuses an unavailable or moved placement instead of falling
back to the primary store. The panel refuses actions if the returned placement
does not match its selection.
For direct calls, add placementId to the query string of
/workflows/flow-control and /workflows/flow-control/validate, or to the JSON
body of /workflows/flow-control/retry and /workflows/flow-control/discard
under /_voltro/inspect. Writes require both the read bearer token and the
separate x-voltro-inspect-write factor. The body still names intentId.
A rejected placement returns HTTP 409. Omitting placementId selects the
primary store; it never searches for a matching intent in other stores.
Bulk run operations and exhaustive cross-placement row pagination remain
separate work; selecting a queue placement does not change their scope.
Run-history HTTP reads accept the same placementId query parameter:
/workflows/runs, /workflows/stats, /workflows/runs/:id/steps,
/workflows/runs/:id/events, and /workflows/children?parentExecutionId=....
The selected identity is echoed in data.placementId. Each read revalidates
ownership; an unavailable or moved placement returns HTTP 409, never a primary
row with the same ID. These are direct HTTP options; the admission panel's
selection does not automatically select a placement in the separate run-history UI.
Discard is durable: the resident admission transaction records the decision,
settles unbound caller receipts as cancelled and deletes the pending input
together. A failed audit or receipt write rolls back the discard, rather than
deleting work without evidence. Low-level discardIntent requires a store with
transactional and insertIgnoreWithOutcome. Inside an existing
withAdmissionTransaction, call discardIntentInTransaction on its transaction
store; never nest admission transactions. Neither operation calls the engine.
When finish lands. The bound is enforced by the admission drainer's tick, not by a timer on the run. Each visited placement contributes its earliest known finish deadline. A placement beyond the current discovery page is checked on a later tick; with many namespaces, allow for a full discovery cycle rather than assuming a one-tick maximum. Pending work is not required to keep this check running.
onFailure — the signal that replaces the sweep
onFailure: 'narrationFailed'A workflow name, not a function. A closure cannot be journaled: the failure may be noticed by a different replica, minutes later, after the process that held it is gone.
It fires for every way a run fails to deliver — not only an exhausted retry:
- the body failed and retries are spent — raised from the run's terminal transition, when its row turns
failed timeouts.finishcancelled an overrunning runtimeouts.startexpired a start that never got a slot- the workflow was renamed away while starts were queued
The last two produce no run row at all, which is exactly why polling listRuns({ status: 'failed' }) could never see them.
The named workflow's payload is WorkflowFailureReport:
export const narrationFailed = workflow({
name: 'narrationFailed',
payload: Schema.Struct({
workflow: Schema.String,
payload: Schema.Unknown,
errorTag: Schema.NullOr(Schema.String),
errorMessage: Schema.NullOr(Schema.String),
runId: Schema.NullOr(Schema.String),
executionId: Schema.NullOr(Schema.String),
reason: Schema.String,
failedAt: Schema.Number,
}),
success: Schema.Void,
idempotencyKey: ({ workflow, runId, failedAt }) =>
runId === null ? `failed-before-run:${workflow}:${failedAt}` : `failed-run:${runId}`,
})For failures of an existing run — including timeouts.finish — runId, executionId, and the original payload are populated. Use runId as the stable idempotency identity. Pre-run failures (timeouts.start and a queued start whose workflow was removed) have both IDs set to null; they are emitted once when their durable pending row is consumed.
A report is a row in _voltro_workflow_failure_reports, written in the transaction that decides the failure — a run's terminal transition, a queued start's expiry or dead-letter, a submission's rejection — and handed to the handler's arrival queue in that same commit. The failure recovery on every drain tick delivers it from the row: a process that dies anywhere between deciding and delivering leaves nothing to redo but the delivery. There is no process-local callback on this path, on purpose; a callback a dead process never calls is the one way a report could still be lost.
There is no onFailure for an onFailure — a handler that fails is logged and not re-notified, because the alternative is one run per failure per level with no floor.
encrypted
encrypted: trueEncrypts every durable copy of this workflow's data with the key behind .encrypted() columns — one key, one rotation story:
VOLTRO_FIELD_ENCRYPTION_KEY=… # voltro dev mints it; `voltro secret generate field-encryption` prints oneNo plugin is needed; under governancePlugin({ fieldEncryption: { secretKey } }) the key is read from the secret named there. Rotating it keeps every sleeping run resumable — see rotating the key.
A workflow's data is not stored once. The durable engine journals the payload, the result and every step's result so a run can resume after a crash; the run row keeps the payload and the result for the dashboard and listRuns; the step journal keeps each step's input and output; a queued start keeps its payload until it is admitted; a failure report keeps the payload for its handler. For a workflow over personal data or free text, each of those is a copy outside the .encrypted() boundary of the table the value came from. With encrypted: true, all of them hold ciphertext:
| Where | What is sealed |
|---|---|
engine journal — cluster_messages, cluster_replies |
each payload field, the result, a typed failure and a thrown error (a defect), the same for every step, and the value awaitSignal / awaitUpdate / awaitEvent / awaitSignalSuspending received |
_voltro_workflow_runs |
each payload field, the result, the error message |
_voltro_workflow_run_steps |
each step's input, output, error message and error cause |
_voltro_workflow_run_events |
a signal's or update's value, an update's result, an error message — ids, names and tags stay readable |
| queued starts, submissions, failure reports, control results | the payload, result or error message they copy, including a queued start's lastError |
The payload stays an object: its field names (which the schema already publishes) remain readable, each value is a ciphertext. It works on every SQL dialect the engine runs on — PostgreSQL, MySQL, MariaDB, SQL Server, SQLite and Turso — because the engine stores what the workflow's schemas encode, and the declaration seals those encodings.
What the body and your readers see is unchanged. The body receives the values it was started with, a step's result replays as it was returned, a thrown error reaches the caller as it was thrown, and the readers the access rules already let in — the live run, step and event rows your app subscribes to, the client's workflow hooks, ctx.workflows.listRuns(...), the dashboard — receive the values, opened. The protection is at rest.
Error text is data. A schema error prints the value it rejected (Expected number, actual "…"), an HTTP client quotes the body it was refused with, a database error names the row. So an error's message is sealed wherever a value is: on the run, the step, the event, the failure report, the queued start. Its tag — the class name, or a typed failure's _tag — stays readable, because grouping, filtering and alerting run on it.
Nothing leaves the process in plaintext. Logs and error reporters are outside the database, and usually outside your infrastructure: a log line about an encrypted workflow's failure carries the error's class and says the full cause is recorded sealed on its row. Every server error is published with contentSensitive: true for an encrypted workflow, and the Sentry plugin then sends the class, the stack frames and the run's identity, not the message, the cause or a typed failure's fields — tagged voltro.contentWithheld, so the issue still groups, with a pointer to the run. A reporter of your own reads the same flag and can use withheldServerError(event) from @voltro/protocol.
Declaring it without a cipher is a boot refusal, not a warning, and an encode that finds no cipher fails rather than writing plaintext: a fallback would leave the declaration reading as protection while the rows are readable.
What it costs, measured with the real AES-256-GCM cipher: about 90 µs to seal and open a 1 KB value; a run of 20 steps with 1 KB results showed no difference beyond the noise of the machine (plain 127–173 ms, encrypted 108–191 ms per run, SQLite). A run event carrying a value costs one extra lookup of its run.
Turning it on is safe for runs already in flight. A value written in plaintext before the declaration — a run started earlier, a queued start — is still read. Turning it off again keeps sealed values readable as long as the cipher is configured.
What it hands on is encrypted too. An encrypted workflow's onFailure handler must be declared encrypted, or the boot refuses and names both: the failure report carries the payload and the error. The handler receives both opened and stores them sealed. A durable queue it enqueues into must be declared durableQueue({ …, encrypted: true }) — a queue journals its payload, result and failure like a workflow — or processQueue fails, naming the queue.
What it does not cover, so it is not assumed:
- keys your code derives —
debounce.key,batch.key,concurrency.key,rateLimit.key. They are stored as your functions return them. Derive them from ids. The engine stores only a hash of theidempotencyKey; - field NAMES of the payload — they are what your schema already publishes;
- what your own code does with a value: a table a step writes to (use
.encrypted()there), a log line your code writes, an HTTP call it makes; - a payload field declared with a property-signature transformation (a default, a renamed key) — refused at definition time, naming the field, rather than left plaintext. Apply the default in the body.
Before 0.82.0 this option was encryptSteps and covered the step journal only; voltro update renames it.
cancelOn — stop live work when a correlated event arrives
workflow({
name: 'tourNarration',
payload: { rowId: Schema.String, issueKey: Schema.String },
idempotencyKey: (p) => `tour:${p.rowId}`,
cancelOn: [{
event: 'jira.issue.deleted',
schema: JiraIssueDeleted,
match: (event, payload) => event.issueKey === payload.issueKey,
}],
})Both sides are typed: event from the entry's own schema, payload from the workflow's.
Why it is a declaration and not a race inside the body. You can express "stop when the issue is deleted" with awaitEvent and an interrupt. That works while the body is running. It does not work while the run is sleeping for six hours, suspended on a signal, or still sitting in the admission queue — which is the case you wanted cancellation for. The event has to reach a run whose fiber is not executing anything, and only something outside the body can do that.
So it is swept: a coordinated tick reads events published since a durable watermark, resolves each declaring workflow's live runs, and cancels the ones that correlate.
The sweep holds this watermark when listing, cancellation or queued-discard
delivery fails. It continues evaluating the current window and reports individual
failures, then retries that window on the next tick; newer events cannot move a
failed operation behind the cursor. The first tick persists its lookback boundary
before reading the journal, so a failed cold start does not slide past old work.
A persistently failing operation can delay later windows: inspect the reported
failures rather than treating a held cursor as healthy idleness. This protects
retryable failures; an adapter's resolved reply still must accurately distinguish
durable acceptance from confirmed execution. The CLI also treats a false
cancellation result as unconfirmed: it appears in cancelHealth().failures,
does not increment cancelled, and does not advance the event window.
Cancellation and queued-discard callbacks receive the complete planner decision,
including eventId, not only the event name. The CLI carries this ID in the
cancellation detail and the queued-discard audit reason. Repeated evaluation of
one publication retains that ID; a name alone cannot distinguish two publications.
This causal identity does not itself prove control acceptance or completion.
It also discards queued starts. Cancelling only the running one leaves a debounced or concurrency-queued duplicate to start seconds later, against the row that was just deleted — the exact outcome the declaration was meant to prevent, arriving late enough that nobody connects the two.
| Field | Meaning |
|---|---|
event |
The name, exactly as ctx.events.publish writes it |
schema |
Decoded before match runs. An event whose shape does not decode is reported and never matched — cancelling on an event you could not read is cancelling blind |
match |
Required. Write match: () => true if you really mean "every live run" |
within |
Only cancel runs started within this window before the event |
reason |
Stored in the resident control request and its control-settled event; defaults to cancelOn:<event> |
match has no default for the same reason singleton.mode has none: the omitted case is "cancel every live run of this workflow", which is a legitimate thing to want and a catastrophic thing to acquire by forgetting a line.
A run that started after the event is never cancelled. Without that rule, a sweep catching up after a deployment gap reads an hour of history and kills runs that started in the meantime — and the symptom (fresh work cancelled for no visible reason) looks nothing like its cause (a restart).
Bulk cancel and bulk replay
A bad deploy leaves four thousand runs that must all stop, or four thousand that must all be re-driven once the downstream is fixed.
voltro workflows cancel-many --workflow tourNarration --reason "bad deploy"
voltro workflows cancel-many --workflow tourNarration --reason "bad deploy" --commit
voltro workflows replay-many --status failed --mode redrive --limit 200 --commit
Three things are deliberately stricter than the obvious design:
--limitis required and there is no "all". The cap is the blast radius.truncatedin the result says whether more matched, so "did I get all of them" stays answerable without an unbounded verb ever existing.- It is a dry run unless you pass
--commit. The default for a verb that can stop a thousand runs is the one that stops none. The dashboard panel enforces the same order: the apply button does not exist until a preview has returned a number. --reasonis required for a cancel. The current bulk path logs it. Its resident request-identity and durable settlement integration is still incomplete; do not treat that log as a persisted cancellation cause.
The result is per-run, not a count: succeeded, failed (with the reason for each) and skipped (with what made each ineligible) are three different outcomes. A bulk op that reports "4000 cancelled" while forty failed is how people learn not to trust bulk ops.
Eligibility is fixed by the verb: a cancel acts on running and suspended runs; replay --mode redrive on failed only (redrive resumes from the step that died, which only exists for a failure); replay --mode retry on failed and cancelled. --mode has no default because the two cost very different amounts.
Delayed one-off starts — start(..., { at })
Run this once, later — without a schedule and without a sleep at the top of the body:
await ctx.workflows.start('orders.remind', { orderId }, {
at: new Date(Date.now() + 24 * 60 * 60_000), // tomorrow, this time
})The start is parked as a durable row in the same pending queue the controls above use (mode: 'delayed'), and the coordinated drainer fires it when at arrives — it survives restarts and fires on whichever replica drains, never from an in-process timer. The handle comes back status: 'queued' with deferral: { mode: 'delayed', dueAt }.
Semantics worth knowing:
atis an absolute instant, deliberately — not adelayduration. A delay is measured "from when?" (enqueue? admission? retry?) and every queueing system answers differently; an instant has no such ambiguity and composes with the schedule/backfill surfaces, which are also instant-based. A relative delay is one line:at: new Date(Date.now() + ms).- At
at, the start becomes an ordinary ARRIVAL. Declared controls judge it as of that moment — a debounce collapses it into whatever window is open then, a rate cap can drop it (recorded, as always).{ at }delays the arrival; it never outranks a control. - An
atin the past starts immediately — "no earlier than" is already satisfied. { at, wait: true }follows the durable caller receipt through its delayed arrival and any further admission deferral, then waits for the accepted execution. It has no implicit deadline; usestart()withoutwaitfollowed bywait(handle, { timeoutMs })for a bounded wait. Waiting inside a mutation transaction remains refused.- Each
{ at }start is its own row. Two delayed starts never collapse into one — unlike debounce, nothing about{ at }says the second supersedes the first. Want collapsing? That isdebounce, and they compose.
Prefer this over sleep as the first step of the body when the wait precedes the work: a parked row costs one row, a sleeping run costs a durable execution the whole time.
What a deferred start returns
start() no longer always returns a running handle:
const handle = await ctx.workflows.start('tourNarration', payload)
if (handle.status === 'queued') { /* handle.deferral.dueAt tells you when */ }
if (handle.status === 'dropped') { /* over a rateLimit cap; it will NOT run */ }
if (handle.status === 'skipped') { /* handle.executionId is the incumbent */ }executionId is null for queued and dropped, because there is no execution and there may never be one. Inventing an id there would produce a handle that polls status: 'unknown' forever.
For queued, handle.id is a durable caller receipt; handle.deferral.intentId
is the queue slot used by voltro workflows flow. They are distinct: several
debounced callers can share a slot, and a later burst can reuse that slot after
the earlier run starts. Mutation receipts are provisional until commit and
roll back with their pending work. Custom enqueue/admission adapters return
receiptId alongside intentId and dueAt and persist both atomically.
A dispatch without an answer is queued, not an error. A controlled start
is dispatched to the engine after admission. When that dispatch fails without a
definite answer — the database pool was exhausted, or the acknowledgement was
lost after the engine took the run — the run may already exist, and recovery
retries the dispatch. start() returns status: 'queued' with
deferral.mode: 'unacknowledged', and handle.id is the receipt recovery binds
to the resulting run. Do not tell a user the start failed; follow the receipt
(start(…, { wait: true }) and run(…) do). A definite rejection still throws.
Blocking callers
Every controlled direct arrival now also persists a resident caller receipt,
including the arrival that fills a batch. Its input, receipt and submission
transfer share one transaction; no extra runnable queue row survives commit.
Until engine acknowledgement the receipt resolves as pending, never accepted
merely because capacity was reserved. This adds a receipt row per direct caller;
the ordinary successful start() still returns its execution handle.
ctx.workflows.run(...), start({ wait: true }) and child({ wait: true })
follow a queued admission through its durable caller receipt. They wait for
acceptance before polling that exact execution; they do not resubmit the
arrival payload or bypass debounce / batch / throttle / concurrency.
A batching receipt follows its batch's execution and result, not a separately
started item. A terminal non-execution receipt throws WorkflowStartNotExecuted;
the accepted execution preserves its original success or failure Cause.
An immediately accepted blocking call also only observes the accepted execution;
it does not submit the input a second time to join its result.
Successful start({ wait: true }) and child({ wait: true }) return handles
with status: 'succeeded', not running. This also applies to delayed and
replayed waits. A failed waited execution rejects with its original Cause.
Update exhaustive handle-status checks and custom wire schemas, and rebuild
generated clients together with the server.
rateLimit still throws WorkflowRateLimitedError for run();
singleton: 'skip' with a confirmed incumbent throws WorkflowSingletonHeldError.
A singleton with unknown acceptance remains queued until its receipt resolves.
Blocking calls have no implicit deadline; use start() then
wait(handle, { timeoutMs }) when the caller needs a bounded wait. Waiting inside
a mutation transaction remains refused; wait only after commit.
Upgrade: WorkflowDeferredResultError has been removed. Remove its imports
and catch branches; do not mechanically rename them to WorkflowStartNotExecuted.
The latter means admission has settled without execution, not that a caller is
still queued. voltro update identifies references needing this manual change.
ctx.workflows.executionId(name, payload) validates the execution payload.
For a batching workflow this means the assembled { items: [...] }, not an
individual arrival item. An item has a caller receipt; its batch execution
identity only becomes known after admission assembles the batch.
Run-history replay lookup happens only after an arrival passes through admission
without being queued or dispatched by a control. It must neither derive a batch
identity from an item nor replace a current admission decision with an old run.
Pausing a workflow
voltro workflows pause tourNarration --reason "deploying a fix"
voltro workflows unpause tourNarration
A pause makes starts collect, never discard — so you come back to a backlog rather than a hole in the data. unpause drains it.
The pause is a row, so it applies fleet-wide; each replica picks it up on its next drain tick (~1 s).
Seeing what happened
voltro workflows flow
voltro workflows flow --workflow tourNarration --format json
voltro workflows flow validate
voltro workflows flow retry <intentId>
voltro workflows flow discard <intentId>
or GET /_voltro/inspect/workflows/flow-control.
This is not optional colour. A debounce that collapses nineteen starts into one is indisputably correct behaviour and indistinguishable from nineteen starts vanishing — unless something writes down that it happened. So every decision is a row in _voltro_workflow_admissions, with its key, its reason, how many starts folded into it, and how long it waited.
Three questions it answers:
| Question | Where |
|---|---|
| "Twenty edits, one run — did that work, or did I lose nineteen?" | collapsed |
| "Nothing has run for an hour. Stuck, or quiet?" | waiting= on the queued row |
| "Why did my run not start?" | the ledger's outcome + reason |
Permanently invalid queued payloads
Workflow apps also declare _voltro_workflow_control_requests in their schema.
It is the resident storage foundation for stable control-request identities and
attempt history, not a public cancellation API or proof of cancellation.
Engine delivery, generation-bound confirmation and recovery are still being
integrated; an empty table must not be interpreted as evidence that no control
operation was requested through the current APIs.
Low-level hosts can call acceptWorkflowControlInTransaction(tx, request) and
beginWorkflowControlDispatchInTransaction(tx, requestId) from @voltro/workflow
inside withAdmissionTransaction. Allocate the request ID before the transaction;
its run, workflow, execution, generation and operation cannot change on replay.
These database-only functions neither invoke the engine nor confirm cancellation.
A dispatch result is not permission to call an unfenced engine interrupt:
the engine boundary must enforce the target generation too.
Acceptance also captures owner: { tenantId } from the original run generation.
owner: null means ownership was not proven; it is not equivalent to
owner: { tenantId: null }. Replaying a request does not replace its owner,
even if the run's subject changes or the run row is removed. Caller-supplied
metadata cannot supply this authority. Public receipt authorization must use
this stored owner; the request ID alone is not a credential.
readWorkflowControlInTransaction(tx, requestId) reads the accepted request and
its nullable result. The owning body's terminal transaction stores that result
alongside the run and capacity release for every existing request of the same
generation. Cancellation confirms cancel/terminate; success or failure retains
the actual outcome as already-terminal, including output and errors. A resume
request is never confirmed by cancellation. This stored evidence survives a later
body generation. Dispatch returns settled without another attempt when a result
exists. A result-write failure rolls back the terminal transaction. Acceptance
after completion stores the matching generation's observed outcome immediately
as already-terminal, even if it was cancelled: the new request did not cause
that earlier exit. A missing/invalid completion timestamp refuses that observation;
a different generation's outcome is never substituted. Public control adapters
and suspended-run/resume delivery remain under integration; absence of a result
is not proof that an engine did nothing.
completeWorkflowRunGenerationInTransaction also accepts
outcome: { status: 'cancelled' } for a confirmed exit of the owning body
generation. It atomically records cancellation and releases linked and
dispatched pre-acknowledgement reservations, without creating a failure report.
An interrupt promise resolving does not satisfy that precondition. This
low-level transaction does not itself interrupt the engine or confirm delivery.
The run recorder now records cancelled when the engine has requested
interruption and the body exits with an interruption-only cause. Capacity
remains held while that body is still active. A real handler failure remains
failed, even if cancellation was requested at the same time. Confirmed
cancellation emits run-cancelled, not a failure report; low-level onTerminal
and recordRun callbacks must handle the cancelled status. This recorder
boundary does not yet replace all control API and ParentClose paths.
The workflow facade's cancel and resume now return
{ operation, kind, requestId, snapshot }. Read the observed run state from
snapshot, not from the requested operation. already-terminal preserves the
original completed result. confirmed requires an observed cancellation exit;
an unresolved command or observation yields unknown. A running snapshot alone
does not prove a particular resume was accepted. The engine-only facade has no
resident request ID (requestId: null) and therefore never claims accepted.
The durable control-request adapter is still pending. poll, query and wait
continue returning snapshots and classify completed interruption-only exits as
cancelled. Custom facade implementations must adopt WorkflowControlResult;
voltro update reports this migration for affected source.
Custom makeWorkflowFacade hosts can supply control({ operation, workflowName, executionId, requestId }) as the resident owner. It returns WorkflowControlResult and
owns acceptance, dispatch and observation together. With this adapter configured,
the facade performs no engine poll, interrupt or resume around it. Its rejection
propagates; it never triggers a second, engine-only attempt. The built-in host
integration is still pending. Do not claim accepted before the resident request
commit, or confirmed from a void engine reply.
For these hosts, cancel(name, executionId, { requestId }) and
resume(name, executionId, { requestId }) forward the caller's non-empty identity
unchanged, including through lazy and caller-context facades. Reuse it for retries
of one intent, not for a new operation or target. The resident adapter reads the
original receipt before considering the current run generation; a retry cannot
bind that identity to a replacement body. Without a resident adapter, supplying
requestId refuses before any engine command instead of promising durable replay.
Use readControl({ operation, workflowName, executionId, requestId }) to observe
that receipt without submitting another request. Custom hosts provide the
makeWorkflowFacade option readControl; unsupported hosts refuse explicitly.
This lookup does not require the workflow to remain in the current declaration
registry. The resident adapter must validate receipt ownership and authorization;
the request ID is not permission. A missing receipt is unknown, and a failed
read propagates rather than silently resubmitting the operation. A saved terminal
result remains the original result even after a replacement generation starts.
Executing bodies and suspended-entry cancellation use the same validated, generation-scoped pending-request scan. Settled receipts do not trigger another interruption, including when they fill a complete page before a pending request. A malformed record causes a diagnostic read failure and a retry, not an abort of the body. Repair of malformed control data is not automatic.
The internal resident control request can persist an optional cause with a
reason and structured detail in the acceptance transaction. Replays retain
the first cause alongside the original request timestamp, including after
dispatch and completion. Settlement writes one control-settled run event per
request in the same transaction: its payload carries requestId, operation,
generation, cause, kind and status (plus enteredGeneration for resume).
An already-terminal observation does not claim the request caused that outcome.
Retries preserve one event; event-write failure rolls back settlement. Inspect
the run-event timeline/table to retrieve this evidence. Terminal run events
(run-succeeded, run-failed, run-cancelled) also commit with their owned
transition, receipts and capacity release, once per generation, including a
version-incompatible entry refusal before the body runs. An event-write
failure rolls that transition back. Custom recording hosts must include the
event table in the resident schema; terminal events no longer pass through
custom recorder.recordEvent. Use emit/onTerminal for post-commit observation,
not a second writer. Live push, wakeup cleanup and parent-close callbacks remain
best-effort; their recovery integration is still unfinished. Not every control
producer is connected yet, and a diagnostic log is not a durable cause.
Every terminal generation also stores a _voltro_workflow_parent_closures
intent in that transaction. It retains the parent run/execution/generation,
terminal status and timestamp, with an initially empty child cursor and
completion timestamp. It contains no workflow payload or caller credentials and
stays in the resident store. Repeated completion cannot create another intent.
Custom resident schemas must include this table. The resident admission drain
visits these intents even when it skips pending arrivals. An intent is not evidence that
children have stopped, and existing parent-close callbacks remain best-effort.
The low-level stageWorkflowParentClosureInTransaction(tx, closureId) stages
at most 100 children per resident transaction, storing cancel/terminate receipts
and cursor together without engine I/O or child status writes. abandon remains
untouched. Replay retains the first child generation and original parent cause.
scanHasOpenChildren retains open-child evidence across pages. End-of-scan wraps
for another pass when any bound child was open; only a complete pass observing
none sets completedAt. Malformed child status is not terminal evidence.
Completion also requires every accepted parent-close control to be settled.
The indexed parentClosureId is derived from the original control cause; a
missing child row or malformed receipt cannot substitute for its result.
An unresolved receipt retains the closure for recovery even when no child rows remain.
Recovery isolates malformed intents without logging their contents. The shared
CLI worker requests another recovery probe within five seconds even with an idle
pending queue (the configured base interval remains the floor); this costs
periodic database checks instead of depending on a wake after
commit. Neither built-in boot path installs an interrupt-and-write parent-close
callback: child finalizers must finish before resident recording confirms cancellation.
Physical store enumeration remains unfinished.
closeWorkflowChildrenForParent and its options/decision/store/emitter/logger
types have been removed. Custom hosts must run drainTick against the same
resident store as terminal recording. Never replace this with direct child
status writes after an engine interrupt. WorkflowParentCloseStatus remains
the recording observer's terminal status type.
Recorded children carry parentGeneration beside parentExecutionId.
Ambient recorded nesting captures the parent body's generation; explicit
recording hosts can supply parentGeneration with their parent execution ID.
An explicit parent never borrows an ambient generation. Staging skips children
of another parent generation and refuses cancellable children with no binding.
Explicit parentGeneration now survives child caller metadata, queue decoding,
durable start-context storage and both boot paths into the recorder. Retrying
the same execution with another parent generation is refused; a fresh host
reads the original binding. The recording wrapper now supplies its committed
body generation before the application context is constructed; both boot paths
bind this identity to that context's child facade, including Promise-based calls.
execution.generation is the current body's identity; execution.parentGeneration
belongs to its parent and must not be substituted. Event-triggered starts from
that context preserve the same parent generation through caller metadata and
start-context storage. Child entry checks that exact parent generation under the
resident admission lock. A recorded closure, even with completedAt set, makes
cancel/terminate children terminal before their body is constructed; the run,
control receipt, terminal event and capacity release commit together. Missing
parent identity or generation is refused for these policies. abandon is exempt.
Queued acceptance can still precede this check. Resident drain recovery stages
controls for children that entered earlier; an accepted control alone does not
complete their parent closure.
Server callers can pass { requestId, cause: { reason, detail } } as the third
argument to facade cancel/resume. Lazy and caller-context wrappers preserve
it, and the resident adapter stores it on first acceptance. A retry on a fresh
host with the same request ID does not replace either the original generation
or cause. Engine-only hosts refuse these options instead of dropping the cause.
The shared canceller reads an explicit receipt even after the run became
terminal, preserving a confirmed cancellation without submitting another request.
Repeated confirmation is not proof that a producer's follow-up ran exactly once.
cancelOn derives that identity from the workflow, run and triggering event,
so a repeated sweep reuses the same control intent. The shared canceller forwards
the original reason and event details. Built-in resident boot/recovery integration
is still incomplete: an engine-only host refuses this persistent control path;
the sweep retains its watermark and reports the failure rather than claiming
cancellation.
@voltro/protocol exports the browser-safe WorkflowRunSnapshotSchema and
WorkflowControlResultSchema; runtime result types derive from this shared
contract. The facade validates resident adapter responses on submission and
observation. Contradictory results, such as confirmed cancellation with a running
snapshot or acceptance without a receipt ID, fail explicitly without falling back
to an engine command. Responses must also match the requested operation, workflow,
execution and any caller-supplied receipt ID. This validates response consistency, not authorization or
durability. Exception: an unrecorded submission returns unknown with a null
receipt ID, even when the caller proposed an ID. That is not acceptance. A receipt
lookup must retain the requested identity, including when it cannot find it.
The built-in cancel/resume RPCs and useWorkflow().cancel/resume
return this complete result too. Rebuild generated clients with the server;
inspect kind and snapshot before reporting completion. The RPC path does not
write its own terminal run state or event after a cancel command. Inspect response
migration is still in progress; its boolean responses are not equivalent receipts.
To retry the same intent after an uncertain browser transport result, retain an ID before the first call and pass it as the second argument:
const control = { requestId: crypto.randomUUID(), cause: { reason: 'operator' } }
await workflow.cancel(run, control)
// If the transport outcome is unknown, reuse control for this same intent.The workflow must declare controls: for any of this to reach it from a client; without it the built-ins refuse every caller but the system and admin:full.
resume(run, control) accepts the same shape. The built-in RPC input carries
it as control: { requestId, cause? }; both boot paths authorize the run before
forwarding it to the resident owner. An explicit ID requires a resident control
adapter; unsupported hosts refuse. Generating another ID for each retry creates
another intent. Omitting control does not establish caller-owned retry identity.
Use workflow.readControl({ workflowName, executionId, operation: 'cancel', requestId: control.requestId }) to observe the same receipt without submitting
another control. The built-in __voltro.workflow.control query authorizes the
stored receipt owner, not the current run row; it can therefore read retained
evidence after run retention. Missing, foreign and unowned receipts refuse with
the same authorization error, preserved by the hook as a typed rejection.
The hook reads one snapshot and closes the stream;
call it again when another observation is needed. It does not automatically poll
or resume anything. Host readControl wiring and physical resident-store routing
are still required.
A configured onFailure handler must accept its report: a dropped start or a
singleton-skipped incumbent does not prove that this report was received, and
the failure recovery retries the handoff with a new receipt generation. A
queued notification is accepted work, not proof its body has completed.
An absent onFailure declaration is an explicit absence, not a delivery to
retry. The internal _voltro_workflow_failure_reports table provides
immutable incident storage beside the resident queue: replaying a report ID
retains its original encoded payload, recipient and failedAt. Every source
writes there in its own transaction — a run's terminal transition (an
exhausted body, a timeouts.finish cancellation), a queued start's expiry or
dead-letter, a submission's rejection — and the recovery delivers from the
row. Recording a report does not mean its handler has received it; the
report's acceptedExecutionId says when it did.
Dispatchers persist dispatchAttempts before engine I/O. Local validation
before the first call, or a guaranteed pre-engine input rejection of that
first call, can prove non-acceptance. A prior or competing unacknowledged call
cannot: a later schema rejection leaves acceptance unknown and retains capacity.
Cancellation or acknowledgement failures after a successful start are not
input rejection. Low-level isPermanentStartError classifiers must guarantee
this pre-engine boundary, not merely classify errors as non-retryable.
Low-level hosts can use recordWorkflowFailureInTransaction(tx, record) from
@voltro/workflow inside withAdmissionTransaction, in the same resident
transaction as the source transition. Allocate the incident ID before that
transaction and retain it for recovery; a reusable debounce slot is not an
incident identity. readWorkflowFailureRecord(store, id) returns the stored
record and refuses malformed data. Neither function invokes a handler or
acknowledges delivery. Never put engine or notification I/O in the transaction.
handoffWorkflowFailureInTransaction(tx, reportId, now) transfers the stored
report to the existing durable arrival queue and records its caller receipt
in the same resident admission transaction. Retries reuse that receipt and the
original recipient, payload and failedAt; an explicitly absent handler stays
unconfigured. A missing or contradictory linked receipt is refused, not repaired
by creating another start. kind: 'queued' describes the handoff, not the
handler's acceptance or live execution status. Handler flow control still applies
when draining the arrival. Proven submission rejection now invokes this handoff
automatically, in the transaction that records the report and releases the lease.
A failed receipt write rolls back all three; retry keeps the original report and
handler. AdmissionDataStore adapters must forward the resident store's native
insertIgnoreWithOutcome, including inside transaction(work); do not emulate
it with a pre-read or upsert.
The normal drainer now discovers resident reports after restart, even when its
local pending-queue hint says idle. reconcileWorkflowFailureInTransaction
checks their receipts: a recorded drop or skip creates a new receipt/arrival
for the original report, preserving the previous terminal receipt. Retries use
the shared bounded exponential backoff (1 s to 5 min, deterministic upper bound)
and never precede a rate-limit retry deadline. Concurrent reconcilers commit one
new attempt, not one per replica. Unknown engine acceptance retains the existing
submission and prepared execution ID; it never creates another arrival.
Only a matching acknowledged submission records acceptedExecutionId on the
report. This confirms handler acceptance, not completion or external delivery.
Report recovery is bounded and paged; malformed reports remain stored, are
reported by ID without their payload, and do not block neighboring records.
scanned includes examined failure reports as well as submissions and arrivals.
When a failure-delivery submission is rejected before engine acceptance, its
own failure is recorded without another onFailure notification. The resident
report/receipt relationship identifies that call; an ordinary invocation of the
same workflow still uses its declared handler. This prevents recursive start-
rejection notifications, without suppressing the original report's retry.
Proven handler-input rejection also retries the original report with backoff,
without mutating the rejected submission or its receipt. The rejection must have
a matching failure record and no earlier ambiguous dispatch; a rejection marker
alone does not prove non-acceptance. A repaired handler can then accept the report.
Missing, foreign-owned or dead-lettered arrivals now fail reconciliation instead
of returning queued. The warning includes failureId and a safe reason
(arrival-missing, arrival-owner-mismatch, arrival-dead-lettered or
rejection-proof-invalid); it never creates another arrival from absence alone.
Replacing a debounce payload settles its still-pending caller receipts with
terminal: { kind: 'superseded', at } in the replacement transaction. Those
receipts cannot be bound to the later execution. Failure recovery retries each
superseded report with backoff and a new receipt; the original reports remain
unchanged. No payload comparison or application-schema guess decides ownership.
Other source hooks and recovery policy for operator-discarded, missing or
dead-lettered arrivals remain unfinished. Batch handlers retain all selected
reports in items; both size-triggered and timeout-triggered batches preserve
their submission and payload through a lost engine acknowledgement. Every
included report is acknowledged only after the matching submission ACK. Receipt
acceptance alone is not proof of body completion or external delivery.
When a delayed arrival is deferred into another queue slot, the drainer now creates that slot, forwards every unbound caller receipt and consumes the old arrival in one resident transaction. Bound receipts do not move. Concurrent drainers re-read the arrival under the lock; a foreign receipt rolls back the whole transfer. An admitted delayed arrival binds its receipts and consumes all selected inputs together with an immutable submission in that same transaction, using the admission already recorded rather than reserving a second slot. It then uses the prepared execution and acknowledgement path described above. A lost engine reply leaves that submission recoverable; recovery retains its original identity and input. This is not a claim of exactly-once external effects or complete coverage of every failure source.
For a recoverable start outside a mutation transaction, pass requestId and an
explicit at to ctx.workflows.start. Create and retain both values before the
first attempt; retries must reuse the same identity, instant, workflow, tenant
and payload. requestId must be non-blank and at most 64 characters. Identified
starts always use a durable receipt, even when at is already in the past;
normal admission still runs when the queue drains. Concurrent matching attempts
reuse the original receipt instead of creating another pending input. A different
payload or instant with the same identity is a conflict. Reuse also requires the
same caller and parent generation; a fresh trace ID on a retry does not change
the request identity. A new intentional start
needs a new identity, even if its payload is identical. An identified start cannot
be combined with a mutation's transactional enqueue or internal admitted dispatch.
Without requestId, existing immediate and delayed-start behavior is unchanged.
Keep the request identity outside a transient response if recovery after a lost
reply is needed; this option does not persist a browser journal for you.
Low-level admission adapters can bind a caller-owned start identity by passing an
optional requestFingerprint to recordWorkflowStartReceipt. The fingerprint
is a lowercase SHA-256 digest of the immutable encoded request; it is retained
after the pending row is consumed. The receipt separately binds the workflow
and tenant. readWorkflowStartRequest(store, { id, workflowName, tenantId, requestFingerprint }) reads that binding without admitting or executing work:
it returns the original receipt, returns undefined for a missing identity,
or rejects conflicting reuse. A legacy receipt without a fingerprint cannot
be adopted as a new request. Perform lookup and the first receipt/intent writes
in the same resident admission transaction. Matching a binding is not evidence
that the workflow completed; resolve the receipt for its actual outcome.
This read helper is a low-level adapter contract; requestFingerprint is not a
ctx.workflows.start option and is computed by the admission adapter.
readWorkflowStartReceipt(store, id) now exposes terminal, initially null.
A queued caller later refused by rate limiting gets
{ kind: 'dropped', at, retryAfterMs }; a singleton skip gets
{ kind: 'skipped', at, executionId, runId } naming the incumbent, not a newly
started run. Both delayed and controlled queue drains persist the decision,
settle all affected receipts and delete their inputs atomically. A failed write
keeps the inputs and receipts available for retry. A terminal receipt is never
rebound or forwarded when the queue slot is reused; it cannot simultaneously
claim a submission. This records the admission outcome, not workflow completion.
Use recordWorkflowStartReceipt for new receipts rather than hand-writing a
row that omits terminal: null.
WorkflowStartTerminalOutcomeSchema and WorkflowStartTerminalOutcome from
@voltro/protocol define these browser-safe admission outcomes. The persisted
receipt uses the same schema. A cancelled start needs no execution ID; a skipped
start's execution/run IDs identify the incumbent. These values do not by
themselves prove an execution was interrupted.
WorkflowQueuedCancellationResultSchema and WorkflowQueuedCancellationResult
from @voltro/protocol describe cancellation evidence addressed by receiptId.
kind distinguishes accepted, confirmed, unknown and already-terminal.
Its observation is separate: pending before execution evidence, admission
with a settled start outcome (or proven rejection), execution with a real run
snapshot, or unavailable with the missing-evidence reason. Only the execution
branch carries a snapshot. Missing requests have requestId: null; loss of run
evidence retains the known request ID. Contradictory combinations refuse decoding.
The runtime facade exposes cancelQueued(target, options) and
readQueuedCancellation(target, options), where target contains workflowName,
receiptId and a stable requestId. Custom hosts supply separate resident
adapters on makeWorkflowFacade; missing adapters refuse. The caller-bound
facade supplies identity, and these operations must run after a mutation commits.
Custom WorkflowsAppContext implementations must provide both methods or
explicitly refuse them. Both voltro dev and voltro serve install the resident
adapters. They locate an existing cancellation request before its source receipt,
so retries and reads retain the original result even after receipt removal.
Missing requests remain unknown; reads never create a cancellation request.
Both operations require an authenticated caller before physical discovery;
an anonymous identity cannot control a tenant-less request.
The generated RPC group includes the __voltro.workflow.queued.cancel action
and __voltro.workflow.queued.control query. Both take workflowName, receiptId
and requestId; cancellation additionally accepts cause. They return the
shared queued cancellation result. The server supplies caller identity from
the authenticated request, never from input fields. Regenerate the client group
when adopting these procedures. useWorkflow exposes cancelQueued(input) and
readQueuedCancellation(target). The latter reads one snapshot and closes its
subscription; a typed query denial stays the original error. Custom
WorkflowState implementations must provide both methods.
Automatic queued control recovery is not yet fully connected: an accepted
response is not confirmation that a running execution has stopped.
Custom hosts can use cancelQueuedWorkflowStartInTransaction and
readQueuedWorkflowCancellationInTransaction from @voltro/workflow inside
their resident admission transaction. Supply the authorized tenantId with
the target identities. These adapters perform database work, not engine I/O;
authorization and selection of the original physical store remain host duties.
Low-level adapters can call resolveWorkflowStartReceipt inside the resident
admission transaction, supplying receiptId, workflowName and the authorized
tenantId. Its WorkflowStartResolution distinguishes pending, accepted
with a confirmed executionId, terminal with the queue outcome, and
rejected with a proven failureId. A prepared ID alone stays pending.
Missing or contradictory owner records, dead-lettered inputs and stopped
unacknowledged submissions fail visibly instead of waiting indefinitely.
ctx.workflows.wait(queuedHandle, options) follows that receipt until confirmed
acceptance, then polls the accepted execution. Its timeoutMs budget starts
before receipt resolution and is not restarted at acceptance. This bounds the
polling loop; it does not cancel a database or engine read already in progress.
The current caller is bound by the context facade; a receipt is not authority.
Custom runtime hosts must supply resolveQueuedStart on makeWorkflowFacade.
The dev and production hosts locate the receipt across declared physical stores and registered tenant namespaces, then reread it under the resident admission lock. They require explicit caller identity and matching workflow/tenant ownership. A duplicate receipt ID, incomplete discovery or unavailable store refuses the observation; a root-store match does not hide another unsearched placement. This lookup does not move receipts or dispatch queued work.
No execution means no run snapshot: WorkflowStartNotExecuted from
@voltro/runtime reports reason as dropped, skipped, superseded,
cancelled, abandoned or rejected, plus handleId, workflowName, nullable failureId
and nullable incumbentExecutionId. This also applies to directly dropped or
skipped handles. To observe a skipped incumbent deliberately, use the explicit
workflow-name/execution-ID wait overload. Inside a mutation, both wait overloads
refuse: return the provisional receipt and wait only after the transaction commits.
If a controlled or batched queue declaration is removed, the drainer records
each original input and caller in a resident failure report and settles its
receipts with { kind: 'abandoned', at, failureId } in the same transaction as
queue removal. It does not call an ephemeral failure callback or invent a
current handler for a removed declaration. The report therefore has no handler;
inspect and reconcile it explicitly. The same outcome is preserved when a
delayed start's workflow disappears. This does not yet cover every dead-letter
path, nor does it make a queued handle waitable by itself.
Queued start-timeouts persist the failure report, admission history, terminal
receipts, input removal and the configured handler's durable arrival together.
A batch keeps its full ordered item list in one report; no execution is claimed
before the handler accepts it. A notification timing out does not recursively
invoke that handler's own onFailure. A batch mixing notification and ordinary
callers is retained with a reconciliation error because it has no single safe
recipient decision. These are resident transactions, not atomic transactions
with the workflow engine or an external provider.
After a transient batch drain failure, all still-matching inputs receive their retry counters and one shared, jittered retry deadline atomically. Updating only the first item—or drawing a separate deadline for each—would let a later tick process an unintended subset of the failed batch. A changed selection is not overwritten by this retry update.
executePreparedWorkflow(definition, decodedPayload, executionId) from
@voltro/workflow is the low-level submission primitive for an identity already
chosen and durably bound to that input. It works with Voltro workflow()
definitions and does not evaluate their idempotencyKey again. Validation,
engine annotations and suspended retry settings remain in effect. It returns
the accepted execution ID, not completion of the body. An explicit fourth
argument { discard: false } waits for the typed body result under the same
ID, preserving declared failures without deriving the key or executing a second
body. Record acceptance before joining the result: a body failure is not a
refused start. The caller must commit
admission, identity/input binding and caller context before invoking it outside
the transaction; this primitive performs none of those persistence steps.
The runtime facade accepts an internal preparedExecutionId only alongside
admitted: true, without wait, at or transaction enqueue. It records caller
context under that same ID before submission. This is a dispatcher contract,
not an application admission bypass. Controlled queued submissions, direct controlled
arrivals and recovery prepare and persist this identity before dispatch. A size-closing
batch transfers all earlier caller receipts with its inputs in the admission transaction.
run() and start({ wait: true }) confirm acceptance before joining the body result;
a later body failure does not reject that admission. Uncontrolled immediate starts
do not use this submission journal. Failure-report delivery uses the queue; terminal
body-failure reporting still needs its separate durable completion boundary. Memory-engine replay
tests prove lost-ACK retry through a fresh facade, not engine restart survival.
Custom WorkflowAdmissionOutcome adapters return accept(): Promise<string> for
kind: 'start', not a post-execution commit callback. Persist the reservation,
encoded input and receipt transfer first with takeAdmittedSubmissionInTransaction;
dispatch outside the transaction through acceptAdmissionSubmission. Return only
the durably acknowledged execution ID. Do not reserve the same admission twice or
execute a second time in the facade. Codec adapters encode each batch item separately.
Recovery can win before the original caller dispatches. The dispatcher rereads the
resident binding and returns its existing acknowledgement; a stale in-process
submission object is not evidence that preparation or acceptance is still pending.
Low-level terminal generation boundary
The framework inventory includes _voltro_workflow_queued_cancellations beside
the resident admission tables. The recorded entry binds retained cancellation
requests for dispatched submissions to its own generation before application
code, across all request pages. A matching request closes the run, settles its
control receipts and releases its local reservation in that same transaction,
even before the submission ACK is recorded. Internal acceptance after entry
captures that resident generation under the same lock; its control monitor
observes the committed request, and only actual exit confirms cancellation.
If that owner is already cleanly suspended, acceptance settles cancellation in
the same transaction without an engine wake. Retrying acceptance may settle
that same original owner after it suspends; a read-only status query never does.
Public queued-cancel acceptance and
post-entry recovery are still unfinished; this table is not a direct-write API.
An unresolved submission cancellation also stops a replacement entry for the
same run and execution before its body starts. It does not transfer or confirm
the original generation's control: that observation remains unknown because
preventing a replacement body does not prove that the old body exited. Only the
replacement generation receives the new terminal event. A clean suspension is
different: its body has exited, so entry can settle the original generation
before any replacement body starts. Settled controls do not become new stop
requests. Process-restart recovery and fencing external work from an old runner
are not proved by this entry boundary.
_voltro_workflow_runs.runnerGeneration stores an opaque body-entry identity.
It is independent of executionId, which survives replay, and reclaimCount,
which resets after a clean exit. Allocate the identity before the transaction;
use enterWorkflowRunGenerationInTransaction for both initial insertion and
re-entry inside the resident transaction. It preserves the original run ID/input,
returns the prior snapshot for crash/version checks and distinguishes created,
entered, replayed, cancelled and already-cancelled. A pending resident
cancel/terminate for a cleanly suspended generation (runnerEnteredAt: null)
completes that owner before replacement:
run, control results and capacity release share the entry transaction. Neither
cancellation decision may invoke application code; only cancelled represents a
new transition to observe. already-cancelled also prevents a repeated engine wake
from reviving the execution. A clean suspended entry also confirms pending resume
requests for the previous generation in the same transaction. Their immutable
result contains status: 'running', generation and enteredAt, not a terminal
completedAt. A running replacement is not resume evidence; cancellation takes
precedence. Later completion does not erase the entry receipt. Automatic wakeup
and resume delivery remain unfinished.
Replaying the same entry identity never increments the
reclaim count or resets an already completed run. A missing or unreadable write
must prevent body execution, not silently disable recording.
completeWorkflowRunGenerationInTransaction accepts that owner and an outcome.
Under withAdmissionTransaction, it checks ownership, records the terminal row,
failure report and handler queue arrival, and releases all admission reservations.
Any write failure rolls the transaction back. superseded and not-running
mean no new completion; do not emit a terminal hook for either result. Failure
identity includes the generation that actually completed. Retrying report delivery
preserves that identity; it does not reopen a failed execution.
The production recorder now commits entry before invoking the body. Custom
wrapWorkflowExecuteWithRunRecording hosts must supply transaction using
withAdmissionTransaction(store, work) against the same resident store as
store. The callback is database-only and may be replayed. An entry commit
failure fails execution; it is not a best-effort observation failure.
Execution recording stores must provide query: the active body observes resident
cancel/terminate requests matching its run, workflow, execution and generation.
It interrupts its own fiber, not a replacement addressed only by execution ID.
The monitor performs one sequential read followed by a one-second pause while
idle and stops when the body completes or suspends. Read errors retry without
claiming cancellation. Control producers, suspended-run delivery and durable
result receipts are still being integrated; this monitor alone does not complete
the control lifecycle.
Custom recording stores may implement query as an instance method: the
replay-shape reader preserves its receiver when loading prior steps.
Stored run inputs and durable failure reports retain the encoded full workflow
payload. Dev and serve provide the declared input encoder, including the full
batch envelope rather than one item. Custom recorder hosts with transformed
inputs supply encodePayload: Schema.encodeUnknownSync(definition.payloadSchema);
omitting it promises JSON-only input. Encoding failure prevents entry and body execution.
The recorder uses this boundary for body success and failure. Its onTerminal
callback is an observation after commit, not a lease-release or handler-start
callback. Custom hosts configure durable delivery with failureContext:
{ handler, callerContext, tenantId }. Framework boot paths provide the bound
identity with cached authority removed. Failure reports retain the original
start identity and correlation from the durable start-context
binding. A repeated delivery keeps its incident's first recipient and timestamp.
A distinct later incident may use the then-current onFailure declaration;
the handler is not frozen at the run's first start. Execution permissions are
still resolved anew, never copied into the delivery context. Notification-body failures are recorded
without a secondary handler, including completion before dispatcher ACK.
It does not fence application side effects or establish cluster runner ownership.
Suspension and crash-loop parking also check the current running generation
inside the resident transaction. A late old suspension neither changes the row
nor reports a suspension event or failure. A failed commit propagates; successful
suspension retains admission capacity and the reclaim counter.
Version rejection uses the same generation-checked terminal transaction as body
failure, including the original stored payload, durable failure arrival and
admission release. Its application body does not run; callbacks and metrics
follow only a committed outcome. Resident parent-close recovery stages controls
in bounded, committed pages; a failed later page preserves earlier cursor progress.
An unresolved control keeps its closure open even when the child row disappears.
For a clean suspended body, cancelSuspendedWorkflowRunInTransaction(tx, owner, now)
settles an accepted cancellation only for the same recorded generation with a
cleared entry marker. It commits run state, receipt, terminal event and admission
release together. The resident adapter and parent-close recovery use this boundary
without an engine wake. A later engine entry observes cancelled and does not run
the body. Active bodies still confirm through their own monitor and finalizers.
Resume delivery and physical-store enumeration remain unfinished.
An ACK that arrives after body
completion also releases that admission's lease in its transaction; a newer
running generation retains capacity. Body completion also releases reservations
bound through dispatched prepared submissions when ACK remains absent. It does
not fabricate an acknowledgement or release an undispatched proposal. Recovery
still confirms the same prepared execution. Do not infer that a complete process-crash protocol
follows from these body-completion tests.
For a resident admission submission,
prepareAdmissionSubmissionExecutionInTransaction(tx, submissionId, proposedExecutionId)
binds the application-derived ID before dispatch. Compute the proposal outside
the database transaction. Competing proposals return the first committed
preparedExecutionId; always use the returned value. Preparation neither
acknowledges execution nor releases admission capacity. An ACK with another
ID is refused. A submission already dispatched without an ID cannot safely
choose one afterward. Both controlled submission producers and recovery invoke
the same preparation step before the dispatch marker. Custom AdmissionDrainDeps
must implement resolveExecutionId(name, decodedPayload) without executing work
and honor the fourth preparedExecutionId argument to startAdmitted unchanged.
The shared boot builder passes that ID to the runtime facade. A contradictory
returned ID fails before singleton eviction, not merely at acknowledgement.
Low-level admission hosts may provide acquireSubmissionCapacity(submission).
It runs after resident commit, outside SQL, before engine preparation in both
queue transfer and submission recovery. Return false for ordinary capacity
waiting: no dispatch attempt or failure retry is consumed; recovery revisits
after one second. Replays must reuse a durable permit for the same submission.
The direct CLI gate also checks before returning its start decision and retains
a queued receipt on false. This adapter does not by itself enable deployment
pool wiring or terminal permit release; those integrations remain unfinished.
A retained succeeded or failed run refuses body re-entry for the same
execution ID. An engine may replay its own complete journal result without
entering the body; if it lost that result, the framework refuses rather than
rerunning side effects. The run-history output may be truncated and is not used
as a substitute result. Reconcile the engine journal; do not submit a new ID
merely to bypass this refusal. After run-history retention, its durable
parent-closure record still prevents a new body entry, including for cancelled
executions. These records are not TTL history, even after child-close recovery
finishes; do not delete them to clear a queue. A late wakeup of a suspended
generation still needs separate fencing.
Admission rows persist concurrencyReservation (pool, resolved key and limit)
with their local reservation. Cross-store capacity adapters read this committed
metadata, not the current workflow declaration. A changed deployment cannot
silently move an existing submission to a new pool. A concurrency admission
without matching reservation metadata requires reconciliation, not a guessed
current limit. The low-level readAdmissionSubmission reader validates stored
metadata and distinguishes an absent row from malformed data.
A queued payload is decoded against the workflow's current input schema
when the drainer starts it. While the pending intent still owns the work, a
WorkflowPayloadError dead-letters the intent and records its onFailure
report in one transaction, logging workflow flow control: intent dead-lettered;
the report reaches the handler through the failure recovery.
After queue transfer, the submission owns the error instead: proven rejection
sets its failureId and removes it from dispatch recovery. Its original input
and retained report remain stored. Do not use the pending-intent retry command
to re-arm this state; submission reconciliation and report delivery are not
finished yet.
Any other failure — the engine unavailable, a store error, an unknown workflow
name after a rename — is retried, but not on every tick: each failed attempt
backs the intent off (one second, doubling, capped at five minutes, jittered),
so one undrainable intent no longer produces a warning every two seconds on
every pod. After workflows.deadLetterAfterAttempts failed attempts (default
50 — several hours under that backoff, which is no longer transient in any sense
a retry can help) the intent is dead-lettered exactly like an invalid payload:
one onFailure report written with the dead-letter, the row retained, voltro workflows flow retry to re-arm it.
false disables the cutoff.
The payload, error, attempt count, and dead-letter time remain visible in
voltro workflows flow and the dashboard. Remove that retained row explicitly
after inspection:
voltro workflows flow discard <intentId>Each queued row also records the app version and a fingerprint of the workflow input schema that existed when it was queued. Before deploying a schema change, validate the backlog against the release currently running the command:
voltro workflows flow validate
voltro workflows flow validate --limit 1000 --format jsonThe result says valid, invalid, workflow-missing or
schema-unavailable per intent and compares queued/current schema
fingerprints. It never returns the stored payload. This is a diagnostic and
does not mutate the queue.
After deploying a workflow/schema fix, explicitly re-arm one retained dead-letter:
voltro workflows flow retry <intentId>Only a dead-letter can be retried. The command retains its payload and
provenance, clears terminal/error/attempt state, and lets the coordinated
drainer validate it again. Use discard when the work is obsolete; do not
retry unchanged invalid bytes merely to make the warning recur.
A later debounce arrival for the same key replaces the obsolete payload and
clears its dead-letter state, so a valid new edit can run without an operator
first deleting the old row. If onFailure itself cannot be signalled, the
intent stays live and visible for another attempt; Voltro does not mark work as
handled before its failure path ran.
Cost
A workflow that declares no control takes exactly the code path it took before this feature existed — no query, no branch beyond one map lookup.
A workflow that declares one pays only for that one: an undeclared control costs zero round trips. The rate/throttle window reads at most limit rows, which is why limit is a throughput knob and not somewhere to put 10⁶.
The drainer runs on one replica per tick through the same claim arbiter the cron scheduler uses. N replicas draining at once would each see a free slot and each take it.
Measured admission throughput
Measured, not estimated — node packages/cli/scripts/admission-throughput.mjs in the framework repo drives the real gate, facade and drainer against the in-memory reference store (slope across N=500/1000/2000 starts, median of 3 sequential repeats; Apple-silicon dev machine, 2026-08):
- no controls (passthrough): ~1 µs/start (~800,000 starts/s) — the do-nothing path really does nothing.
- through a concurrency gate: ~110 µs/start (~9,000 starts/s) — dominated by the reference store's unindexed admission-state scan, which grows with the ledger; a real dialect serves that read from an index, but also adds its round-trips. Read this as the machinery's worst-case CPU floor, not a database benchmark.
- durable park → drain → start: ~4 µs/row (~240,000 rows/s) of pure machinery per queued row.
The deployed ceiling is min(these numbers, what your database serves for the admission reads/writes) — on any SQL dialect the database is the bound long before the gate is. That is also why there is no key-group batching in the admission path: at ~9k gated starts/s worst-case CPU, batching would add a flush boundary to a path whose bound is elsewhere.
Per step, the recorder adds exactly 2 fire-and-forget store writes (insert at step start, update at settle), off the step's critical path — plus the cluster engine's own journal write, which is the durability you asked for. Hot high-step workflows can turn the introspection copy off: workflows: { recording: 'coarse' } — see Debugging.
Composing admission transactions
Custom server-side hosts can use reserveAdmissionSubmissionInTransaction from
@voltro/workflow inside an existing withAdmissionTransaction callback. This
database-only operation lets the host reserve admission and bind its durable
start receipt and prepared execution identity in one transaction. A failed
binding must roll back the capacity reservation too. Do not open a nested
transaction or call the workflow engine inside that callback; dispatch only
after commit. Ordinary workflow calls keep using the framework admission path.