Debugging & inspection

Voltro records workflow runs, step attempts, and lifecycle events into framework tables. The dashboard, CLI, and inspect endpoints all read the same data.

The signal dialog distinguishes delivery from consumption. A matching signal-received event recorded while the send request is still in flight also confirms consumption; events preceding that send do not. Signal and update fields have associated labels, and validation errors are announced without submitting invalid input. Live refreshes keep the open message dialog and its draft for the same app, run, and replica selection. Once consumption is observed, the confirmation closes automatically; background rerenders do not restart its timer.

The workflow detail keeps its Start run action available after previous runs exist. If a run ends before signal consumption is confirmed, the signal dialog shows that state and any available run error instead of continuing to promise consumption. The operator closes that notice explicitly. Update errors preserve the entered name and payload; a timeout does not prove that the submitted update was cancelled.

Run details link a child back to its parent and display the parent-close policy. Parent/child links retain the selected replicas. When a parent link identifies an execution rather than a run-record ID, the dashboard resolves the run first and loads its steps and events using the canonical run ID.

#Why admission keeps ticking

Use the existing live-process command:

voltro workflows flow --format json

Read admissionDiagnostics, one { name, admission } entry per responding process, not a merged fleet verdict. Older servers report admission: null. The same snapshot is data.admission on authenticated GET /_voltro/inspect/workflows/flow-control. It describes the serving process even when the queue view selects a resident placement. Reading it does not wake a task, claim a tick, or write diagnostic rows.

• scheduler.currentIntervalMs is the scheduled delay, not elapsed time; nextTickAt is the absolute Unix-millisecond deadline, or null without a timer.
• scheduler.lastDecision, claim/task failure counters, claim losses and lastOutcome distinguish idle backoff from unsuccessful attempts. The outcome is null if the latest tick did not finish successfully.
• wakeReasons counts pending-row, pause, retry and bounded-run arrivals; scheduler.wakes and coalescedWakes show how often they reached the scheduler.
• lastDrain is the latest completed drain, timestamped separately. It includes scanned/admitted/deferred/failed counts, discovery completion/problems, placement/release failures and the next known work deadline when present.
• disarmReason explains whether physical discovery, missing remote visibility, or a missing change channel prevents disarming. Physical admission keeps a five-second recovery bound; a known earlier deadline can shorten it. The generic polling ceiling does not remove this bound, and the configured base interval remains the timer's minimum non-wake delay.

All counters are process-lifetime observations, not database statement counts, fleet totals, or proof that a workflow handler executed. A healthy empty worker with default settings backs off to five seconds; one-second ticks alone do not establish a defect or an empty queue. Compare two snapshots while the writes occur. For a support report, share only admissionDiagnostics and relevant fetchErrors: the rest of the flow response can contain workflow payloads. No payloads or raw driver errors are included in admission diagnostics. Inspect remains protected by the existing read credential; do not publish it or disable coordination to make the log quiet.

#What is recorded

Table	Contents
_voltro_workflow_start_contexts	One row per started execution id: starter subject, trace id, source, parent execution id, parent-close policy, and creation time. Used for cross-runner context handoff.
_voltro_workflow_execution_placements	Internal routing metadata: hashed execution identity, hashed workflow name and physical placement ID. Registered for workflow apps. It contains no caller or payload and is not run-history retention data.
_voltro_workflow_restart_requests	One row per resident restart reservation: the run, execution and generation being re-entered, its journal delivery and replies. Read by the admission drainer's release census on every tick; registered for workflow apps.
_voltro_workflow_pool_permits	Coordinator-side permits for shared concurrency.pool capacity: pool key, limit and whether the permit is held or released. Metadata only — no payload or caller identity. Registered for workflow apps.
_voltro_workflow_runs	One row per run: id, tag, executionId, status, payload, workflowVersion, workflowPatches, output, error fields, subject, start source, timing, trace id, parent execution id, parent-close policy, plus the crash-loop bookkeeping runnerEnteredAt + reclaimCount.
_voltro_workflow_run_steps	One row per step attempt: step name, attempt number, recorded input, retry metadata, output or error, duration.
_voltro_workflow_run_events	Lifecycle events: run-started, run-succeeded, run-failed, run-suspended, run-resumed, run-redriven, run-cancelled, timer-set, timer-fired, signal-awaited, signal-sent, signal-received, update-requested, update-received, update-completed, update-failed, nondeterminism-suspected, run-crashlooped, run-stalled.
_voltro_workflow_events	Domain events emitted through ctx.events.publish(...): event id, name, payload, source, subject, trace id, occurred time.
_voltro_workflow_event_deliveries	One row per workflow trigger delivery: event id, trigger id, workflow name, execution id, idempotency key, status, error.

Run status is running, succeeded, failed, cancelled, or suspended.

If a workflow start acknowledgement is lost, admission recovery checks for the already recorded body using its persisted prepared execution identity. A matching body confirms acceptance without another dispatch or capacity acquisition, even after automatic submission retries have stopped. Preparing an identity alone does not prove acceptance: a dispatch attempt and resident execution evidence must exist. After run-history retention, the durable terminal closure supplies that evidence. A current run takes precedence over an older terminal closure; the child-cleanup completion time is never used as the run's completion time. This reconciles the original start; it does not authorize an operator redrive. Repeating the original admission acceptance callback likewise returns its already confirmed execution identity without dispatching again.

#Turning the step rows off — workflows.recording: 'coarse'

Every step() costs two fire-and-forget writes to _voltro_workflow_run_steps (insert at start, update at settle) — measured, and off the step's critical path, but on a hot workflow with many steps they dominate the table's growth. 'coarse' skips both:

// app.config.ts
export default {
  workflows: { recording: 'coarse' },   // default: 'full'
}

Runtime override without a rebuild: VOLTRO_WORKFLOW_RECORDING=coarse (env wins over config; anything unrecognised falls back to 'full').

What 'coarse' does not touch, deliberately: run rows, run events (signals, timers, cancels, stall reports — everything the dead-letter view and the sweeps read), and the cluster engine's own durable journal — replay and redrive are unaffected. The cost is exactly the dashboard's step timeline: empty for runs recorded under 'coarse'. Flip it back when you need to see inside a run.

#Dead-letter: triaging failed runs

The framework does not retry a workflow on its own (see Retries — you compose retries inside execute: with Effect.retry). So a run that reaches failed is terminal: it is the dead-letter. voltro workflows list --dead-letter (or the Prometheus series voltro_workflow_runs_total{status="failed"} and the stale voltro_workflow_last_success_timestamp_seconds gauge — see Prometheus) is your queue of unhandled failures.

Each run that reaches failed also writes one warning to the server log, under voltro dev and voltro serve alike, with or without an error reporter configured: workflow run failed, with the workflow name, executionId, runId, the error tag and the cause. For a workflow declared encrypted: true the cause names only the error's class and says the full text is recorded sealed on the run.

Triage a dead-letter run one of three ways:

• Retry — voltro workflows retry <id> --request-id <intent-id> persists a deliberate fresh attempt. Reusing the same ID observes or resumes that intent; a new ID creates a separate retry. Acceptance is not execution completion.
• Restart failed steps — voltro workflows restart <id> --request-id ID --failed-steps retains a durable request for the original execution. Save ID before submitting; --observe reads it after an uncertain response. Completed steps replay; failed activities re-execute. pending means reserved, entered confirms body entry, and neither means successful completion.
• Discard it — voltro workflows discard <id> acknowledges the failure so it drops off the --dead-letter view. It is an ack, not a re-classification: the run stays status: 'failed' (the outcome + audit trail survive) and gains a discardedAt timestamp. --status failed still lists it, marked discarded; only --dead-letter hides it. Discarding a non-failed run is refused, and discarding is idempotent.

#Stuck runs and crash loops

Three failure modes are detected rather than left for whoever opens the dashboard.

A run that stops moving. A staleness sweep reports live runs (running or suspended) that have made no progress for longer than the threshold — default 30 minutes — writing a run-stalled event carrying idleMs, the reason (awaiting-signal, suspended, no-progress) and the last progress instant. It changes no run state; it is a signal, not an intervention. The classic catch is a run parked on a signal nobody ever sends.

What the report says about a running run nobody drives. On the durable engine, run-stalled reports the run's journal messages (run, activity/<step>/<attempt>), their processing and last-read observations, and shard-lock holders for unprocessed messages. own: true is the engine's own ownership observation, not an inference from database addresses. The engine re-reads old unprocessed messages; the staleness tick reports those outside its re-read window as workflow.cluster: unprocessed messages no runner has read for the re-read window. These observations identify missing progress, not proof that the owner has died.

Read own first: own: false means another runner holds the shard, not that the runner is dead. PostgreSQL advisory locks live as long as their database session. A session retained by a pooler or not yet detected as disconnected can therefore keep a shard after its client disappears. Lock-heartbeat observations (VOLTRO_WORKFLOW_FAILOVER_HEARTBEAT, 10 seconds by default) help investigate, but an old state_change is not proof of death: a paused process, event-loop stall or delayed database call can also stop updates.

Do not terminate a backend based only on its age, address, own flag or runner-table absence. Behind PgBouncer, Supavisor or RDS Proxy, client_addr identifies the pooler, not the application runner; backend_start can predate the current pod. First establish the runner's lifecycle and the exact backend-session ownership using deployment and pooler evidence. If that cannot be established, leave the session alone and involve the database operator. Backend termination is a disruptive recovery action, not a diagnostic probe or a guarantee that a run will resume on the next poll. VOLTRO_WORKFLOW_SHARD_LOCK=row uses expiring leases instead of session-lifetime advisory locks; it does not turn a stale heartbeat into proof that the previous worker has stopped. Inspect holders without changing them:

SELECT l.objid::bigint - 1000000 AS shard, a.pid, a.client_addr, a.state, a.state_change, a.backend_start
FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.locktype = 'advisory' AND l.granted AND l.objid::bigint BETWEEN 1000001 AND 1000300;

A message the owner re-reads and never processes. Unanswered engine requests can survive a stopped executor. A cancel acknowledgement alone is not proof that deleting those requests is safe: another generation may own the same execution identity. The control path therefore does not independently discard engine messages. The staleness tick reports unread messages and their shard owners without deleting them. Safe generation-fenced cleanup is not yet implemented; unread messages can therefore remain pending. Do not manually delete journal messages based only on a row status or an operator acknowledgement.

The engine is up from the boot. Both boot paths acquire the workflow runtime as a boot step (workflow runtime ready — N entity type(s) registered), so every pod registers its runner and takes its shards before the first request. This line is emitted once per successful runtime acquisition, never merely for assembling its layers. Its entities use the registered Workflow/<name> names, including Workflow/voltro.deliverWebhook when enabled. Failed acquisition emits no ready line. A pod that only built the engine on first use had no runner after a restart until something asked — and in a fleet, the first thing that asked was the once-per-fleet staleness tick, five minutes later, in one pod.

Two things it deliberately stays quiet about, because otherwise the signal is worthless:

• A run inside a durable sleep / sleepUntil whose wake instant is still in the future. That run is waiting by design, and a seven-day timer is not a stall.
• A run already reported since its last progress. A stall is reported once and again only after the run moves and stalls afresh.

Configure it in app.config.ts — every field optional:

export default defineApiApp({
  workflows: {
    staleness: {
      // Set this above your slowest single STEP, not above your longest RUN:
      // a run waiting on a durable timer is already excluded. Default 30 min.
      stallAfterMs: 30 * 60_000,
      runPage: 200,                       // live runs examined per tick, oldest first
      onStalled: async (run) => {
        await page(`${run.tag} ${run.runId} idle ${run.idleMs}ms (${run.reason})`)
      },
    },
  },
  // How often the sweep looks. `VOLTRO_STALENESS_SWEEP_MS` overrides it.
  scheduling: { stalenessSweepMs: 5 * 60_000 },
})

onStalled must not throw — a rejection is collected and reported, so one bad pager integration cannot stop detection for every other workflow.

The sweep runs on a coordinated schedule, so one firing per interval fleet-wide rather than one per replica. Unlike tasks that can disarm on arrivals, it never stops ticking when idle: those disarm because an arrival wakes them, and a run going stale writes nothing there is to wake on. That makes the cadence an unconditional cost, which is why it defaults to five minutes rather than one second — on a thirty-minute threshold that is 12 coordination rows an hour instead of 3 600.

voltro doctor runs the same detection once, for the moment you are standing in front of a deployment asking whether anything is wedged:

✗  stuck runs: 2 run(s) have made no progress
   invoices.settle  wr_01J…  idle 4h  awaiting-signal
   report.nightly   wr_01J…  idle 2h  no-progress
   Nothing was changed — this is a SUSPICION, not a verdict. `awaiting-signal`
   usually means the sender never came.

It is the one doctor rule that reads your DATABASE rather than your source, so run it where the app's DB env vars are set; anywhere else it prints a named skip rather than a clean tick. It records nothing and calls no onStalled, so running it neither pages anyone nor suppresses the background sweep's next real report.

A run that keeps killing its runner. A step that crashes the process — OOM, a native crash — cannot be caught as an error: the shard lease expires, a surviving replica claims it, and executes the same payload. Without a ceiling that rotates around the fleet forever. Voltro counts consecutive runner deaths on the run row (runnerEnteredAt is set on every body entry and cleared on every clean exit; reclaimCount counts entries that found the previous marker still set). At VOLTRO_WORKFLOW_MAX_RECLAIMS (default 3) the run is parked as suspended with errorTag: 'WorkflowCrashLooped' and a run-crashlooped event, and the body is not entered again. The counter is consecutive — any clean re-entry resets it — so a long-lived healthy run is never parked for a crash it had months ago, and an operator resume gives it a fresh budget.

A run whose fiber is parked. no-progress is the hardest of the three, because from the engine's point of view nothing is broken: the run's fiber is alive, waiting for a step reply that is not coming. Three things follow, and the first two are what people try first.

• Resume does not apply. It wakes a run carrying a terminal Suspended reply. A run stuck in running is not suspended — there is nothing waiting to be woken — so the action refuses instead of reporting a resume that did not happen.
• Redrive does not apply either. It replays a failed run from its journal and refuses one that is still running, because re-driving would race the live fiber.
• The admission slot stays held. An admission lease lives 24 hours unless the workflow declares timeouts: { finish }, which is the only mechanism that both bounds the slot and cancels the run. Declare one on any workflow whose concurrency slot matters — see flow control.

The stall report is what tells you the run is in this state at all, so treat no-progress as a page-worthy event rather than a log line.

A run replaying against edited code. See Versioning for the nondeterminism-suspected event.

#Dashboard

When voltro dev is running, the Workflows panel lists recent runs, their status, start source, timing, payload, output/error, step attempts, and events. The run detail view is the fastest way to answer:

• Which step is currently running?
• Was the run started by an RPC, ctx.workflows, a schedule, an incoming webhook, or inspect tooling?
• Is this run a child, and will it be cancelled, terminated, or abandoned when the parent closes?
• Which attempt failed?
• What input did the step receive?
• What output or typed error did it produce?
• Did a timer or signal fire?

The panel also exposes run controls for users with the right capability: cancel, retry, suspend, resume, redrive (re-drive a failed run from its journal), discard (acknowledge a failed run), send signal, and send a tracked update.

Inspect signal and tracked-update actions use the same resident target resolver as ctx.workflows.signal and ctx.workflows.update. They write into the run's validated physical store, not the boot's primary store; ambiguous or incomplete target discovery refuses instead of guessing. Updates honor the requested timeoutMs. These message actions do not establish resident coverage for the remaining run-listing and operator-control paths, whose integration is ongoing.

Store-backed inspect controls (cancel, resume, discard and retry) also resolve the run's physical owner before reading its row. Discard acknowledges the failed row in that store and remains idempotent. Missing, ambiguous or unavailable ownership refuses; it does not fall back to a primary-store lookup and claim the run is absent. This read/write placement guarantee is separate from the retry observation contract described below.

The same physical-owner resolution also applies to ctx.workflows.retry in development and production. The shared retry helper requires the resolver before reading the original run; it never searches the primary store as a fallback when ownership cannot be established.

A durable admission submission ID is bound to one encoded payload. Repeating the same ID with identical payload content reuses the reservation, even if JSON object properties are ordered differently. The comparison uses persisted JSON semantics, including normalization of signed zero and omitted object properties. Reusing it with different payload content is rejected before any new admission or engine dispatch; the original payload remains unchanged. A deliberately different request needs its own identity.

External suspend is refused. The suspend command/endpoint cannot create a cooperative engine checkpoint. It reports an error without changing the run, emitting run-suspended, or claiming the body stopped. Use workflow-owned suspension, such as suspendOnFailure, rather than a database status override.

Inspect resume delegates to the workflow facade and returns its full WorkflowControlResult; it does not write running or emit run-resumed itself. Only kind: 'confirmed' with a running snapshot proves resume. An already-running target refuses without submitting another resume; a terminal target returns its recorded already-terminal outcome. A missing runtime refuses. The dashboards retain the returned control receipt and distinguish acceptance from confirmation. Refresh status reads that receipt without repeating the action. A failed read or a response for another execution keeps the last matching observation visible and reports that the new status could not be verified. Custom WorkflowsPage integrations must return WorkflowControlResult from useCancelRun and useResumeRun, and supply useControlReceipt as a read-only DataSource<WorkflowControlResult> hook with refresh. Its input is WorkflowControlReadInput, not a new run-control request.

Set controlScope to the stable identity of the inspected app. Before dispatch, the page saves the control identity in this browser's local storage, separately from the filtered run list. The reference survives a page reload, including when the action's response was lost. Mutation adapters must forward the supplied control unchanged alongside runId; generating another request ID would lose that protection. Storage read/write failures prevent dispatch. Only identifiers are persisted, never workflow inputs, outputs or error details. This is browser- local recovery metadata, not a backup of the server's durable receipt. Clearing browser storage removes it. Each request has its own storage entry, so saving or dismissing a reference does not replace another tab's reference inventory. This does not merge independently submitted actions from different tabs into one action. Dismissing a reviewed terminal receipt removes only the local reference and does not delete or cancel anything on the server.

Read an Inspect control receipt without resubmitting the action:

For a caller-owned identity, send cancel/resume with { "control": { "requestId": "your-stable-request-id" } }. Generate and retain that ID before sending the request. If the transport response is lost, reuse the same ID; the resident owner returns the existing receipt instead of creating a second intent. A retry of a known resume request can read its receipt after the run has become running; a new resume request against a running run still refuses. Malformed JSON or an invalid control identity returns 400 without submitting an action. This HTTP capability does not by itself persist a custom dashboard's in-memory state across navigation.

curl -G "$API_URL/_voltro/inspect/workflows/control" \
  -H "Authorization: Bearer $VOLTRO_INSPECT_TOKEN" \
  --data-urlencode "operation=$OPERATION" \
  --data-urlencode "workflowName=$WORKFLOW_NAME" \
  --data-urlencode "executionId=$EXECUTION_ID" \
  --data-urlencode "requestId=$REQUEST_ID"

Use the returned operation, snapshot.workflowName, snapshot.executionId, and non-null requestId. This read survives run-history removal and resolves retained ownership. Invalid input returns 400; unavailable or mismatched ownership returns 409. voltro workflows cancel|resume prints the outcome and receipt; accepted exits 0 without claiming completion, unknown exits 1. JSON output retains the full result. Do not repeat a mutation just to observe its status.

The resident control record has a nullable resumeJournal binding for the original suspension's request/reply identity. A stored binding is not a resume confirmation. Repeated dispatch must retain it rather than capture the latest reply; an earlier ambiguous attempt without a binding remains unknown. This binding is also used by automatic resident control recovery. Low-level hosts can call bindWorkflowResumeJournalInTransaction inside withAdmissionTransaction; its observation callback must read the journal under that resident lock, never reset or wake the engine.

workflowEngineLayer also supplies WorkflowResumeDispatcher from @voltro/workflow/cluster. Its dispatch({ requestId, transaction }) binds, records an attempt, resets the exact suspended reply and polls using that engine's captured journal connection. Supply the resident admission transaction as transaction; do not supply a primary or resident SQL layer to select the journal. The returned effect requires no additional engine services. An attempted result is not completion: only recorded entry confirms the receipt. Dev and serve use this capability after committing resident resume acceptance. A dispatch failure retains the receipt and logs its request ID; use ctx.workflows.readControl({ operation, workflowName, executionId, requestId }) to read it without submitting another command. Retry with the same request ID to retain the original suspension binding. Custom engines without this capability refuse resume before accepting a request.

Dev and serve run resident control recovery while workflows are registered. It scans declared physical roots and persisted namespace registrations, with 100 registrations per source page and 100 control rows per placement visit. One non-overlapping tick runs per second; larger fleets or histories therefore take multiple ticks. Per-placement control cursors carry across discovery passes so one large history cannot prevent another placement from being visited. Cursors are process-local scheduling state; accepted controls and their journal bindings are durable. A new worker starts its scan from stored records. Malformed controls and unavailable/moved placements produce warnings naming requestId and placementId, without logging payloads. Incomplete namespace discovery is reported separately and retried. Recovery never rebinds an old request to a newer suspension. A local SQLite integration test kills the producer after durable acceptance and either a rejected journal reset or a lost dispatch acknowledgement after the real reset and poll complete, then boots a successor over separate retained primary, resident, and journal files. Recovery confirms the new entry without resubmission and retains the original binding. Replaying the old request leaves a later suspension unchanged. In the lost-acknowledgement case, the original journal reply is already gone while the control is still pending. These tests do not prove the equivalent crash paths on other database dialects.

Inspect cancellation returns the facade's WorkflowControlResult: operation, kind, requestId, and snapshot, not a cancelled Boolean. Keep the receipt and distinguish accepted, confirmed, unknown, and already-terminal; read it through ctx.workflows.readControl using the returned identity. The internal sweep Boolean counts only confirmed cancellations. An unconfirmed result does not alter the run, emit a terminal event or close its children. Dev and serve persist the control request and its cause in the execution's resident store. Active bodies consume cancellation for their own generation; clean suspended cancellation settles under the resident entry lock. Receipt reads resolve retained execution placement, not the latest run generation. Follow-up delivery remains in progress; do not treat an accepted request as completed work.

Queued-start cancellation is still being integrated into the public control surface. Its resident transaction boundary records cancelled on start receipts only when it removes the still-pending input atomically; after takeover it must follow the submission instead. WorkflowStartNotExecuted.reason can represent this pre-execution cancellation. Failure-handler recovery preserves an explicitly cancelled arrival rather than enqueueing it again. Do not pass a queued receipt ID to the execution-ID-based cancel method.

The runtime facade can follow a queued admission for a blocking run() or start({ wait: true }) when a receipt resolver is supplied: it observes that receipt's accepted execution and preserves its original success or failure, without submitting the arriving payload again. Both CLI boot paths supply this resolver, and their admission gate applies deferring controls to blocking calls as well. A superseded, cancelled or rejected arrival fails explicitly rather than borrowing an unrelated run's result. For a caller-controlled timeout use start() followed by wait(handle, { timeoutMs }).

To learn what a recorded handle has become WITHOUT waiting — the question a reconciler asks over rows it wrote before a restart — use ctx.workflows.readStart(handle). It answers { kind: 'pending' }, { kind: 'accepted', executionId } or { kind: 'not-executed', reason, failureId, incumbentExecutionId } with the same reasons WorkflowStartNotExecuted carries, as a value rather than a throw. A queued handle is resolved through its durable receipt; a dropped or skipped handle answers from itself. Inside a mutation transaction it refuses, like wait: the receipt is provisional until commit.

The recorder binds a cluster run to its persisted request and restart identity (engineDelivery). A different delivery is refused before the body runs, including when a replacement is already running. An absent binding is not permission to adopt a cluster request. Preserve the run and engine journal for reconciliation; deleting history is not a recovery procedure. Authorized operator redrive and that reconciliation are still being integrated, so this check must not be read as a complete retry implementation.

Run filters can be saved as named views. The selected view and ad-hoc filters are mirrored into the URL query string, so a teammate can open the same filtered run list. The Workflows panel also includes dedicated Incoming and Flow tabs: Incoming groups runs whose source is incoming:<id> beside incoming-sourced domain events, while Flow groups queued/running/waiting work by workflow lane and start source.

#CLI

voltro workflows list --status running --tail 50
voltro workflows list --tag notes.summarise --format json
voltro workflows list --dead-letter                  # failed runs not yet discarded
voltro workflows list --statuses failed,cancelled --q orders --since 2026-08-01T00:00:00Z
voltro workflows list --id-prefix wfrun_01K          # matches the run id OR the execution id
voltro workflows stats --hours 24                    # bucketed activity sparkline + per-workflow totals
voltro workflows start notes.summarise --payload '{"noteId":"note_123"}'
voltro workflows show wfrun_01H...
voltro workflows retry wfrun_01H... --request-id retry-incident-001
voltro workflows cancel wfrun_01H...
voltro workflows suspend wfrun_01H...              # refuses: cooperative suspension required
voltro workflows resume wfrun_01H...
voltro workflows restart wfrun_01H... --request-id incident-1 --failed-steps                # re-drive a failed run from where it died
voltro workflows discard wfrun_01H...                # acknowledge a failed run
voltro workflows signal wfrun_01H... --name approval --payload '{"approved":true}'
voltro workflows update wfrun_01H... --name approve --payload '{"decision":true}'
voltro workflows children exec_01H...

The CLI discovers live API processes the same way as voltro logs and voltro traces, then calls inspect endpoints.

#Inspect endpoints

GET  /_voltro/inspect/workflows/runs
GET  /_voltro/inspect/workflows/runs?tag=notes.summarise&status=failed
GET  /_voltro/inspect/workflows/runs/:id/steps
GET  /_voltro/inspect/workflows/runs/:id/events
GET  /_voltro/inspect/workflows/events
GET  /_voltro/inspect/workflows/events/:id/deliveries
GET  /_voltro/inspect/workflows/children?parentExecutionId=exec_01H...

POST /_voltro/inspect/workflows/runs/:id/cancel
POST /_voltro/inspect/workflows/runs/:id/retry
POST /_voltro/inspect/workflows/runs/:id/suspend
POST /_voltro/inspect/workflows/runs/:id/resume
POST /_voltro/inspect/workflows/runs/:id/restart
GET  /_voltro/inspect/workflows/restart?runId=:id&requestId=:requestId
POST /_voltro/inspect/workflows/runs/:id/discard
POST /_voltro/inspect/workflows/runs/:id/signal
POST /_voltro/inspect/workflows/runs/:id/update

Signal body:

{
  "signalName": "approval",
  "payload": { "approved": true }
}

Update body:

{
  "updateName": "approve",
  "payload": { "decision": true },
  "timeoutMs": 30000
}

Retry normally uses the original payload. The inspect handler also supports payload override for operator tooling.

#SQL inspection

Recent failed runs:

SELECT id, tag, status, source, "errorTag", "errorMessage", "startedAt", "completedAt"
FROM "_voltro_workflow_runs"
WHERE status = 'failed'
ORDER BY "startedAt" DESC
LIMIT 50;

Slow steps:

SELECT "stepName", percentile_cont(0.95) WITHIN GROUP (ORDER BY "durationMs")
FROM "_voltro_workflow_run_steps"
WHERE status = 'succeeded'
  AND "startedAt" > now() - interval '24 hours'
GROUP BY "stepName"
ORDER BY 2 DESC
LIMIT 10;

Events for one run:

SELECT "eventType", payload, "occurredAt"
FROM "_voltro_workflow_run_events"
WHERE "runId" = 'wfrun_01H...'
ORDER BY "occurredAt" ASC;

#Inspecting from code

inspectWorkflow(idOrExecutionId, store) assembles one run into the same shape used by the dashboard detail view:

import { inspectWorkflow } from '@voltro/workflow'

const state = await inspectWorkflow(runId, ctx.store)

It groups step attempts by step name, so a step that failed twice and succeeded on attempt three reports attempts: 3 with the latest output/error surfaced.

#Testing workflows

Use makeWorkflowRunner from @voltro/testing to run a workflow in-process over the in-memory workflow engine and an in-memory recorder:

import { makeTestContext, makeWorkflowRunner } from '@voltro/testing'
import { SummariseNote } from '../workflows/notes.summarise.workflow'
import buildSummariseNote from '../workflows/notes.summarise.workflow'

test('notes.summarise retries the LLM step', async () => {
  const ctx = makeTestContext()
  const runner = makeWorkflowRunner({
    ctx,
    workflows: [{
      workflow: SummariseNote as never,
      execute: buildSummariseNote(ctx) as never,
    }],
  })

  const result = await runner.start('notes.summarise', { noteId: 'n_1' })
  expect(result.status).toBe('succeeded')
  expect(result.steps.find((s) => s.name === 'summarise-with-llm')?.attempts).toBe(3)
})

The result includes { status, output, error, steps, runId }. runner.inspect(runId) returns the assembled run later without re-running it.

#A start with the wrong payload

ctx.workflows.start(name, payload) validates the payload against the workflow's payload schema before the engine sees it. A mismatch throws a WorkflowPayloadError naming three things:

Workflow "sprint.report" was started with an invalid payload. missing required
field(s): teamId. { readonly teamId: string } └─ ["teamId"] is missing

The error also carries them structurally — workflowName, missingFields, _tag: 'WorkflowPayloadError' — so a handler can branch on it.

This matters most where nobody is watching. A cron whose payload drifted from the workflow's schema fails on every single firing; the schedule run is recorded failed in _voltro_schedule_runs and the log line is an error, not a warn, so voltro logs --level error and any alert wired to it see it. A nightly job that has been dead since a refactor is the exact failure this pair of behaviours exists to surface.

An unknown workflow name lists the registered ones, so a rename reads differently from a deletion:

Unknown workflow: sprint.reports. Registered workflows: billing.run, sprint.report.

Validation runs on the decoded value, so a Schema.Date payload accepts a Date — passing the already-domain-shaped value is correct and is not rejected.

#Anti-patterns

• Querying old table names. Use _voltro_workflow_runs, _voltro_workflow_run_steps, and _voltro_workflow_run_events.
• Filtering for status=dead. Current failed runs use status=failed.
• Logging giant step inputs/outputs. Step input/output is persisted for inspection. Store references to huge blobs instead of the blob itself.

#Recover from a recorded step

For a failed run, expand its recorded steps and choose Resume from a recorded step. Select an actual recorded step and confirm that its side effects and the effects of later steps can safely repeat. Earlier steps replay from the durable journal; the selected step and all later steps execute again, including previously successful steps. A suspended run uses the separate Resume action. Discarded runs cannot resume from a step.

The dashboard retains the restart request and observes its durable receipt. Engine entry confirms that execution resumed, not that the workflow completed successfully. Cloud and local dashboards expose the same action.

On narrow screens, version and step metrics scroll horizontally within their own panels. Focus either table region to scroll it with the keyboard; the surrounding page keeps its viewport width.

The browser keeps at most 2,048 unresolved restart identities per inspected app. When storage is full or unavailable, a new request is not sent. Existing identities are never evicted to make room; dismiss only requests with confirmed engine entry.

#Durable restart commands through Inspect

For a failed run on a durable workflow engine, submit POST /_voltro/inspect/workflows/runs/:id/restart with a request ID saved before sending and an explicit selection:

{"requestId":"operator-restart-1","selection":{"kind":"from-step","stepName":"refresh"}}

Use {"kind":"failed-steps"} to select the currently failed activities instead. The request uses Inspect write authorization. Reuse its request ID and selection when repeating the same intent. A different step selection requires a new intent. The host resolves the run's resident store and reserves capacity before resetting journal replies. The recovery worker revisits pending requests after capacity becomes available and after a host restart; a second dashboard click is not needed.

Read the same request with GET /_voltro/inspect/workflows/restart?runId=:id&requestId=operator-restart-1 using Inspect read authorization. pending means the request is stored, not that the engine has entered it. entered includes originalGeneration and enteredGeneration; it confirms recorder entry, not workflow success. Follow the run for its terminal result. not-found means no request was observed at that read; it does not prove an in-flight submission cannot still commit. A not-reserved submission result distinguishes admission refusal from unavailable journal evidence. After a lost transport response, read the saved target before submitting a new intent.

A withdrawn observation confirms that the saved identity is permanently fenced and cannot start a restart. It includes workflowName, the original selection, and withdrawnAt, but no executionId or engine generation. Custom readers must validate the workflow and selection as well as the run/request IDs. Delayed missing or refused replies do not supersede this terminal evidence; a contradictory reservation is an error. To permanently withdraw a never-reserved request, send POST /_voltro/inspect/workflows/runs/:id/withdraw-restart with its saved requestId and original selection, using Inspect write authorization. Reconcile an uncertain response through the same read-only receipt endpoint. A repeated withdrawal returns its original timestamp. If reservation won first, the response is the existing pending or entered receipt: withdrawal does not cancel that work, release its capacity, or dispatch it. Only a confirmed withdrawn response allows a replacement intent with a new request ID and selection. In the dashboard, Withdraw request submits the saved identity and selection. If the response is lost, Read request status reconciles it without creating a new request. After confirmation, Dismiss confirmed request clears the saved intent so you can explicitly choose and confirm a new selection. Already accepted work keeps running; it cannot be withdrawn with this action.

Unknown steps and ambiguous recorded ordering are refused. In particular, equal start timestamps at the rewind boundary are not enough to decide which earlier side effects may be repeated.

The run detail page and the expanded workflow-list rows save the selected step and request identity before sending. It retains that choice across reloads and lost replies. A pending receipt is automatically refreshed and needs no second submission. An uncertain response offers Resend this request, always with the original identity. Confirmed engine entry or permanent withdrawn evidence can be dismissed; a transient reservation refusal is not a durable rejection. Browser storage and Web Locks must be available. Confirming entry does not claim workflow success: follow the run's steps and final status. Step progress counts the newest recording per step; repeated successful activities do not inflate the completed-step count. Historical attempts remain visible in the timeline.

Custom WorkflowRunPage and WorkflowsPage providers supply useRestartRun and useRestartReceipt instead of useResumeFromStep, alongside a stable inspected-app controlScope. The mutation returns WorkflowRestartResult; the reader accepts WorkflowRestartTarget and returns DataSource<WorkflowRestartObservation> with automatic refresh and a manual refresh callback. Keep all receipt variants intact. To enable withdrawal, also supply useWithdrawRestart returning MutateSource<WorkflowRestartCommand, WorkflowRestartResult> and route it to the authorized withdraw-restart endpoint. Preserve its receipt unchanged; not-found is not a successful withdrawal. Read-only contexts hide the action. Both pages share the app-scoped request journal. The list keeps saved requests visible even when their runs leave the current filter. Live refreshes preserve an unsent step selection within the same app, replica scope and run filter. WorkflowStepResumeInput and WorkflowStepResumeResult are removed; use WorkflowRestartCommand and WorkflowRestartResult from @voltro/protocol.

#Entries without a recorded ending

After a run finishes, an activity or wait can still lack its own ending record, for example after its process was killed. The timeline labels it No completion recorded, stops its live animation, and leaves its outcome and duration unknown. The bar is bounded by the run’s completion time for display only; this is not a recorded activity end. Recorded success and failure remain unchanged. A child workflow has its own lifecycle and can remain running after its parent finishes.

#Local dashboard entry-point failure

When the selected entry process disappears, the local dashboard can read through another recently observed fleet member allowed by its proxy target policy. The registered app URL, replica selection and saved restart identity stay unchanged. This requires a successful fleet observation before the outage; remembered routes expire after two minutes unless refreshed. Reloading the page keeps using the same dashboard server's route cache.

Only reads retry after a connection failure. Authentication, authorization and other HTTP errors do not trigger an alternative request. A failed mutation is never replayed automatically: read its saved receipt to reconcile it. Subsequent explicit writes use the reachable entry selected by successful reads. Download and upload operations keep their existing target handling.

#Inspect and recover queued starts

In Workflows → Flow, use Check queued payloads to compare up to 500 queued starts against the schemas loaded by the responding process. Expand a result to compare the release and schema fingerprint recorded when queued with the current fingerprint. Valid, invalid, missing-workflow, and unavailable-schema results stay distinct; a full 500-entry window may omit further starts.

After correcting a workflow or payload schema, Retry queued start re-arms a dead letter under the usual admission rules. Discard queued start permanently removes that intent and every start collapsed into it; the dashboard names the intent and asks for explicit confirmation. If it was already processed, the response says nothing changed.

These are operator actions in both dashboards, using the app’s separate inspect write credential. An inspection credential does not bypass workflow admission rules or supply a customer-app user identity.

Cancellation requests made by shared operator/producer helpers retain their reason and details in the resident control receipt, including when the helper generates the request ID. Repeating an explicit request ID retains the original cause. Internal cancellation sweeps still require confirmed cancellation; durable acceptance alone does not satisfy that condition.

Workflow-enabled applications also declare _voltro_workflow_retry_requests through the shared framework schema assembly. Development boot and deployment migrations therefore use the same table definition. This stores internal retry intent preparation; its presence alone does not change the dashboard retry action or confirm that an execution has started.

The shared coordinated admission worker revisits pending retry preparations in each discovered physical workflow store. It retains deferred deadlines, reports preparation failures, and dispatches admitted submissions through the normal workflow facade. This internal recovery uses the existing worker lifecycle; it does not introduce a separate timer or a new dashboard retry action.

WorkflowRetryObservationSchema in @voltro/protocol describes a durable retry: queued, accepted, unconfirmed, not-started, or rejected. Acceptance has a different execution ID from the original run, but does not mean completion. An unconfirmed dispatch may have reached the engine. automaticRetryStopped means automatic dispatch retries stopped, not that execution never started.

The server SDK submits with ctx.workflows.retry(runId, { requestId, payloadOverride? }). Persist one request ID before each deliberate retry; reuse it after a lost reply. Dev and Serve record the intent in the original run's physical store before admission and dispatch. The full encoded workflow payload is validated, and the original run remains unchanged. A new deliberate intent requires a new ID. Read with ctx.workflows.readRetry(runId, requestId) without submitting work.

Inspect uses POST /_voltro/inspect/workflows/runs/:id/retry with JSON { "requestId": "your-persisted-intent-id" } and optional payloadOverride. It returns the same observation, not { newExecutionId }. Read with GET /_voltro/inspect/workflows/retry?runId=<run-id>&requestId=<request-id> using the Inspect read credential. Missing or blank IDs return 400; unavailable or inconsistent read evidence returns 409 without exposing storage errors. The read endpoint uses the normal shared-store observation envelope.

WorkflowRetryTargetSchema validates { runId, requestId } and WorkflowRetryCommandSchema adds the optional encoded payload override. Both are exported from @voltro/protocol; blank identities fail before owner resolution. The single-run CLI requires --request-id <persisted-intent-id>. Bulk retry requires a controls entry with run, workflow, original execution and request IDs for every previewed run. Its retries list preserves each observation; engine acceptance is never counted as completed execution in succeeded.

The local dashboard and both hosted dashboard adapters use this contract for single-run retries. Status refreshes read the existing request; they never submit another retry. Bulk dashboard actions persist one reference per previewed run before dispatch and keep each returned observation. The CLI uses a saved preview file to bind bulk retry commands to the same original runs and request IDs.

The shared workflow list and run-detail pages persist retry references before calling useRetryRun. Adapters accept WorkflowRetryCommand and must return WorkflowRetryObservation, without discarding it. Provide a stable controlScope for the inspected app and useRetryReceipt(target) as a read-only DataSource with refresh. Reopened pages recover the identity even if the original run is no longer in the visible list. The browser stores IDs and a payload fingerprint, not the workflow input or output. Storage failures prevent submission; a changed override cannot silently replace an unresolved intent. Only a conclusive observed outcome can be dismissed to allow a deliberate new retry.

Bulk dashboard retries use the same app-scoped journal as individual retries. The selected IDs must exactly match the preview targets. The journal validates the whole batch before writing, reuses retained identities, and refuses dispatch when storage fails or the 128-reference capacity would be exceeded. It removes only new entries from a failed partial write. Queued, accepted, unconfirmed, not-started and rejected results remain distinct; none counts as a completed workflow. Custom bulk panels provide prepareRetries(targets) and observeRetry(result); BulkRunResult.retries must survive every adapter without a success projection.

For CLI bulk retry, save a preview before submitting it:

voltro workflows replay-many --mode retry --status failed --limit 50 --plan-file retry-preview.json
voltro workflows replay-many --mode retry --status failed --limit 50 --plan-file retry-preview.json --commit
voltro workflows replay-many --mode retry --status failed --limit 50 --plan-file retry-preview.json --observe

Use the same selection options for all three commands. The file contains process URLs and recovery identities, never Inspect credentials or workflow payloads. Creation is exclusive, with private file permissions; an existing plan is never overwritten. Commit uses exactly the saved targets without running the filter again. A repeat commit reuses its request IDs. Replicas reporting the same original run share one request ID; conflicting replica identities are refused. --observe only reads receipt status and cannot be combined with --commit. A changed selection or process set requires a new preview rather than silently retargeting an old command. Keep the old file to recover any unresolved request. Queued and accepted outcomes are not completion. Unconfirmed, rejected, not-started, failed or malformed responses produce a nonzero command exit status.

The CLI also requires saved previews for bulk cancellation and redrive:

voltro workflows cancel-many --reason "Operator recovery" --status running --plan-file cancel-preview.json
voltro workflows cancel-many --reason "Operator recovery" --status running --plan-file cancel-preview.json --commit
voltro workflows cancel-many --reason "Operator recovery" --status running --plan-file cancel-preview.json --observe
voltro workflows replay-many --mode redrive --status failed --plan-file redrive-preview.json
voltro workflows replay-many --mode redrive --status failed --plan-file redrive-preview.json --commit
voltro workflows replay-many --mode redrive --status failed --plan-file redrive-preview.json --observe

Cancellation preserves the resident control receipt: acceptance is not confirmed cancellation. Its observer reads the saved control identities without cancelling again. Redrive binds the original run and execution from the preview and refuses an execution identity that changed. Each target keeps its saved request ID through the durable restart dispatcher, with selection: { kind: 'failed-steps' }. --observe reads those restart receipts without submitting again, even after the run no longer matches the original failure filter. pending means reserved; entered confirms engine entry, not workflow completion. A refusal or missing receipt remains unresolved and is never counted in succeeded.

The dashboard persists every target before sending a bulk restart. Reloads and lost replies keep those identities; receipt readers remain visible outside the original run filter. Custom BulkRunActionsPanel integrations must provide submitRestarts(targets, submit) backed by the restart journal. All provider adapters preserve BulkRunResult.restarts; they must not turn restart receipts into successful executions. Local, customer and admin dashboards share this panel and receipt contract.

Changing the inspected app keeps retry observations isolated: delayed preparation or a lost response from the previous app does not replace the current app’s references or error state. The previous app’s persisted intent remains available when you return; navigation does not submit it again.

The restart command and read-only receipt are available through Inspect in both voltro dev and voltro serve. A confirmed engine entry remains readable after a process restart; the unfinished execution resumes with the same generation. This does not guarantee exactly-once external effects inside an interrupted activity.

The run-detail page keeps an open restart form, its selected step and confirmation through transient step refreshes and connection errors. The last confirmed snapshot stays visible with a load-error message while receipt reads reconnect. Switching app, run or replica selection clears that retained snapshot; an explicit missing run or empty step result is not replaced by stale data.