Debugging & inspection

Voltro records workflow runs, step attempts, and lifecycle events into framework tables. The dashboard, CLI, and inspect endpoints all read the same data.

#What is recorded

Table	Contents
_voltro_workflow_start_contexts	One row per started execution id: starter subject, trace id, source, parent execution id, parent-close policy, and creation time. Used for cross-runner context handoff.
_voltro_workflow_runs	One row per run: id, tag, executionId, status, payload, workflowVersion, workflowPatches, output, error fields, subject, start source, timing, trace id, parent execution id, parent-close policy, plus the crash-loop bookkeeping runnerEnteredAt + reclaimCount.
_voltro_workflow_run_steps	One row per step attempt: step name, attempt number, recorded input, retry metadata, output or error, duration.
_voltro_workflow_run_events	Lifecycle events: run-started, run-succeeded, run-failed, run-suspended, run-resumed, run-redriven, run-cancelled, timer-set, timer-fired, signal-awaited, signal-sent, signal-received, update-requested, update-received, update-completed, update-failed, nondeterminism-suspected, run-crashlooped, run-stalled.
_voltro_workflow_events	Domain events emitted through ctx.events.publish(...): event id, name, payload, source, subject, trace id, occurred time.
_voltro_workflow_event_deliveries	One row per workflow trigger delivery: event id, trigger id, workflow name, execution id, idempotency key, status, error.

Run status is running, succeeded, failed, cancelled, or suspended.

#Turning the step rows off — workflows.recording: 'coarse'

Every step() costs two fire-and-forget writes to _voltro_workflow_run_steps (insert at start, update at settle) — measured, and off the step's critical path, but on a hot workflow with many steps they dominate the table's growth. 'coarse' skips both:

// app.config.ts
export default {
  workflows: { recording: 'coarse' },   // default: 'full'
}

Runtime override without a rebuild: VOLTRO_WORKFLOW_RECORDING=coarse (env wins over config; anything unrecognised falls back to 'full').

What 'coarse' does not touch, deliberately: run rows, run events (signals, timers, cancels, stall reports — everything the dead-letter view and the sweeps read), and the cluster engine's own durable journal — replay and redrive are unaffected. The cost is exactly the dashboard's step timeline: empty for runs recorded under 'coarse'. Flip it back when you need to see inside a run.

#Dead-letter: triaging failed runs

The framework does not retry a workflow on its own (see Retries — you compose retries inside execute: with Effect.retry). So a run that reaches failed is terminal: it is the dead-letter. voltro workflows list --dead-letter (or the Prometheus series voltro_workflow_runs_total{status="failed"} and the stale voltro_workflow_last_success_timestamp_seconds gauge — see Prometheus) is your queue of unhandled failures.

Triage a dead-letter run one of three ways:

• Retry it — voltro workflows retry <id> starts a fresh execution against the original (or an overridden) payload once you've fixed the cause. Every step runs again from scratch.
• Re-drive it — voltro workflows redrive <id> re-drives the run from its durable journal: completed steps replay, only the failed step(s) re-execute. Use this instead of retry for a long pipeline where redoing steps 1…N‑1 is expensive or unsafe. It is the after-the-fact counterpart to suspendOnFailure + resume and works under voltro serve too. Refuses a non-failed / already-discarded run; declines cleanly when there is no durable journal (the memory store). Records a run-redriven event.
• Discard it — voltro workflows discard <id> acknowledges the failure so it drops off the --dead-letter view. It is an ack, not a re-classification: the run stays status: 'failed' (the outcome + audit trail survive) and gains a discardedAt timestamp. --status failed still lists it, marked discarded; only --dead-letter hides it. Discarding a non-failed run is refused, and discarding is idempotent.

#Stuck runs and crash loops

Three failure modes are detected rather than left for whoever opens the dashboard.

A run that stops moving. A staleness sweep reports live runs (running or suspended) that have made no progress for longer than the threshold — default 30 minutes — writing a run-stalled event carrying idleMs, the reason (awaiting-signal, suspended, no-progress) and the last progress instant. It changes no run state; it is a signal, not an intervention. The classic catch is a run parked on a signal nobody ever sends.

Two things it deliberately stays quiet about, because otherwise the signal is worthless:

• A run inside a durable sleep / sleepUntil whose wake instant is still in the future. That run is waiting by design, and a seven-day timer is not a stall.
• A run already reported since its last progress. A stall is reported once and again only after the run moves and stalls afresh.

Configure it in app.config.ts — every field optional:

export default defineApiApp({
  workflows: {
    staleness: {
      // Set this above your slowest single STEP, not above your longest RUN:
      // a run waiting on a durable timer is already excluded. Default 30 min.
      stallAfterMs: 30 * 60_000,
      runPage: 200,                       // live runs examined per tick, oldest first
      onStalled: async (run) => {
        await page(`${run.tag} ${run.runId} idle ${run.idleMs}ms (${run.reason})`)
      },
    },
  },
  // How often the sweep looks. `VOLTRO_STALENESS_SWEEP_MS` overrides it.
  scheduling: { stalenessSweepMs: 5 * 60_000 },
})

onStalled must not throw — a rejection is collected and reported, so one bad pager integration cannot stop detection for every other workflow.

The sweep runs on a coordinated schedule, so one firing per interval fleet-wide rather than one per replica. Unlike every other framework background task it never stops ticking when idle: those disarm because an arrival wakes them, and a run going stale writes nothing there is to wake on. That makes the cadence an unconditional cost, which is why it defaults to five minutes rather than one second — on a thirty-minute threshold that is 12 coordination rows an hour instead of 3 600.

voltro doctor runs the same detection once, for the moment you are standing in front of a deployment asking whether anything is wedged:

✗  stuck runs: 2 run(s) have made no progress
   invoices.settle  wr_01J…  idle 4h  awaiting-signal
   report.nightly   wr_01J…  idle 2h  no-progress
   Nothing was changed — this is a SUSPICION, not a verdict. `awaiting-signal`
   usually means the sender never came.

It is the one doctor rule that reads your DATABASE rather than your source, so run it where the app's DB env vars are set; anywhere else it prints a named skip rather than a clean tick. It records nothing and calls no onStalled, so running it neither pages anyone nor suppresses the background sweep's next real report.

A run that keeps killing its runner. A step that crashes the process — OOM, a native crash — cannot be caught as an error: the shard lease expires, a surviving replica claims it, and executes the same payload. Without a ceiling that rotates around the fleet forever. Voltro counts consecutive runner deaths on the run row (runnerEnteredAt is set on every body entry and cleared on every clean exit; reclaimCount counts entries that found the previous marker still set). At VOLTRO_WORKFLOW_MAX_RECLAIMS (default 3) the run is parked as suspended with errorTag: 'WorkflowCrashLooped' and a run-crashlooped event, and the body is not entered again. The counter is consecutive — any clean re-entry resets it — so a long-lived healthy run is never parked for a crash it had months ago, and an operator resume gives it a fresh budget.

A run replaying against edited code. See Versioning for the nondeterminism-suspected event.

#Dashboard

When voltro dev is running, the Workflows panel lists recent runs, their status, start source, timing, payload, output/error, step attempts, and events. The run detail view is the fastest way to answer:

• Which step is currently running?
• Was the run started by an RPC, ctx.workflows, a schedule, an incoming webhook, or inspect tooling?
• Is this run a child, and will it be cancelled, terminated, or abandoned when the parent closes?
• Which attempt failed?
• What input did the step receive?
• What output or typed error did it produce?
• Did a timer or signal fire?

The panel also exposes run controls for users with the right capability: cancel, retry, suspend, resume, redrive (re-drive a failed run from its journal), discard (acknowledge a failed run), send signal, and send a tracked update.

Run filters can be saved as named views. The selected view and ad-hoc filters are mirrored into the URL query string, so a teammate can open the same filtered run list. The Workflows panel also includes dedicated Incoming and Flow tabs: Incoming groups runs whose source is incoming:<id> beside incoming-sourced domain events, while Flow groups queued/running/waiting work by workflow lane and start source.

#CLI

voltro workflows list --status running --tail 50
voltro workflows list --tag notes.summarise --format json
voltro workflows list --dead-letter                  # failed runs not yet discarded
voltro workflows list --statuses failed,cancelled --q orders --since 2026-08-01T00:00:00Z
voltro workflows list --id-prefix wfrun_01K          # matches the run id OR the execution id
voltro workflows stats --hours 24                    # bucketed activity sparkline + per-workflow totals
voltro workflows start notes.summarise --payload '{"noteId":"note_123"}'
voltro workflows show wfrun_01H...
voltro workflows retry wfrun_01H...
voltro workflows cancel wfrun_01H...
voltro workflows suspend wfrun_01H...
voltro workflows resume wfrun_01H...
voltro workflows redrive wfrun_01H...                # re-drive a failed run from where it died
voltro workflows discard wfrun_01H...                # acknowledge a failed run
voltro workflows signal wfrun_01H... --name approval --payload '{"approved":true}'
voltro workflows update wfrun_01H... --name approve --payload '{"decision":true}'
voltro workflows children exec_01H...

The CLI discovers live API processes the same way as voltro logs and voltro traces, then calls inspect endpoints.

#Inspect endpoints

GET  /_voltro/inspect/workflows/runs
GET  /_voltro/inspect/workflows/runs?tag=notes.summarise&status=failed
GET  /_voltro/inspect/workflows/runs/:id/steps
GET  /_voltro/inspect/workflows/runs/:id/events
GET  /_voltro/inspect/workflows/events
GET  /_voltro/inspect/workflows/events/:id/deliveries
GET  /_voltro/inspect/workflows/children?parentExecutionId=exec_01H...

POST /_voltro/inspect/workflows/runs/:id/cancel
POST /_voltro/inspect/workflows/runs/:id/retry
POST /_voltro/inspect/workflows/runs/:id/suspend
POST /_voltro/inspect/workflows/runs/:id/resume
POST /_voltro/inspect/workflows/runs/:id/redrive
POST /_voltro/inspect/workflows/runs/:id/discard
POST /_voltro/inspect/workflows/runs/:id/signal
POST /_voltro/inspect/workflows/runs/:id/update

Signal body:

{
  "signalName": "approval",
  "payload": { "approved": true }
}

Update body:

{
  "updateName": "approve",
  "payload": { "decision": true },
  "timeoutMs": 30000
}

Retry normally uses the original payload. The inspect handler also supports payload override for operator tooling.

#SQL inspection

Recent failed runs:

SELECT id, tag, status, source, "errorTag", "errorMessage", "startedAt", "completedAt"
FROM "_voltro_workflow_runs"
WHERE status = 'failed'
ORDER BY "startedAt" DESC
LIMIT 50;

Slow steps:

SELECT "stepName", percentile_cont(0.95) WITHIN GROUP (ORDER BY "durationMs")
FROM "_voltro_workflow_run_steps"
WHERE status = 'succeeded'
  AND "startedAt" > now() - interval '24 hours'
GROUP BY "stepName"
ORDER BY 2 DESC
LIMIT 10;

Events for one run:

SELECT "eventType", payload, "occurredAt"
FROM "_voltro_workflow_run_events"
WHERE "runId" = 'wfrun_01H...'
ORDER BY "occurredAt" ASC;

#Inspecting from code

inspectWorkflow(idOrExecutionId, store) assembles one run into the same shape used by the dashboard detail view:

import { inspectWorkflow } from '@voltro/workflow'

const state = await inspectWorkflow(runId, ctx.store)

It groups step attempts by step name, so a step that failed twice and succeeded on attempt three reports attempts: 3 with the latest output/error surfaced.

#Testing workflows

Use makeWorkflowRunner from @voltro/testing to run a workflow in-process over the in-memory workflow engine and an in-memory recorder:

import { makeTestContext, makeWorkflowRunner } from '@voltro/testing'
import { SummariseNote } from '../workflows/notes.summarise.workflow'
import buildSummariseNote from '../workflows/notes.summarise.workflow'

test('notes.summarise retries the LLM step', async () => {
  const ctx = makeTestContext()
  const runner = makeWorkflowRunner({
    ctx,
    workflows: [{
      workflow: SummariseNote as never,
      execute: buildSummariseNote(ctx) as never,
    }],
  })

  const result = await runner.start('notes.summarise', { noteId: 'n_1' })
  expect(result.status).toBe('succeeded')
  expect(result.steps.find((s) => s.name === 'summarise-with-llm')?.attempts).toBe(3)
})

The result includes { status, output, error, steps, runId }. runner.inspect(runId) returns the assembled run later without re-running it.

#A start with the wrong payload

ctx.workflows.start(name, payload) validates the payload against the workflow's payload schema before the engine sees it. A mismatch throws a WorkflowPayloadError naming three things:

Workflow "sprint.report" was started with an invalid payload. missing required
field(s): teamId. { readonly teamId: string } └─ ["teamId"] is missing

The error also carries them structurally — workflowName, missingFields, _tag: 'WorkflowPayloadError' — so a handler can branch on it.

This matters most where nobody is watching. A cron whose payload drifted from the workflow's schema fails on every single firing; the schedule run is recorded failed in _voltro_schedule_runs and the log line is an error, not a warn, so voltro logs --level error and any alert wired to it see it. A nightly job that has been dead since a refactor is the exact failure this pair of behaviours exists to surface.

An unknown workflow name lists the registered ones, so a rename reads differently from a deletion:

Unknown workflow: sprint.reports. Registered workflows: billing.run, sprint.report.

Validation runs on the decoded value, so a Schema.Date payload accepts a Date — passing the already-domain-shaped value is correct and is not rejected.

#Anti-patterns

• Querying old table names. Use _voltro_workflow_runs, _voltro_workflow_run_steps, and _voltro_workflow_run_events.
• Filtering for status=dead. Current failed runs use status=failed.
• Expecting replay-from-step APIs. Current operator retry starts a new execution from the workflow payload; completed steps are not selectively replayed through a public API.
• Logging giant step inputs/outputs. Step input/output is persisted for inspection. Store references to huge blobs instead of the blob itself.