Retries & failure handling

Per-step retries, failed workflow runs, retry controls, compensation, and timeouts.

Workflow durability and retry are related, but not identical. The engine checkpoints completed step({...}) activities so an interrupted run can resume without re-running successful steps. Transient failures still need an explicit retry policy around the step that can fail.

Per-step retries

Declare a retry: policy on step({...}) and the framework enforces it — it compiles the policy to an Effect Schedule and retries execute for you. No hand-written retry needed:

import { workflow, step } from '@voltro/workflow'
import { Schema } from 'effect'

class ProviderDown extends Schema.TaggedError<ProviderDown>()('ProviderDown', {
  message: Schema.String,
}) {}

const summary = yield* step({
  name: 'summarise-with-llm',
  input: { noteId },
  success: Schema.String,
  error: ProviderDown,
  retry: {
    maxAttempts: 5,
    strategy: 'exponential',
    baseDelay: '500 millis',
  },
  execute: callLlm(note.body),
})

Retries run inside the one step and are transparent to the durable engine — completed steps still checkpoint; the step's final outcome is recorded. retry: { maxAttempts: 5 } is already a good policy (exponential backoff, jittered).

The conditions you actually want

The useful retry question is rarely "how many times" — it's which failures, for how long, and how spread out. The policy covers all of it:

retry: {
  maxAttempts: 5,                          // total attempts including the first (default 3)
  strategy: 'exponential',                 // 'exponential' | 'fixed' | 'linear'
  baseDelay: '500 millis',
  maxDelay: '30 seconds',                  // ceiling so exponential growth can't run away
  factor: 2,                               // exponential growth factor
  jitter: true,                            // full jitter (default true) — anti-thundering-herd
  maxElapsed: '5 minutes',                 // a total time BUDGET — stop retrying after this
  retryableErrors: ['ProviderDown'],       // retry ONLY these typed errors; others fail fast
  respectRetryAfter: true,                 // honor a 429: retryAfter REPLACES this attempt's backoff
}

retryableErrors (or a retryable: (error) => boolean predicate) is the important one: retry the transient failures, fail fast on the permanent ones. A ValidationError should never be retried; a ProviderDown should.
maxElapsed is a deadline across all attempts, not another count — the right bound when "keep trying for up to 5 minutes" matters more than "try 8 times".
respectRetryAfter uses exactly the delay a provider asked for (a retryAfterMillis number, or retryAfter in seconds, on the thrown error) as the next delay, replacing the computed backoff for that attempt; falls back to backoff when there is no hint.

Retries here run inside the one step attempt, so the step is recorded as a SINGLE row in _voltro_workflow_run_steps with its final outcome — the individual in-step retries are not separate rows. respectRetryAfter uses the provider's delay AS the next delay (replacing the computed backoff for that attempt), falling back to backoff when the error carries no hint. For per-attempt rows in the dashboard, or a bespoke Schedule, use stepModule.retry(step({...}), schedule) instead. Do NOT set retry: on a step you ALSO wrap in stepModule.retry / Effect.retry: it would retry twice.

What to retry

Failure kind	Recommended handling
HTTP 5xx, connection reset, provider 429	Retry the step with backoff and jitter
Validation error, malformed payload	Fail the workflow
Permission error	Fail the workflow; fix caller or payload
Timeout	Convert to a typed transient error, then retry if safe
Cancellation interruption	Do not catch unless you are deliberately cleaning up

Keep retry windows close to the side effect. A flaky LLM call should retry inside the LLM step, not by restarting the whole workflow body.

Failed runs

When a workflow ultimately fails, Voltro records the run as failed. Query failed runs from any handler, action, or mutation with the typed SDK — no raw SQL, no knowledge of internal tables:

const failed = await ctx.workflows.listRuns({
  status: 'failed',
  workflowName: 'notes.summarise',
})
// failed: WorkflowRunSummary[] — id, workflowName, status, payload,
// errorTag, errorMessage, startedAt, completedAt, durationMs, …

listRuns(filter?) accepts { workflowName / tag, status, limit, offset } (all optional) and returns the most-recent runs first. The inspect dashboard surfaces the same data live, but the SDK is the primary path — it's typed and reactive-friendly.

There is no separate voltro_workflow_dlq table in the current runtime. Failed runs are the triage queue.

Retrying a failed run

Re-run a failed run by its id from a handler, action, or mutation:

for (const run of failed) {
  // Re-run against the original payload …
  await ctx.workflows.retry(run.id)
  // … or replay against a corrected input after fixing bad data:
  await ctx.workflows.retry(run.id, { payloadOverride: { ...run.payload, retries: 1 } })
}

retry(runId, options?) resolves the workflow by its tag, re-executes it, and returns { executionId } — the engine-assigned id of the fresh run. By default it uses the original run's recorded payload; pass payloadOverride to replay against a different input.

For ad-hoc ops, the same operation is available manually — the CLI (voltro workflows retry <runId>) and the dashboard's "Retry" button both delegate to the same runtime code:

voltro workflows retry wfrun_01H...

retry starts a fresh execution — a new execution id, an empty journal, every step runs again. That is the right tool for a short, idempotent job, or when the input itself was wrong (payloadOverride). For a long multi-step pipeline where re-doing steps 1…N‑1 is expensive or unsafe, you want the opposite: resume from where it failed.

Resume from where it failed — `suspendOnFailure`

Declare suspendOnFailure: true on a workflow and a failure no longer becomes a terminal failed run — it suspends with the durable journal intact:

export default workflow({
  name: 'billing.close-month',
  payload: { orgId: Schema.String },
  success: Schema.Void,
  error: Schema.Unknown,
  idempotencyKey: ({ orgId }) => `billing.close-month:${orgId}`,
  suspendOnFailure: true,          // a failure suspends (recoverable), not fails (terminal)
  execute: ({ orgId }) => Effect.gen(function* () {
    yield* step('snapshot-ledger', /* … */)   // completed steps are journaled
    yield* step('call-tax-provider', /* … */)  // ← a transient 503 here …
    yield* step('finalise-invoices', /* … */)
  }),
})

When call-tax-provider fails, the run goes to suspended (not failed), carrying the failure reason. Fix the cause, then resume — the engine replays snapshot-ledger from the journal (it does not re-run) and continues from the failed step:

voltro workflows resume wfrun_01H...      # re-drives from the failure point

await ctx.workflows.resume(run.id)         // the same, from a handler

Because a suspended-on-failure run is recoverable, not dead, it shows up under --status suspended, NOT in the dead-letter view (voltro workflows list --dead-letter, which is failed-and-unhandled). Choose per workflow: suspendOnFailure: true for a long pipeline where prior work must not be redone; the default (retry from scratch) for short idempotent jobs.

Restart a failed run with its journal

A run that already failed can restart without repeating its successful steps. Fix the cause first, then submit an immutable restart request. Both the SDK and CLI use the same resident reservation and engine-entry receipt as the dashboard.

voltro workflows restart wf_01H... --request-id incident-1 --failed-steps
voltro workflows restart wf_01H... --request-id incident-1 --observe

// Save this identity and selection before submitting. Reuse them after a lost reply.
const intent = { requestId: 'incident-1', selection: { kind: 'failed-steps' as const } }
const result = await ctx.workflows.restart(run.id, intent)
const observed = await ctx.workflows.readRestart(run.id, intent.requestId)

not-reserved reports a reservation refusal, not a permanent rejection of the identity. pending means the reservation is durable; it does not prove engine entry. entered confirms the next generation started, not that the run succeeded. Check the run and step records for completion. Unknown outcomes retain the same request ID and selection; status reads never submit another restart.

To abandon an unresolved request, call ctx.workflows.withdrawRestart(run.id, intent). A withdrawn receipt permanently prevents that identity from starting. If reservation already won, the existing pending or entered receipt is returned unchanged and the work continues. Confirm withdrawal before creating a different request. These methods operate outside mutation transactions: invoke them from an action or workflow after the application's mutation has committed.

Completed steps replay from the durable journal; only failed activities execute again. A missing journal refuses reservation. Admission limits, resident placement and the original execution remain authoritative in dev and production. Engine entry records a run-redriven event. Memory storage cannot provide durable restart.

Tool	Use when	Journal
`retry`	Input was wrong, or the job can safely run from the beginning	Fresh execution; every step runs again
`resume`	The run is suspended	Completed steps replay
`restart` with `failed-steps`	The run failed and earlier side effects must not repeat	Completed steps replay; failed steps run again
`restart` with `from-step`	Recovery must include an earlier step	Steps before the selected step replay; selected and later steps run again

Rewind further back — `restart --step`

restart with failed-steps continues from the step that failed. Sometimes that isn't the right place to restart: step 5 failed, but the real problem is that step 3 completed against stale external state, and re-running from the failure point would carry that bad result forward. restart --step rewinds to a step you choose:

voltro workflows restart wf_01H... --request-id incident-1 --step charge-card
voltro workflows restart wf_01H... --request-id incident-1 --observe

It resets the chosen step and every step after it — including steps that succeeded — so they re-execute, while the steps before your target replay from the journal unchanged. restart with failed-steps is the special case where your chosen step is exactly the one that failed. Like restart with failed-steps, it works under both voltro serve and voltro dev, refuses a run that isn't a not-yet-discarded failure, refuses an unknown step name, and declines cleanly on a run with no durable journal.

The CLI sends the same durable restart command as the dashboard. Save the request ID before submitting; reuse it after an uncertain response. --observe only reads its receipt and never submits another restart. pending means reserved; entered confirms engine entry, not completion. Select exactly one API with --process or --url when multiple APIs are available. Use --failed-steps instead of --step NAME to restart only failed activities.

Compensation

For saga-style workflows, model compensation explicitly with Effect.catchAll around the step that can fail after an earlier side effect:

yield* step({
  name: 'reserve-inventory',
  success: Schema.Void,
  execute: reserveInventory(orderId),
})

yield* step({
  name: 'charge-card',
  success: Schema.Void,
  execute: chargeCard(orderId).pipe(
    Effect.catchAll((error) =>
      releaseInventory(orderId).pipe(
        Effect.zipRight(Effect.fail(error)),
      ),
    ),
  ),
})

There is no defineSaga(...) helper today. Compensation is ordinary Effect code, checkpointed when you put it inside a step.

Timeouts

A hung external call is not a failure until it errors. Wrap it:

const response = yield* step({
  name: 'fetch-provider',
  success: ProviderResponse,
  error: ProviderDown,
  execute: fetchProvider(input).pipe(
    Effect.timeoutFail({
      duration: '30 seconds',
      onTimeout: () => new ProviderDown({ message: 'provider timed out' }),
    }),
  ),
})

If the operation is not idempotent, pass an idempotency key to the provider and also enforce a unique key in your own database.

Failure observability

Every run has three inspectable layers:

_voltro_workflow_runs — run status, payload, output, top-level error, subject, start source, timing, trace id, parent execution id, parent-close policy.
_voltro_workflow_run_steps — each step attempt, recorded input/output/error, retry metadata, duration.
_voltro_workflow_run_events — run lifecycle, timers, signals, suspend/resume/cancel events.

The dashboard and voltro workflows show <runId> read these same tables through inspect endpoints.

Anti-patterns

Catching errors and returning success. Operators lose the failed run.
Retrying non-idempotent side effects. Double charges and duplicate emails are workflow bugs, not retry bugs.
Declaring retry: on a step you ALSO wrap in Effect.retry / stepModule.retry. retry: is enforced now — the step would retry twice. Keep one.
Expecting a DLQ table. The current triage surface is failed runs plus retry controls.

Retries & failure handling

#Per-step retries

#The conditions you actually want

#What to retry

#Failed runs

#Retrying a failed run

#Resume from where it failed — suspendOnFailure

#Restart a failed run with its journal

#Rewind further back — restart --step

#Compensation

#Timeouts

#Failure observability

#Anti-patterns