Retries & failure handling

Per-step retries, failed workflow runs, retry controls, compensation, and timeouts.

Workflow durability and retry are related, but not identical. The engine checkpoints completed step({...}) activities so an interrupted run can resume without re-running successful steps. Transient failures still need an explicit retry policy around the step that can fail.

Per-step retries

Declare a retry: policy on step({...}) and the framework enforces it — it compiles the policy to an Effect Schedule and retries execute for you. No hand-written retry needed:

import { workflow, step } from '@voltro/workflow'
import { Schema } from 'effect'

class ProviderDown extends Schema.TaggedError<ProviderDown>()('ProviderDown', {
  message: Schema.String,
}) {}

const summary = yield* step({
  name: 'summarise-with-llm',
  input: { noteId },
  success: Schema.String,
  error: ProviderDown,
  retry: {
    maxAttempts: 5,
    strategy: 'exponential',
    baseDelay: '500 millis',
  },
  execute: callLlm(note.body),
})

Retries run inside the one step and are transparent to the durable engine — completed steps still checkpoint; the step's final outcome is recorded. retry: { maxAttempts: 5 } is already a good policy (exponential backoff, jittered).

The conditions you actually want

The useful retry question is rarely "how many times" — it's which failures, for how long, and how spread out. The policy covers all of it:

retry: {
  maxAttempts: 5,                          // total attempts including the first (default 3)
  strategy: 'exponential',                 // 'exponential' | 'fixed' | 'linear'
  baseDelay: '500 millis',
  maxDelay: '30 seconds',                  // ceiling so exponential growth can't run away
  factor: 2,                               // exponential growth factor
  jitter: true,                            // full jitter (default true) — anti-thundering-herd
  maxElapsed: '5 minutes',                 // a total time BUDGET — stop retrying after this
  retryableErrors: ['ProviderDown'],       // retry ONLY these typed errors; others fail fast
  respectRetryAfter: true,                 // honor a 429: retryAfter REPLACES this attempt's backoff
}

retryableErrors (or a retryable: (error) => boolean predicate) is the important one: retry the transient failures, fail fast on the permanent ones. A ValidationError should never be retried; a ProviderDown should.
maxElapsed is a deadline across all attempts, not another count — the right bound when "keep trying for up to 5 minutes" matters more than "try 8 times".
respectRetryAfter uses exactly the delay a provider asked for (a retryAfterMillis number, or retryAfter in seconds, on the thrown error) as the next delay, replacing the computed backoff for that attempt; falls back to backoff when there is no hint.

Retries here run inside the one step attempt, so the step is recorded as a SINGLE row in _voltro_workflow_run_steps with its final outcome — the individual in-step retries are not separate rows. respectRetryAfter uses the provider's delay AS the next delay (replacing the computed backoff for that attempt), falling back to backoff when the error carries no hint. For per-attempt rows in the dashboard, or a bespoke Schedule, use stepModule.retry(step({...}), schedule) instead. Do NOT set retry: on a step you ALSO wrap in stepModule.retry / Effect.retry: it would retry twice.

What to retry

Failure kind	Recommended handling
HTTP 5xx, connection reset, provider 429	Retry the step with backoff and jitter
Validation error, malformed payload	Fail the workflow
Permission error	Fail the workflow; fix caller or payload
Timeout	Convert to a typed transient error, then retry if safe
Cancel/suspend by operator	Do not catch unless you are deliberately cleaning up

Keep retry windows close to the side effect. A flaky LLM call should retry inside the LLM step, not by restarting the whole workflow body.

Failed runs

When a workflow ultimately fails, Voltro records the run as failed. Query failed runs from any handler, action, or mutation with the typed SDK — no raw SQL, no knowledge of internal tables:

const failed = await ctx.workflows.listRuns({
  status: 'failed',
  workflowName: 'notes.summarise',
})
// failed: WorkflowRunSummary[] — id, workflowName, status, payload,
// errorTag, errorMessage, startedAt, completedAt, durationMs, …

listRuns(filter?) accepts { workflowName / tag, status, limit, offset } (all optional) and returns the most-recent runs first. The inspect dashboard surfaces the same data live, but the SDK is the primary path — it's typed and reactive-friendly.

There is no separate voltro_workflow_dlq table in the current runtime. Failed runs are the triage queue.

Retrying a failed run

Re-run a failed run by its id from a handler, action, or mutation:

for (const run of failed) {
  // Re-run against the original payload …
  await ctx.workflows.retry(run.id)
  // … or replay against a corrected input after fixing bad data:
  await ctx.workflows.retry(run.id, { payloadOverride: { ...run.payload, retries: 1 } })
}

retry(runId, options?) resolves the workflow by its tag, re-executes it, and returns { executionId } — the engine-assigned id of the fresh run. By default it uses the original run's recorded payload; pass payloadOverride to replay against a different input.

For ad-hoc ops, the same operation is available manually — the CLI (voltro workflows retry <runId>) and the dashboard's "Retry" button both delegate to the same runtime code:

voltro workflows retry wfrun_01H...

retry starts a fresh execution — a new execution id, an empty journal, every step runs again. That is the right tool for a short, idempotent job, or when the input itself was wrong (payloadOverride). For a long multi-step pipeline where re-doing steps 1…N‑1 is expensive or unsafe, you want the opposite: resume from where it failed.

Resume from where it failed — `suspendOnFailure`

Declare suspendOnFailure: true on a workflow and a failure no longer becomes a terminal failed run — it suspends with the durable journal intact:

export default workflow({
  name: 'billing.close-month',
  payload: { orgId: Schema.String },
  success: Schema.Void,
  error: Schema.Unknown,
  idempotencyKey: ({ orgId }) => `billing.close-month:${orgId}`,
  suspendOnFailure: true,          // a failure suspends (recoverable), not fails (terminal)
  execute: ({ orgId }) => Effect.gen(function* () {
    yield* step('snapshot-ledger', /* … */)   // completed steps are journaled
    yield* step('call-tax-provider', /* … */)  // ← a transient 503 here …
    yield* step('finalise-invoices', /* … */)
  }),
})

When call-tax-provider fails, the run goes to suspended (not failed), carrying the failure reason. Fix the cause, then resume — the engine replays snapshot-ledger from the journal (it does not re-run) and continues from the failed step:

voltro workflows resume wfrun_01H...      # re-drives from the failure point

await ctx.workflows.resume(run.id)         // the same, from a handler

Because a suspended-on-failure run is recoverable, not dead, it shows up under --status suspended, NOT in the dead-letter view (voltro workflows list --dead-letter, which is failed-and-unhandled). Choose per workflow: suspendOnFailure: true for a long pipeline where prior work must not be redone; the default (retry from scratch) for short idempotent jobs.

Re-drive a failed run — `redrive`

suspendOnFailure is a decision you make before the run. What if a run already failed — it's sitting in the dead-letter view — and you still want to continue it from where it died, not re-run it from scratch? That is redrive:

voltro workflows redrive wf_01H...   # re-drive a FAILED run from the step it died on

await ctx.workflows.redrive(run.id)  // the same, from a handler; addressed by run id

redrive re-drives the run from its durable journal: every completed step replays from the journal (it does not re-run), and only the failed step(s) re-execute. It is the after-the-fact counterpart to suspendOnFailure + resume — same "continue from the failure point" outcome, but for a run that already went terminal without being marked suspend-on-failure. Fix the downstream cause first, then redrive.

The three recovery tools, and when each applies:

Tool	Use when	Journal
`retry`	The input was wrong, or the job is short + idempotent	Fresh execution, empty journal — every step runs again
`resume`	The run is `suspended` (you set `suspendOnFailure: true`, or it awaits a signal)	Continues from the failure/suspend point — completed steps replay
`redrive`	The run is `failed` (dead-letter) and re-running earlier steps is expensive/unsafe	Continues from the failure point — completed steps replay, failed steps re-run
`resume-from-step`	The run is `failed`, but the failure point is not the right recovery point — an earlier step ran on stale/wrong state	Rewinds to a chosen step — steps before it replay, that step + everything after re-run

redrive refuses a run that isn't a not-yet-discarded failure (use resume for a suspended run, retry for a fresh execution), and it needs a durable journal — on the memory store there is nothing to re-drive, so it declines cleanly (redriven: false with a reason) rather than pretending. It works under voltro serve as well as voltro dev, because dead-letter recovery happens in production. A re-drive records a run-redriven event on the run's timeline.

Rewind further back — `resume-from-step`

redrive continues from the step that failed. Sometimes that isn't the right place to restart: step 5 failed, but the real problem is that step 3 completed against stale external state, and re-running from the failure point would carry that bad result forward. resume-from-step rewinds to a step you choose:

voltro workflows resume-from-step wf_01H... charge-card   # rewind to 'charge-card' and re-run from there

It resets the chosen step and every step after it — including steps that succeeded — so they re-execute, while the steps before your target replay from the journal unchanged. redrive is the special case where your chosen step is exactly the one that failed. Like redrive, it works under both voltro serve and voltro dev, refuses a run that isn't a not-yet-discarded failure, refuses an unknown step name, and declines cleanly on a run with no durable journal.

Compensation

For saga-style workflows, model compensation explicitly with Effect.catchAll around the step that can fail after an earlier side effect:

yield* step({
  name: 'reserve-inventory',
  success: Schema.Void,
  execute: reserveInventory(orderId),
})

yield* step({
  name: 'charge-card',
  success: Schema.Void,
  execute: chargeCard(orderId).pipe(
    Effect.catchAll((error) =>
      releaseInventory(orderId).pipe(
        Effect.zipRight(Effect.fail(error)),
      ),
    ),
  ),
})

There is no defineSaga(...) helper today. Compensation is ordinary Effect code, checkpointed when you put it inside a step.

Timeouts

A hung external call is not a failure until it errors. Wrap it:

const response = yield* step({
  name: 'fetch-provider',
  success: ProviderResponse,
  error: ProviderDown,
  execute: fetchProvider(input).pipe(
    Effect.timeoutFail({
      duration: '30 seconds',
      onTimeout: () => new ProviderDown({ message: 'provider timed out' }),
    }),
  ),
})

If the operation is not idempotent, pass an idempotency key to the provider and also enforce a unique key in your own database.

Failure observability

Every run has three inspectable layers:

_voltro_workflow_runs — run status, payload, output, top-level error, subject, start source, timing, trace id, parent execution id, parent-close policy.
_voltro_workflow_run_steps — each step attempt, recorded input/output/error, retry metadata, duration.
_voltro_workflow_run_events — run lifecycle, timers, signals, suspend/resume/cancel events.

The dashboard and voltro workflows show <runId> read these same tables through inspect endpoints.

Anti-patterns

Catching errors and returning success. Operators lose the failed run.
Retrying non-idempotent side effects. Double charges and duplicate emails are workflow bugs, not retry bugs.
Declaring retry: on a step you ALSO wrap in Effect.retry / stepModule.retry. retry: is enforced now — the step would retry twice. Keep one.
Expecting a DLQ table. The current triage surface is failed runs plus retry controls.

Retries & failure handling

#Per-step retries

#The conditions you actually want

#What to retry

#Failed runs

#Retrying a failed run

#Resume from where it failed — suspendOnFailure

#Re-drive a failed run — redrive

#Rewind further back — resume-from-step

#Compensation

#Timeouts

#Failure observability

#Anti-patterns