Production hardening

The defaults are tuned for voltro dev, where fast iteration wins. Before you point real traffic at voltro serve, walk this checklist — most items are a single env var or a one-line config, but skipping them is how a service leaks across tenants, forges sessions, or falls over under load.

#1. Session secret (REQUIRED)

In production (NODE_ENV=production) voltro serve refuses to boot unless VOLTRO_SESSION_SECRET is set to a real value. It rejects three cases explicitly:

• a missing value,
• the built-in public dev fallback (it is committed in the framework source),
• any value shorter than 32 chars — which catches placeholders like changeme.

Generate a real one:

voltro secret generate session

Store it in your secrets manager and inject it as an env var — never commit it. Why the hard fail: a forgotten secret would otherwise sign session cookies with a key that is public in the framework source, letting anyone forge any session.

Rotation is zero-downtime. Set the new value as VOLTRO_SESSION_SECRET and move the old one to VOLTRO_SESSION_SECRET_PREVIOUS:

VOLTRO_SESSION_SECRET=<new>            # signs new cookies
VOLTRO_SESSION_SECRET_PREVIOUS=<old>   # still verifies live cookies

Keep _PREVIOUS in place for one session-TTL window, then drop it. Existing cookies verify against previous until they naturally expire — no live session is invalidated.

If you embed the framework's middleware yourself rather than booting through voltro serve, there is no boot gate to catch a missing secret. In that case a presented voltro:session cookie that cannot be verified now logs, once per process and at error level, that no credential-expiry bound is being imposed — so realtime subscriptions on that connection will not be cut off when the session expires. It is a diagnostic, not a refusal: a stale voltro:session cookie from another app on the same host is a normal thing for a browser to carry, and failing the request would turn that into a denial of service.

#2. Kubernetes health probes

voltro serve exposes two unauthenticated endpoints, both handled before any rate-limit interceptor:

• GET /internal/liveness — always 200 ok. The process is up.
• GET /internal/readiness — 200 ready only after full boot, 503 before.

Readiness also runs a DB ping (SELECT 1) on SQL stores. So a pod whose connection pool has died reports 503 and is pulled from the Service endpoints — instead of staying in rotation and erroring every request.

# k8s Deployment — probes
livenessProbe:
  httpGet:
    path: /internal/liveness
    port: 4000
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /internal/readiness
    port: 4000
  periodSeconds: 5
  failureThreshold: 3

#Configuring them — health in app.config.ts

The paths and the answers are declarable, for both app types:

export default {
  type: 'web' as const,
  health: {
    // Default `/internal`; move it when your app owns a route there.
    path: '/api/health',
    liveness:  () => true,
    readiness: async () => catalogLoaded(),
  },
}

readiness is the one worth writing, because "ready" genuinely differs. For an api it is the dependency ping above. For a screen on a wall it is the opposite — keep showing the last rendered frame while the api wobbles, and count as ready precisely then. Only the app knows.

liveness should stay cheap and dependency-free. A liveness probe that consults a database restarts pods when the database is slow, which is the one thing that makes an outage worse.

The probes are answered before routing on every boot path, so no route and no guard can claim them. That is not a detail: on a web app voltro dev used to have no probe surface, and /internal/* fell into the page router — an SPA shell answered 200 with HTML, a page whose loader redirects answered 303, an auth guard answered 303 to /login. A kubelet reads all three as PASS, which is the one property a health probe must not have. Worse, one of those pages had a loader that calls the api, so the web pod's readiness hung on the api's reachability, once per probe interval.

Run voltro serve in serving pods — not voltro dev. voltro dev is the local-iteration supervisor: file-watch, respawn, codegen, and a boot-time auto-migrate that introspects the whole schema. It does not expose the probes above and binds its port only after that boot work finishes — so a TCP probe can't tell "still booting" from "dead", and a large-schema migrate can blow past a fixed startup window and get a healthy pod killed. Production pods run voltro serve.

#3. Schema migration at deploy (don't migrate in the serving pod)

voltro serve does not auto-migrate — schema changes go through an explicit deploy step, never on the serving pod's boot. The full contract (per-env fingerprint check, refuse-to-boot on mismatch, apply timing relative to the image swap) is Prod pipeline; the k8s wiring is here.

Run the apply as a pre-deploy Job (or initContainer) that holds the migration credentials and executes voltro db apply. It re-diffs the deployed code's declared schema against the live DB and applies the resulting plan — so it's a clean no-op on an already-current DB, which makes it safe to re-run and to run in every pod of a stateless deploy. A bare apply refuses under NODE_ENV=production by design (auto-apply on prod is not allowed), so the Job runs with NODE_ENV unset or staging; the serving pods keep NODE_ENV=production:

# Helm pre-install/pre-upgrade Job — runs to completion BEFORE the new pods roll.
apiVersion: batch/v1
kind: Job
metadata:
  name: myapp-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: myapp:{{ .Values.image.tag }}
          command: ["voltro", "db", "apply", "--note", "release {{ .Values.image.tag }}"]
          env:
            - { name: NODE_ENV, value: "staging" }   # bare apply is refused under "production"
            # …plus the same DB_* connection env as the serving pods

# Serving Deployment — auto-migrate OFF; the Job already applied the schema.
env:
  - name: VOLTRO_AUTO_MIGRATE
    value: "0"

For the reviewed-exact-diff flow — generate voltro db plan --json > plan.json in CI, apply it with voltro db apply --plan plan.json — see Prod pipeline. That variant IS allowed under NODE_ENV=production (it refuses unless both fingerprints still match the reviewed plan). A Job runs once per release vs an initContainer's once per replica, so it's the better fit for a multi-replica rollout.

#Expand/contract — the migration that's safe while old pods still serve

The pre-deploy Job applies the schema before the new pods roll — so during a rolling update, old pods (old code) run against the already-migrated schema for the length of the rollout. A migration that DROPS or RENAMES a column, NARROWS a type, or ADDS a constraint breaks those old pods mid-rollout: they 500 reading a column that's gone, or their writes are rejected by the new constraint. The migration "succeeded" and the app served errors anyway.

voltro db plan flags these — the operations unsafe under a rolling deploy are listed with a ⚠, separately from the data-safety (lossy / blocked) gate, since the two are orthogonal: a dropped() column is blessed for data loss and still breaks an old reader.

⚠ 1 operation(s) UNSAFE under a rolling deploy
  (old + new instances overlap → old code breaks against the new schema):
    • drop-column: old instances still SELECT/INSERT "orders"."legacy_total"; …
      → stop reading the column in code and deploy that first; drop it in a LATER deploy

Two ways to handle it:

1. No overlap window — a maintenance-window or scale-to-zero deploy (old pods gone before new ones start) has no simultaneous old code, so a single-step drop/rename is fine. The advisory doesn't apply; ignore it.
2. Zero-downtime rollout — split the breaking change into two releases, each of which keeps both code versions working (expand/contract):
   • Expand (release N): add the new shape — a nullable column, a new table, a backfill, dual-write from the new code. Old code ignores it.
   • Cut over: the new code reads/writes the new shape; deploy it.
   • Contract (release N+1): once no pod runs the old code, drop/rename/narrow the now-unused old shape. This step's db plan is clean.

Renaming orders.total → orders.amount under zero downtime is: add amount (expand) → backfill + dual-write → cut reads over → drop total (contract) — three releases, never one, so no in-flight pod ever references a column that isn't there.

Make it a hard gate if you always rolling-deploy — VOLTRO_ROLLING_DEPLOY=1. The db plan ⚠ is advisory by default, because a maintenance-window / scale-to-zero deploy has no overlap window and the framework can't tell which you run. If your pipeline is always a rolling update, set VOLTRO_ROLLING_DEPLOY=1 in the migration Job's env: voltro db apply then refuses (exit 2) a plan containing a rolling-unsafe operation instead of warning, so an un-split breaking change fails the deploy rather than breaking pods at runtime. Override a specific apply with --force. Unset (the default) leaves today's advisory behaviour untouched.

#If you DO run voltro dev in a cluster (dev / staging only)

voltro dev binds a small boot-health surface on its own port so a probe can watch the slow boot (codegen + migrate) it otherwise couldn't see:

• GET /internal/liveness → 200 ok from the moment the process is up, through the whole boot. Point startupProbe and livenessProbe here so migration time counts as alive, not dead.
• GET /internal/readiness → 503 until the app port is serving, then 200.
• GET /internal/startup → 200 JSON { phase, ready, tablesTotal, elapsedMs } (phase is booting → migrating → ready, or error with a message) — for humans and dashboards watching progress.

The port defaults to app port + 1; override with VOLTRO_DEV_HEALTH_PORT (set 0 to disable). A generous startupProbe.failureThreshold on /internal/liveness then gives a large-schema first migrate minutes instead of a fixed TCP window:

# dev/staging pod running `voltro dev` — probe the boot-health port (4001)
startupProbe:
  httpGet: { path: /internal/liveness, port: 4001 }
  periodSeconds: 10
  failureThreshold: 60        # up to 10 minutes for a first cold migrate
livenessProbe:
  httpGet: { path: /internal/liveness, port: 4001 }
readinessProbe:
  httpGet: { path: /internal/readiness, port: 4001 }

Even a local voltro dev boots faster on reboots now: the auto-migrate skips the full introspect when the declared schema is unchanged (a fingerprint check — one indexed query instead of scanning every table). Force a full re-introspect with VOLTRO_MIGRATE_FORCE=1.

#4. Request limits & DoS

POST /rpc (the buffered JSON endpoint the SSR loaders use) is capped, and the cap is enforced as bytes arrive:

VOLTRO_MAX_RPC_BODY_BYTES=8388608   # default 8 MiB
VOLTRO_MAX_BODY_BYTES=8388608       # default 8 MiB

A declared Content-Length over the cap is refused up front, so an honest client gets its 413 without uploading anything — a courtesy, not the enforcement, since a Transfer-Encoding: chunked body declares no length at all. The byte counter is the enforcement: it stops accumulating the moment the running total crosses the limit, drains the rest of the upload rather than dropping the connection, and answers 413. Both shapes therefore end in the same status code, and the refusal is logged under the voltro:security scope with the cap and the byte count at which the server stopped reading.

Scope: this guards the /rpc JSON path only. File uploads ride separate storage routes with their own limits.maxBytes, and WebSocket frames are capped by the ws library default (100 MiB).

Put per-IP rate limiting and the primary body-size cap at the ingress — that is the correct layer: it holds per-IP state and works across replicas, which an in-process limit can't.

# nginx ingress — annotations on the Ingress resource
nginx.ingress.kubernetes.io/proxy-body-size: "8m"
nginx.ingress.kubernetes.io/limit-rps: "20"

For app-level throttling (per-subject / per-tenant, e.g. an expensive action), use @voltro/plugin-ratelimit and the plugin onHttpRequest interceptor seam. It complements the ingress cap — it does not replace it.

There is no rate limit in the box. The body cap above is the only request guard the runtime applies by default; @voltro/plugin-ratelimit is opt-in, so an app that has not installed and configured it has no per-IP, per-API-key or per-tenant cap on /rpc at all. The one on-by-default throttle anywhere in the framework is plugin-auth's brute-force lockout, which covers sign-in credential attempts and nothing else. Treat the ingress limit as required, not as belt-and-braces.

#Per-IP limits need a trusted proxy

The address the runtime rate-limits, geo-blocks and audits by is socket.remoteAddress — not x-forwarded-for, which any client can write. Behind an ingress that means every request counts against the proxy's address — and every sessions.ipAddress row records the proxy — until you declare the hop:

// app.config.ts
export default {
  security: {
    trustedProxies: ['private'],   // or: ['loopback'] · ['10.0.0.0/8'] · ['2'] · ['*']
  },
}

The same setting decides whether x-forwarded-proto is believed, which is what lets the runtime emit HSTS behind a TLS-terminating load balancer. Override it on a running deployment with VOLTRO_TRUSTED_PROXIES=private (comma-separated).

#Cross-site protection and its allowlist

Every state-changing request — POST /rpc, the /ws upgrade, every REST route projected from a publicApi: mutation, everything in apiConfig.restRoutes, and POST /v1/api-keys — refuses a browser request whose Origin is neither the Host it was sent to nor an allowlisted origin. A split web/api deployment must declare its web origin or the browser gets a 403 on every mutation, REST write and socket:

// app.config.ts
export default {
  security: {
    allowedOrigins: ['https://app.example.com'],
  },
}

The environment override is VOLTRO_ALLOWED_ORIGINS (comma-separated). Server-to-server callers (SSR loaders, mobile SDKs, other services) send no Origin and are unaffected. Full detail in Security.

#5. Multi-tenant isolation

Tables carrying the tenant() mixin are auto-scoped to the request's tenant — nothing more to do there. The gap is the anonymous request that matches no auth strategy: by default it resolves to a tenant-less anonymous Subject that can read any non-tenant() table across the DB.

For an app where every request must be tenant-scoped (no anonymous public data), close that door:

// app.config.ts
export default {
  // ...
  auth: {
    anonymousTenantRequired: true,
  },
}

A request that matches no auth strategy and sends no x-tenant header is then rejected with Unauthenticated instead of resolving to a tenant-less Subject. It applies identically under voltro dev and voltro serve.

Do NOT enable this if the app serves legitimate anonymous public data — public read endpoints, reference tables — it would reject those callers. In that case, rely on putting tenant() on every private table instead.

#6. Observability (wire it or fly blind)

By default nothing is exported. Wire it before you need it.

Traces + metrics ship to any OTLP collector (Tempo / Jaeger / Honeycomb / Grafana Agent) from one env var:

OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
OTEL_SERVICE_NAME=my-api

Structured logs as JSON:

VOLTRO_LOG_FORMAT=json
VOLTRO_LOG_LEVEL=info

In json mode (the default off a TTY, so a pod gets it without configuration) EVERY line the framework emits is a parseable record — the boot banner, the app surface, voltro db apply's plan summary, refusal detail, and every subsystem logger (schedule, broadcast, workflow, flow-control). A hand-formatted table or a [tag]-prefixed adapter line between JSON records is a bug, not a style: log collectors show it as unparsed noise. On a TTY the same surfaces render as the human-readable banners and tables.

Error reporting — add sentryPlugin() from @voltro/plugin-sentry. It stays inert until SENTRY_DSN is set, and reported errors correlate to the request traceId:

import { sentryPlugin } from '@voltro/plugin-sentry'

export default {
  // ...
  plugins: [sentryPlugin()],
}

Durable in-DB traces are off in production by default — traces belong in your OTLP backend, not your OLTP database.

#7. Graceful shutdown

On SIGTERM / SIGINT, voltro serve shuts down cleanly (exit 0), in this order: it stops accepting new connections and lets in-flight requests finish against a fully-alive app (bounded — the request drain gets 60% of the shutdown grace, so the teardown behind it always fits inside the deadline), then stops schedulers, detaches subscribers / reactions / aggregates, drains the analytics sink and mirror, ends any remaining WebSocket, and closes the SQL connection pool last (waiting for in-flight transactions). Verified against a real serve under signal: a request in flight when SIGTERM lands completes with a full response before the process exits — no preStop hook, no orchestrator. A bare voltro serve under docker compose drains itself.

The transactional-outbox worker is part of that sequence: its poll timer and its change subscription are released, and a delivery already in flight is awaited, before the pool closes. An outbox row that was still pending is not lost — it is durable, and the next process's first pass picks it up, which is one of the three reasons that poll exists.

What an orchestrator still adds: routing. The app finishes every request it has accepted — but a request that arrives after SIGTERM is refused (the listener closes immediately, on purpose), and only the layer that routes traffic can stop sending it. Failing readiness from inside the process doesn't help: the listener is already closed. Under k8s, close that window with a preStop hook:

spec:
  # Must exceed the preStop sleep + the app's own shutdown.
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            # k8s removes the pod from the Service endpoints AND runs this
            # BEFORE sending SIGTERM. The sleep holds the pod up (listener open,
            # finishing in-flight) while endpoint removal propagates — so no new
            # request lands on a pod that's about to close.
            command: ["sh", "-c", "sleep 5"]

Without the preStop hook, a rolling update refuses the small window of requests that still route to a terminating pod before k8s finishes removing it from the Service endpoints — refused with a connection error, not truncated mid-response (everything already accepted completes either way). With it, that window is served too. terminationGracePeriodSeconds must be larger than the sleep plus the app's own teardown, or k8s SIGKILLs mid-drain. Environments with no endpoint removal at all — docker compose above all — need nothing: there is no routing layer to lag behind the shutdown, so the built-in drain is the whole story.

Bound the app's own teardown with VOLTRO_SHUTDOWN_GRACE_MS. After SIGTERM, the runtime runs its finalizers (pool close, plugin onDeactivate, analytics flush, trace persist) and then exits — but installing the signal handler removes node's default kill, so a finalizer that never completes (a pool drain against a database that is already gone, a wedged onDeactivate) would otherwise hang the process forever. A hard deadline caps that: teardown gets until the deadline, then the process exits regardless. It defaults to 10s; set VOLTRO_SHUTDOWN_GRACE_MS (clamped to 1s–5min) to sit JUST UNDER your terminationGracePeriodSeconds minus the preStop sleep — so the app drains and exits cleanly on its own before k8s SIGKILLs it mid-drain:

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      env:
        # preStop sleep (5s) + app teardown (≤22s) < 30s grace, with headroom.
        - name: VOLTRO_SHUTDOWN_GRACE_MS
          value: "22000"

Live WebSockets are ENDED promptly at shutdown — the drain never lets an open socket hold the process to the deadline — and the web client's supervisor treats any close as a reconnect signal, so an open dashboard re-attaches to a healthy replica across a rolling deploy without a page reload.

#8. Multiple replicas

Cache and KV default to in-process (per-replica). For a shared backend across replicas:

CACHE_BACKEND=redis
KV_BACKEND=redis

Cross-replica reactivity is the subtle one: a write on one pod never surfaces on another pod's open subscriptions unless the replicas share a change bus. Use:

• postgres — LISTEN/NOTIFY (built in),
• mariadb — binlog CDC (built in),
• other dialects — @voltro/plugin-broadcast (Redis).

Schedules and aggregates auto-coordinate via an advisory lock on SQL stores — no extra config to keep them from double-firing across replicas.

#The prerequisites you learn at the SECOND pod

Every item below is invisible on one replica and breaks on two. They are collected here because an operator reported each of them separately, each found the same way: the first pod proved the configuration worked.

The connection pool multiplies, the database limit does not.

DB_MAX_CONNECTIONS=10       # per replica — the fleet opens up to this × replicaCount

The framework opens ONE pool per process. At 4 replicas a pool of 10 is 40 connections against a database that still allows whatever it allowed before you scaled. An operator's second pod died on Connection timed out for exactly this. voltro serve now prints the number and the arithmetic at boot:

db pool: max=10 per replica (DB_MAX_CONNECTIONS) + 1 = 11 × 4 replicas = up to 44 connections.
Check that against your database's limit. PLUS 1 outside the pool (CDC LISTEN consumer) —
those do not come out of the pool budget, they come out of the DATABASE's.

Set REPLICA_COUNT from your deployment (Helm: {{ .Values.replicaCount }}) and the line does the multiplication for you; without it the line still names the formula.

voltro dev prints it too, when the environment says it is not a laptop. A bare voltro dev stays silent — one process, no replicas, nothing to multiply. But voltro dev is a supported way to RUN an app, and a deployment that uses it needs this line as much as any other. So it prints whenever REPLICA_COUNT, DB_MAX_CONNECTIONS / PG_MAX_CONNECTIONS, or DB_REPLICA_URLS is set — each of which means somebody has already decided something about the number.

Some connections are not in the pool, and the count is per process. A connection that speaks a long-lived protocol cannot be returned to a pool, so the driver opens a standalone one. There are four such places and a full deployment can hold several at once:

Process	Connection	When
api voltro serve / dev	CDC LISTEN consumer	postgres, changeStrategy: 'cdc' (the default)
api voltro serve / dev	binlog CDC reader	mysql / mariadb, changeStrategy: 'cdc'
web voltro start	ISR invalidator LISTEN	any page declares cacheInvalidatesOn
web voltro start	postgres ISR cache client	SSR_CACHE=postgres

SQL Server is the one CDC dialect that costs nothing here: Change Tracking is read with ordinary queries through the pool, so its out-of-pool count is a verified zero rather than an omission.

Two of the four are not a LISTEN at all — the binlog reader speaks the replication protocol, and the ISR cache client is an ordinary client — which is why counting LISTEN rows in pg_stat_activity undercounts. Each process prints its own number in the boot line above — including No connections outside the pool in this process when there are none, so "counted, zero" is distinguishable from "not counted".

And a rolling update needs the surge pod's connections too. A budget sized for replicaCount is exactly full at steady state and short during every deploy: maxSurge adds a pod that opens a full pool of its own. If that pod cannot connect it never becomes ready, so the rollout does not complete and the cluster stays at the higher pod count — the deploy cannot free itself. Size for (replicaCount + maxSurge) × (DB_MAX_CONNECTIONS + out-of-pool).

POD_IP is each replica's identity, not only a workflow setting.

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Without it every replica registers under the same host, so they are one runner as far as the cluster is concerned. The boot warning for it fires only on SQL cluster storage, so a deployment that has not adopted durable workflows yet gets no signal at all — inject it as a matter of course.

Derive the broadcast namespace from something that cannot be forgotten.

- name: VOLTRO_BROADCAST_NAMESPACE
  value: {{ .Release.Namespace }}

Staging and production of the same app share a name, code and fingerprint, so the auto-derived namespace does NOT separate them — only this variable does. An operator's own guidance, and better than ours was: a value taken from the release namespace cannot be left out of one environment's config file, because there is no file to forget.

#The framework's own background pollers

Two framework tasks ride the coordinated scheduler and write a row into _voltro_schedule_claims on every tick they win: the workflow admission drainer and the offloaded-inference dispatcher.

Where a peer replica's write is visible here, they do not poll at all. Each runs one tick at startup — not optional; it is what finds work a previous process left behind — and then stops until something arrives. The wake comes from the change events their queue tables already emit, which is the same mechanism the rest of the framework's reactivity runs on. Measured against a real Postgres on a deployment that never uses either queue: 2 claim rows in five minutes, one per task, both written at boot.

That "where" is the whole condition, and it is satisfied by Postgres LISTEN/NOTIFY or by a broadcast broker (Redis/NATS — which a multi-replica deployment already runs for cross-replica reactivity). Without either, a remote replica's enqueue produces no local event, so stopping would mean sleeping through it. There the tasks back off to a ceiling instead:

VOLTRO_POLL_CEILING_MS=30000   # how long an arrival can wait when NOTHING woke the task

Nothing is lost in that case either — the replica that enqueued always sees its own write inline and drains it itself. What the ceiling covers is the narrower case of a crashed writer's lease being reclaimed by someone else.

A number worth knowing before you tune anything: on a two-replica deployment that had never enqueued into either queue, these two tasks accounted for 99.3 % of the claim ledger — 2 506 rows an hour against 18 from the app's own eight schedules. A fixed interval has no way to learn a queue is empty. That is what changed; the ceiling is the fallback, not the fix.

#Tuning the cadence

The intervals are declarable, with defaults most apps never change:

// app.config.ts
export default {
  scheduling: {
    admissionDrainMs: 1000,   // workflow admission drainer
    inferenceTickMs:  250,    // offloaded-inference dispatcher
    cancelSweepMs:    2000,   // cancelOn sweep
    pollCeilingMs:    30000,  // idle ceiling, where nothing can wake a task
  },
}

Each has a matching env var — VOLTRO_ADMISSION_DRAIN_MS, VOLTRO_INFERENCE_TICK_MS, VOLTRO_CANCEL_SWEEP_MS, VOLTRO_POLL_CEILING_MS — which overrides the config field, the same way VOLTRO_TENANT_ISOLATION overrides tenancy.isolation. The config is what a project declares; the env var is what an operator changes on a running deployment without a rebuild.

Lowering an interval does not make anything more responsive: an arrival already wakes the task at once. It only bounds the case where nothing announced the work. A value of 0 or a non-number is ignored rather than honoured — a zero interval would turn an idle task into a spin.

#Workflow failover across replicas

On a SQL store (postgres / mysql / mariadb / mssql), durable workflows survive a replica crash: completed step({...}) activities are checkpointed in the cluster journal, so when a replica dies mid-run, a surviving replica takes over the run and continues it from the last completed step — it replays the completed steps rather than re-running them. (On sqlite the engine is single-process — durable within one replica, no cross-replica failover.) Two requirements:

• Inject POD_IP (K8s downward API, fieldRef: status.podIP) or set VOLTRO_WORKFLOW_RUNNER_HOST. This is each replica's cluster identity — without a distinct value, every replica registers as the same runner and they stop distributing shards (and cross-pod resume degrades). The boot logs a warning if it sees localhost with SQL storage.
• Make step side effects idempotent. Failover is at-least-once at the step boundary: a crash between a side effect and its journal write re-runs that step. A step's own retry: does not change this — it's about the step you're inside, not the replica handoff.

Is polling the bottleneck? No — reclaim is a lease, not a poll. A crashed replica keeps its shards until its heartbeat goes stale; only then can a survivor claim them. So takeover latency is bounded by the lease TTL (~35s by default), not by any message-poll interval, and a push mechanism (LISTEN/NOTIFY) does not move it. Two knobs tune it:

VOLTRO_WORKFLOW_FAILOVER_LEASE=15       # seconds a dead replica's work stays locked (default 35)
VOLTRO_WORKFLOW_FAILOVER_HEARTBEAT=5    # lease-refresh cadence (default 10; keep ≈ lease/3)

Lower the lease for faster failover, at the cost of false-positive reclaims: if a healthy replica is paused longer than the lease by a GC pause or a DB-latency spike, another replica may briefly also claim its shards. Keep the heartbeat around a third of the lease so one slow refresh doesn't trip a reclaim. For crash detection that doesn't depend on the timeout at all, pair it with a K8s liveness probe so a dead pod is removed promptly.

(A separate concern is new-message pickup: a workflow triggered on the replica that owns its shard starts immediately, but one owned by ANOTHER replica is otherwise picked up on that replica's next storage poll — up to 10s. If you run a broadcast broker (Redis/NATS — which a multi-replica deployment already does for cross-replica reactivity), this is automatic and near-instant: a trigger pushes a "wake" over the bus and the shard owner re-polls at once, on any SQL dialect. Without a broker, the change stream does the same job wherever remote changes reach the spine — a Postgres-only multi-replica fleet (LISTEN/NOTIFY CDC) is the common case: a remote replica's signal, start context, or run transition arrives as a change event and triggers an immediate, coalesced re-poll, so signal/step latency stops being poll-bounded there too. Only with neither a broker nor CDC does the poll interval remain the bound — tune VOLTRO_WORKFLOW_POLL_INTERVAL=2 then. Unrelated to the failover path above.)

#Checklist

• VOLTRO_SESSION_SECRET set from voltro secret generate session, in a secrets manager
• Rotation window uses VOLTRO_SESSION_SECRET_PREVIOUS
• Liveness / readiness probes point at /internal/liveness + /internal/readiness
• Serving pods run voltro serve (not voltro dev), with VOLTRO_AUTO_MIGRATE=0
• Schema applied by a pre-deploy Job / initContainer (voltro db apply), not in the serving pod
• VOLTRO_MAX_RPC_BODY_BYTES + VOLTRO_MAX_BODY_BYTES (plugin routes/webhooks) sane; ingress caps body size + per-IP rate
• VOLTRO_TRUSTED_PROXIES set if you run behind an ingress AND rate-limit per IP
• VOLTRO_ALLOWED_ORIGINS set if the web app is on a different origin than the api
• Security headers reviewed (VOLTRO_SECURITY_HEADERS, VOLTRO_CSP); HSTS reaching the browser over https
• Every *.webhook.tsx declares its verification, and each signature-verified one has its VOLTRO_WEBHOOK_SECRET_<ID>
• auth.anonymousTenantRequired: true (unless the app serves anonymous public data)
• OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME pointed at your collector
• VOLTRO_LOG_FORMAT=json; sentryPlugin() + SENTRY_DSN for errors
• terminationGracePeriodSeconds generous for graceful drain; VOLTRO_SHUTDOWN_GRACE_MS set just under it (minus the preStop sleep)
• CACHE_BACKEND / KV_BACKEND + a cross-replica change bus when running >1 replica
• DB_MAX_CONNECTIONS set so pool × replicaCount fits your database's limit — read the db pool: boot line before raising replicaCount
• POD_IP injected via the downward API on EVERY multi-replica deployment, not only for durable workflows
• VOLTRO_BROADCAST_NAMESPACE derived from the release namespace — staging and production do not separate themselves
• For durable workflows on >1 replica: SQL store + POD_IP injected; tune VOLTRO_WORKFLOW_FAILOVER_LEASE if a 35s takeover is too slow; step side effects idempotent