Production hardening

The defaults are tuned for voltro dev, where fast iteration wins. Before you point real traffic at voltro serve, walk this checklist — most items are a single env var or a one-line config, but skipping them is how a service leaks across tenants, forges sessions, or falls over under load.

#1. Session secret (REQUIRED)

In production (NODE_ENV=production) voltro serve refuses to boot unless VOLTRO_SESSION_SECRET is set to a real value. It rejects three cases explicitly:

• a missing value,
• the built-in public dev fallback (it is committed in the framework source),
• any value shorter than 32 chars — which catches placeholders like changeme.

Generate a real one:

voltro secret generate session

Store it in your secrets manager and inject it as an env var — never commit it. Why the hard fail: a forgotten secret would otherwise sign session cookies with a key that is public in the framework source, letting anyone forge any session.

Rotation is zero-downtime. Set the new value as VOLTRO_SESSION_SECRET and move the old one to VOLTRO_SESSION_SECRET_PREVIOUS:

VOLTRO_SESSION_SECRET=<new>            # signs new cookies
VOLTRO_SESSION_SECRET_PREVIOUS=<old>   # still verifies live cookies

Keep _PREVIOUS in place for one session-TTL window, then drop it. Existing cookies verify against previous until they naturally expire — no live session is invalidated.

If you embed the framework's middleware yourself rather than booting through voltro serve, there is no boot gate to catch a missing secret. In that case a presented voltro:session cookie that cannot be verified now logs, once per process and at error level, that no credential-expiry bound is being imposed — so realtime subscriptions on that connection will not be cut off when the session expires. It is a diagnostic, not a refusal: a stale voltro:session cookie from another app on the same host is a normal thing for a browser to carry, and failing the request would turn that into a denial of service.

#2. Kubernetes health probes

Every serving command — voltro serve, voltro start and voltro dev alike — binds its app port as the first thing it does once it knows the port, and answers three unauthenticated probes there from that moment, before any route, guard or rate-limit interceptor:

• GET /internal/liveness — 200 ok from the first moment of the process, through the whole boot. The process is up.
• GET /internal/readiness — 503 not-ready while booting, 200 ready once the app serves, 503 again from the moment SIGTERM lands.
• GET /internal/startup — JSON { phase, ready, elapsedMs } (plus tablesTotal during a migrate and error with the message on a failed one); 503 until the boot finished, 200 from then on. phase is booting → migrating → ready, then draining, or error.

Readiness also runs a DB ping (SELECT 1) on SQL stores, under voltro serve and voltro dev alike. The ping goes through the app's own pool. The readiness answer has an independent ten-second limit: a stalled ping returns 503, including on MySQL/MariaDB, whose driver does not time-limit waiting for a busy pool. Repeated probes share one outstanding ping until it settles, so their timeouts do not add more waiting database queries. Readiness recovers after the pool can answer again. Give the readiness probe a timeoutSeconds that covers that wait, or a probe timeout fails it first, which is the same verdict reached sooner. An app-declared health.readiness replaces this store check. The startup probe deliberately does not consult the ping: it asks whether the boot finished, which is a different question.

While the process is still booting — a slow database connect, a fingerprint check, a cold migrate — every other request and every WebSocket upgrade on the app port answers 503 with Retry-After: 1. Once ready, the same socket serves the app, and the probe paths keep answering before routing. There is one server and one port, so a probe path means the same thing on every command, and a kubelet can always tell "still booting" from "dead": a refused connection now means the process is gone.

The probes are answered by the process that is booting, so a probe waits while the boot holds the thread. The longest such stretch in a large app is loading its modules, and that step hands the thread back every 25 ms — measured on a generated app of 1798 modules booting voltro serve from the production bundle the slowest liveness answer was 89 ms with the machine at load 14 on 12 cores, where the same boot had held one probe for 630–860 ms before. A single module that does heavy work at import time, or a boot step of your own, can still delay a probe for as long as it runs, so give the probes a timeoutSeconds above Kubernetes' default of 1 s, as the manifest below does.

# k8s Deployment — probes
startupProbe:
  httpGet:
    path: /internal/startup
    port: 4000
  periodSeconds: 5
  timeoutSeconds: 5           # the booting process answers; leave it room
  failureThreshold: 60        # up to 5 minutes for a slow boot — liveness waits for it
livenessProbe:
  httpGet:
    path: /internal/liveness
    port: 4000
  periodSeconds: 10
  timeoutSeconds: 5
readinessProbe:
  httpGet:
    path: /internal/readiness
    port: 4000
  periodSeconds: 5
  failureThreshold: 3

#Configuring them — health in app.config.ts

The paths and the answers are declarable, for both app types:

export default {
  type: 'web' as const,
  health: {
    // Default `/internal`; move it when your app owns a route there.
    path: '/api/health',
    liveness:  () => true,
    readiness: async () => catalogLoaded(),
  },
}

readiness is the one worth writing, because "ready" genuinely differs. For an api it is the dependency ping above. For a screen on a wall it is the opposite — keep showing the last rendered frame while the api wobbles, and count as ready precisely then. Only the app knows.

liveness should stay cheap and dependency-free. A liveness probe that consults a database restarts pods when the database is slow, which is the one thing that makes an outage worse.

The probes are answered before routing on every boot path, so no route and no guard can claim them. That is not a detail: on a web app voltro dev used to have no probe surface, and /internal/* fell into the page router — an SPA shell answered 200 with HTML, a page whose loader redirects answered 303, an auth guard answered 303 to /login. A kubelet reads all three as PASS, which is the one property a health probe must not have. Worse, one of those pages had a loader that calls the api, so the web pod's readiness hung on the api's reachability, once per probe interval.

Run voltro serve in serving pods — not voltro dev. voltro dev is the local-iteration supervisor: file-watch, respawn, codegen, and a boot-time auto-migrate that introspects the whole schema. It answers the same probes on the same port, so a kubelet can watch that boot — but a large-schema migrate still takes as long as it takes, and a supervisor restart on every file change is not a serving process. Production pods run voltro serve.

#3. Schema migration at deploy (don't migrate in the serving pod)

voltro serve does not auto-migrate — schema changes go through an explicit deploy step, never on the serving pod's boot. The full contract (per-env fingerprint check, refuse-to-boot on mismatch, apply timing relative to the image swap) is Prod pipeline; the k8s wiring is here.

Run the apply as a pre-deploy Job (or initContainer) that holds the migration credentials and executes voltro db apply. It re-diffs the deployed code's declared schema against the live DB and applies the resulting plan — so it's a clean no-op on an already-current DB, which makes it safe to re-run and to run in every pod of a stateless deploy. A bare apply refuses under NODE_ENV=production by design (auto-apply on prod is not allowed), so the Job runs with NODE_ENV unset or staging; the serving pods keep NODE_ENV=production:

# Helm pre-install/pre-upgrade Job — runs to completion BEFORE the new pods roll.
apiVersion: batch/v1
kind: Job
metadata:
  name: myapp-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: myapp:{{ .Values.image.tag }}
          command: ["voltro", "db", "apply", "--note", "release {{ .Values.image.tag }}"]
          env:
            - { name: NODE_ENV, value: "staging" }   # bare apply is refused under "production"
            # …plus the same DB_* connection env as the serving pods

# Serving Deployment — auto-migrate OFF; the Job already applied the schema.
env:
  - name: VOLTRO_AUTO_MIGRATE
    value: "0"

For the reviewed-exact-diff flow — generate voltro db plan --json > plan.json in CI, apply it with voltro db apply --plan plan.json — see Prod pipeline. That variant IS allowed under NODE_ENV=production (it refuses unless both fingerprints still match the reviewed plan). A Job runs once per release vs an initContainer's once per replica, so it's the better fit for a multi-replica rollout.

#Expand/contract — the migration that's safe while old pods still serve

The pre-deploy Job applies the schema before the new pods roll — so during a rolling update, old pods (old code) run against the already-migrated schema for the length of the rollout. A migration that DROPS or RENAMES a column, NARROWS a type, or ADDS a constraint breaks those old pods mid-rollout: they 500 reading a column that's gone, or their writes are rejected by the new constraint. The migration "succeeded" and the app served errors anyway.

voltro db plan flags these — the operations unsafe under a rolling deploy are listed with a ⚠, separately from the data-safety (lossy / blocked) gate, since the two are orthogonal: a dropped() column is blessed for data loss and still breaks an old reader.

⚠ 1 operation(s) UNSAFE under a rolling deploy
  (old + new instances overlap → old code breaks against the new schema):
    • drop-column: instances on the previous release still SELECT/INSERT "orders"."legacy_total"; …
      → stop reading the column in code and deploy that first; drop it in a LATER deploy
    Premise: an instance on the previous release keeps serving these tables while the schema lands. Tables this plan creates are exempt. No such instance (first deployment, scale-to-zero, maintenance window) → nothing here applies.

Tables the same plan creates are never flagged. Every verdict rests on one premise — an instance on the previous release still reads or writes the table — and a table that this plan CREATEs has no such instance, whatever the operation on it is: its CHECK, foreign key, unique constraint or NOT NULL land as separate operations after the create-table and carry no advisory. A first deployment against an empty database therefore prints nothing here, and a mixed plan (some tables new, others altered) flags only the pre-existing tables. Each remaining verdict names that premise (instances on the previous release …), and the premise field of the JSON warning repeats it, so a job that has no previous release can set the advisory aside on the evidence of the message alone.

Two ways to handle it:

1. No overlap window — a maintenance-window or scale-to-zero deploy (old pods gone before new ones start) has no simultaneous old code, so a single-step drop/rename is fine. The advisory doesn't apply; ignore it.
2. Zero-downtime rollout — split the breaking change into two releases, each of which keeps both code versions working (expand/contract):
   • Expand (release N): add the new shape — a nullable column, a new table, a backfill, dual-write from the new code. Old code ignores it.
   • Cut over: the new code reads/writes the new shape; deploy it.
   • Contract (release N+1): once no pod runs the old code, drop/rename/narrow the now-unused old shape. This step's db plan is clean.

Renaming orders.total → orders.amount under zero downtime is: add amount (expand) → backfill + dual-write → cut reads over → drop total (contract) — three releases, never one, so no in-flight pod ever references a column that isn't there.

Make it a hard gate if you always rolling-deploy — VOLTRO_ROLLING_DEPLOY=1. The db plan ⚠ is advisory by default, because a maintenance-window / scale-to-zero deploy has no overlap window and the framework can't tell which you run. If your pipeline is always a rolling update, set VOLTRO_ROLLING_DEPLOY=1 in the migration Job's env: voltro db apply then refuses (exit 2) a plan containing a rolling-unsafe operation instead of warning, so an un-split breaking change fails the deploy rather than breaking pods at runtime. Override a specific apply with --force. Unset (the default) leaves today's advisory behaviour untouched.

#If you DO run voltro dev in a cluster (dev / staging only)

voltro dev answers the same three probes on the app port as voltro serve, from the moment it knows the port — before discovery, codegen and the auto-migrate — so a probe can watch the slow boot it otherwise couldn't see:

• GET /internal/liveness → 200 ok from the first moment of the process, through the whole boot — and through a failed auto-migrate, where the process stays up waiting for the next file change.
• GET /internal/readiness → 503 until the app is serving, then 200.
• GET /internal/startup → JSON { phase, ready, elapsedMs } with tablesTotal during the migrate and error with the message on a failed one; 503 until ready, 200 from then on — for a startupProbe, and for humans and dashboards watching progress.

There is no second probe port and no variable to move it: point every probe at the app port, exactly as the production manifest above does. A generous startupProbe.failureThreshold on /internal/startup gives a large-schema first migrate minutes instead of a fixed window, and holds the liveness probe back for exactly that long:

# dev/staging pod running `voltro dev` — the SAME port and paths as production
startupProbe:
  httpGet: { path: /internal/startup, port: 4000 }
  periodSeconds: 10
  failureThreshold: 60        # up to 10 minutes for a first cold migrate
livenessProbe:
  httpGet: { path: /internal/liveness, port: 4000 }
readinessProbe:
  httpGet: { path: /internal/readiness, port: 4000 }

Even a local voltro dev boots faster on reboots now: the auto-migrate skips the full introspect when the declared schema is unchanged (a fingerprint check — one indexed query instead of scanning every table). Force a full re-introspect with VOLTRO_MIGRATE_FORCE=1.

#4. Request limits & DoS

POST /rpc (the buffered JSON endpoint the SSR loaders use) is capped, and the cap is enforced as bytes arrive:

VOLTRO_MAX_RPC_BODY_BYTES=8388608   # default 8 MiB
VOLTRO_MAX_BODY_BYTES=8388608       # default 8 MiB

A declared Content-Length over the cap is refused up front, so an honest client gets its 413 without uploading anything — a courtesy, not the enforcement, since a Transfer-Encoding: chunked body declares no length at all. The byte counter is the enforcement: it stops accumulating the moment the running total crosses the limit, drains the rest of the upload rather than dropping the connection, and answers 413. Both shapes therefore end in the same status code, and the refusal is logged under the voltro:security scope with the cap and the byte count at which the server stopped reading.

Scope: this guards the /rpc JSON path only. File uploads ride separate storage routes with their own limits.maxBytes, and WebSocket frames are capped by the ws library default (100 MiB).

Put per-IP rate limiting and the primary body-size cap at the ingress — that is the correct layer: it holds per-IP state and works across replicas, which an in-process limit can't.

# nginx ingress — annotations on the Ingress resource
nginx.ingress.kubernetes.io/proxy-body-size: "8m"
nginx.ingress.kubernetes.io/limit-rps: "20"

For app-level throttling (per-subject / per-tenant, e.g. an expensive action), use @voltro/plugin-ratelimit and the plugin onHttpRequest interceptor seam. It complements the ingress cap — it does not replace it.

There is no rate limit in the box. The body cap above is the only request guard the runtime applies by default; @voltro/plugin-ratelimit is opt-in, so an app that has not installed and configured it has no per-IP, per-API-key or per-tenant cap on /rpc at all. The one on-by-default throttle anywhere in the framework is plugin-auth's brute-force lockout, which covers sign-in credential attempts and nothing else. Treat the ingress limit as required, not as belt-and-braces.

#Per-IP limits need a trusted proxy

The address the runtime rate-limits, geo-blocks and audits by is socket.remoteAddress — not x-forwarded-for, which any client can write. Behind an ingress that means every request counts against the proxy's address — and every sessions.ipAddress row records the proxy — until you declare the hop:

// app.config.ts
export default {
  security: {
    trustedProxies: ['private'],   // or: ['loopback'] · ['10.0.0.0/8'] · ['2'] · ['*']
  },
}

The same setting decides whether x-forwarded-proto is believed, which is what lets the runtime emit HSTS behind a TLS-terminating load balancer. Override it on a running deployment with VOLTRO_TRUSTED_PROXIES=private (comma-separated).

#Cross-site protection and its allowlist

Every state-changing request — POST /rpc, the /ws upgrade, every REST route projected from a publicApi: mutation, everything in apiConfig.restRoutes, and POST /v1/api-keys — refuses a browser request whose Origin is neither the Host it was sent to nor an allowlisted origin. A split web/api deployment must declare its web origin or the browser gets a 403 on every mutation, REST write and socket:

// app.config.ts
export default {
  security: {
    allowedOrigins: ['https://app.example.com'],
  },
}

The environment override is VOLTRO_ALLOWED_ORIGINS (comma-separated). Server-to-server callers (SSR loaders, mobile SDKs, other services) send no Origin and are unaffected. Full detail in Security.

#5. Multi-tenant isolation

Tables carrying the tenant() mixin are auto-scoped to the request's tenant — nothing more to do there. The gap is the anonymous request that matches no auth strategy: its tenant decides what every tenant-scoped table and every openAccess procedure answers, and voltro serve refuses to boot until app.config.ts has decided it:

// app.config.ts
export default {
  // ...
  auth: {
    anonymous: 'no-tenant',   // or 'refuse' | 'header' | { tenantId: 'public' }
  },
}

• 'refuse' — no unauthenticated callers at all.
• 'no-tenant' — they carry the empty tenant: tenant-scoped tables read empty, tables without tenant() and procedures that need no tenant answer. The choice for an api whose public surface is token-shaped.
• 'header' — the tenant is what the request's x-tenant header names. Only for public-per-tenant data: every openAccess procedure and every table without tenant() is then public for a client-chosen tenant.
• { tenantId } — every unauthenticated caller is that one tenant, the server's choice.

voltro dev runs an undecided app on the header, else the dev tenant, and says so; that convenience never reaches a voltro serve process. Full detail in Subject → Anonymous callers.

#6. Observability (wire it or fly blind)

By default nothing is exported. Wire it before you need it.

Traces + metrics ship to any OTLP collector (Tempo / Jaeger / Honeycomb / Grafana Agent) from one env var:

OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
OTEL_SERVICE_NAME=my-api

Structured logs as JSON:

VOLTRO_LOG_FORMAT=json
VOLTRO_LOG_LEVEL=info

In json mode (the default off a TTY, so a pod gets it without configuration) EVERY line the framework emits is a parseable record — the boot banner, the app surface, voltro db apply's plan summary, refusal detail, and every subsystem logger (schedule, broadcast, workflow, flow-control). A hand-formatted table or a [tag]-prefixed adapter line between JSON records is a bug, not a style: log collectors show it as unparsed noise. On a TTY the same surfaces render as the human-readable banners and tables.

#Boot refusal after a container restart

A stopped voltro dev supervisor carries the child's exit code. Its successor is a new process, not a continuation of its memory. Voltro does not fabricate refused to boot N times from the pod name: a restart count does not establish that all exits had the same cause. Nor does a file in a container's writable layer provide a cross-container failure journal. Keep that history in the deployment's log collector, keyed by pod UID, container and time.

For the immediate Kubernetes diagnosis, use the previous container's log and the recorded container status (replace api-pod, api and the namespace):

kubectl logs api-pod -n staging -c api --previous --timestamps
kubectl get pod api-pod -n staging -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" exit="}{.lastState.terminated.exitCode}{" reason="}{.lastState.terminated.reason}{" message="}{.lastState.terminated.message}{"\n"}{end}'

Set this on the API container in the pod template to retain the tail of a failed container's log as its termination message when its termination file is empty:

terminationMessagePolicy: FallbackToLogsOnError

This is a bounded last-exit clue, not a full history or a promise that the first refusal survives later log output. It does not replay logs into the next container. See Kubernetes termination messages.

Error reporting — add sentryPlugin() from @voltro/plugin-sentry. It stays inert until SENTRY_DSN is set, and reported errors correlate to the request traceId:

import { sentryPlugin } from '@voltro/plugin-sentry'

export default {
  // ...
  plugins: [sentryPlugin()],
}

Durable in-DB traces are off in production by default — traces belong in your OTLP backend, not your OLTP database.

#7. Graceful shutdown

On SIGTERM / SIGINT, voltro serve shuts down cleanly (exit 0), in this order: it stops accepting new connections and lets in-flight requests finish against a fully-alive app (bounded — the request drain gets 60% of the shutdown grace, so the teardown behind it always fits inside the deadline), then stops schedulers, detaches subscribers / reactions / aggregates, drains the analytics sink and mirror, ends any remaining WebSocket, and closes the SQL connection pool last (waiting for in-flight transactions). Verified against a real serve under signal: a request in flight when SIGTERM lands completes with a full response before the process exits — no preStop hook, no orchestrator. A bare voltro serve under docker compose drains itself.

The transactional-outbox worker is part of that sequence: its poll timer and its change subscription are released, and a delivery already in flight is awaited, before the pool closes. An outbox row that was still pending is not lost — it is durable, and the next process's first pass picks it up, which is one of the three reasons that poll exists.

What an orchestrator still adds: routing. The app finishes every request it has accepted — but a request that arrives after SIGTERM is refused (the listener closes immediately, on purpose), and only the layer that routes traffic can stop sending it. Failing readiness from inside the process doesn't help: the listener is already closed. Under k8s, close that window with a preStop hook:

spec:
  # Must exceed the preStop sleep + the app's own shutdown.
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            # k8s removes the pod from the Service endpoints AND runs this
            # BEFORE sending SIGTERM. The sleep holds the pod up (listener open,
            # finishing in-flight) while endpoint removal propagates — so no new
            # request lands on a pod that's about to close.
            command: ["sh", "-c", "sleep 5"]

Without the preStop hook, a rolling update refuses the small window of requests that still route to a terminating pod before k8s finishes removing it from the Service endpoints — refused with a connection error, not truncated mid-response (everything already accepted completes either way). With it, that window is served too. terminationGracePeriodSeconds must be larger than the sleep plus the app's own teardown, or k8s SIGKILLs mid-drain. Environments with no endpoint removal at all — docker compose above all — need nothing: there is no routing layer to lag behind the shutdown, so the built-in drain is the whole story.

Bound the app's own teardown with VOLTRO_SHUTDOWN_GRACE_MS. After SIGTERM, the runtime runs its finalizers (pool close, plugin onDeactivate, analytics flush, trace persist) and then exits — but installing the signal handler removes node's default kill, so a finalizer that never completes (a pool drain against a database that is already gone, a wedged onDeactivate) would otherwise hang the process forever. A hard deadline caps that: teardown gets until the deadline, then the process exits regardless. It defaults to 10s; set VOLTRO_SHUTDOWN_GRACE_MS (clamped to 1s–5min) to sit JUST UNDER your terminationGracePeriodSeconds minus the preStop sleep — so the app drains and exits cleanly on its own before k8s SIGKILLs it mid-drain:

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      env:
        # preStop sleep (5s) + app teardown (≤22s) < 30s grace, with headroom.
        - name: VOLTRO_SHUTDOWN_GRACE_MS
          value: "22000"

Live WebSockets are ENDED promptly at shutdown — the drain never lets an open socket hold the process to the deadline — and the web client's supervisor treats any close as a reconnect signal, so an open dashboard re-attaches to a healthy replica across a rolling deploy without a page reload.

#8. Multiple replicas

Cache and KV default to in-process (per-replica). For a shared backend across replicas:

CACHE_BACKEND=redis
KV_BACKEND=redis

Cross-replica reactivity is the subtle one: a write on one pod never surfaces on another pod's open subscriptions unless the replicas share a change bus. Use:

• postgres — LISTEN/NOTIFY (built in),
• mariadb — binlog CDC (built in),
• other dialects — @voltro/plugin-broadcast (Redis).

Schedules and aggregates auto-coordinate via an advisory lock on SQL stores — no extra config to keep them from double-firing across replicas.

#The prerequisites you learn at the SECOND pod

Every item below is invisible on one replica and breaks on two. They are collected here because an operator reported each of them separately, each found the same way: the first pod proved the configuration worked.

The connection pool multiplies, the database limit does not.

DB_MAX_CONNECTIONS=10       # per replica — the fleet opens up to this × replicaCount

The framework opens ONE pool per process. At 4 replicas a pool of 10 is 40 connections against a database that still allows whatever it allowed before you scaled. An operator's second pod died on Connection timed out for exactly this. voltro serve now prints the number and the arithmetic at boot:

In voltro dev and voltro serve, every framework user of the database shares that pool: the store, the plugin bind context, the workflow runtime, aggregates, analytics, and the boot steps that migrate or check the schema. Each of them used to open its own pool of this size, so a process could hold several times the printed number. A deployment of two pods at DB_MAX_CONNECTIONS=5 behind a pooler limited to 15 clients measured it. Separate from it, by design: read-replica pools (DB_REPLICA_URLS, counted in the line), a separate migration URL (DB_DIRECT_URL / DB_MIGRATE_URL), a provisioned tenant's database, and pools your own code builds.

The workflow runtime has a floor. On a first boot it holds its migration lock and its runner lock while the migration runs on a third connection, and once running it keeps the runner lock. So it needs 3 connections, or 2 with VOLTRO_WORKFLOW_RUNNER_STORAGE=memory (a single-process runtime without cross-pod shard handoff). Below that the boot refuses and names the number, instead of waiting out the acquire timeout on Failed to acquire connection.

db pool: max=10 per replica (DB_MAX_CONNECTIONS) + 1 outside the pool (CDC LISTEN consumer)
= 11 per process × 4 replicas = up to 44 connections. Check that against your database's limit.
The one outside the pool does not come out of the pool budget; it comes out of the DATABASE's.

Set REPLICA_COUNT from your deployment (Helm: {{ .Values.replicaCount }}) and the line does the multiplication for you; without it the line still names the formula.

voltro dev prints it too, when the environment says it is not a laptop. A bare voltro dev stays silent — one process, no replicas, nothing to multiply. But voltro dev is a supported way to RUN an app, and a deployment that uses it needs this line as much as any other. So it prints whenever REPLICA_COUNT, DB_MAX_CONNECTIONS / PG_MAX_CONNECTIONS, or DB_REPLICA_URLS is set — each of which means somebody has already decided something about the number.

Some connections are not in the pool, and the count is per process. A connection that speaks a long-lived protocol cannot be returned to a pool, so the driver opens a standalone one. There are four such places and a full deployment can hold several at once:

Process	Connection	When
api voltro serve / dev	CDC LISTEN consumer	postgres, changeStrategy: 'cdc' (the default)
api voltro serve / dev	binlog CDC reader	mysql / mariadb, changeStrategy: 'cdc'
web voltro start	ISR invalidator LISTEN	any page declares cacheInvalidatesOn
web voltro start	postgres ISR cache client	SSR_CACHE=postgres

Each of them connects with the app's own resolved connection, TLS included (?sslmode=require, ?ssl=true, PG_SSL); none reads a database URL of its own. A database that only accepts TLS therefore serves the out-of-pool connections exactly as it serves the pool.

SQL Server is the one CDC dialect that costs nothing here: Change Tracking is read with ordinary queries through the pool, so its out-of-pool count is a verified zero rather than an omission.

Two of the four are not a LISTEN at all — the binlog reader speaks the replication protocol, and the ISR cache client is an ordinary client — which is why counting LISTEN rows in pg_stat_activity undercounts. Each process prints its own number in the boot line above — including No connections outside the pool in this process when there are none, so "counted, zero" is distinguishable from "not counted".

And a rolling update needs the surge pod's connections too. A budget sized for replicaCount is exactly full at steady state and short during every deploy: maxSurge adds a pod that opens a full pool of its own. If that pod cannot connect it never becomes ready, so the rollout does not complete and the cluster stays at the higher pod count — the deploy cannot free itself. Size for (replicaCount + maxSurge) × (DB_MAX_CONNECTIONS + out-of-pool).

POD_IP is each replica's identity, not only a workflow setting.

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Without it every replica registers under the same host, so they are one runner as far as the cluster is concerned. The boot warning for it fires only on SQL cluster storage, so a deployment that has not adopted durable workflows yet gets no signal at all — inject it as a matter of course.

Derive the broadcast namespace from something that cannot be forgotten.

- name: VOLTRO_BROADCAST_NAMESPACE
  value: {{ .Release.Namespace }}

Staging and production of the same app share a name, code and fingerprint, so the auto-derived namespace does NOT separate them — only this variable does. An operator's own guidance, and better than ours was: a value taken from the release namespace cannot be left out of one environment's config file, because there is no file to forget.

#The framework's own background pollers

Two framework tasks ride the coordinated scheduler and write a row into _voltro_schedule_claims on every tick they win: the workflow admission drainer and the offloaded-inference dispatcher.

Disarming requires complete wake coverage and no recovery deadline. A primary-store LISTEN/NOTIFY or broadcast channel alone does not establish this for physical workflow placements. Admission keeps discovering resident stores and retains a five-second recovery ceiling, or an earlier known deadline. With the default one-second base, a healthy empty admission worker backs off through 2, 4, then 5 seconds. It still writes coordination claims; this is not a promise of zero idle writes. Failed discovery, pending work, lost claims or repeated wakes can keep it at the base interval. Inspect the actual reason with voltro workflows flow --format json (diagnostics).

Other tasks may disarm when all conditions hold. Without complete remote change visibility they retain polling. The generic ceiling is:

VOLTRO_POLL_CEILING_MS=30000   # how long an arrival can wait when NOTHING woke the task

Nothing is lost in that case either — the replica that enqueued always sees its own write inline and drains it itself. What the ceiling covers is the narrower case of a crashed writer's lease being reclaimed by someone else.

A number worth knowing before you tune anything: on a two-replica deployment that had never enqueued into either queue, these two tasks accounted for 99.3 % of the claim ledger — 2 506 rows an hour against 18 from the app's own eight schedules. A fixed interval has no way to learn a queue is empty. That is what changed; the ceiling is the fallback, not the fix.

#Tuning the cadence

The intervals are declarable, with defaults most apps never change:

// app.config.ts
export default {
  scheduling: {
    admissionDrainMs: 1000,   // workflow admission drainer
    inferenceTickMs:  250,    // offloaded-inference dispatcher
    cancelSweepMs:    2000,   // cancelOn sweep
    pollCeilingMs:    30000,  // generic ceiling; admission retains a 5s recovery bound
    // Leave unset. The bound on adopting a `queue` firing stranded by its
    // replica is DERIVED per schedule (twice that schedule's own period) —
    // a fixed number is either generous enough to resurrect an outage's work
    // on a fast schedule or tight enough to drop a handoff on a slow one.
    // Set it only to state a policy the cadence cannot express.
    // queueAdoptionWindowMs: 300000,
  },
}

Each has a matching env var — VOLTRO_ADMISSION_DRAIN_MS, VOLTRO_INFERENCE_TICK_MS, VOLTRO_CANCEL_SWEEP_MS, VOLTRO_POLL_CEILING_MS, VOLTRO_QUEUE_ADOPTION_WINDOW_MS — which overrides the config field, the same way VOLTRO_TENANT_ISOLATION overrides tenancy.isolation. The config is what a project declares; the env var is what an operator changes on a running deployment without a rebuild.

Lowering an interval does not make anything more responsive: an arrival already wakes the task at once. It only bounds the case where nothing announced the work. A value of 0 or a non-number is ignored rather than honoured — a zero interval would turn an idle task into a spin.

#Workflow failover across replicas

On a SQL store (postgres / mysql / mariadb / mssql), durable workflows survive a replica crash: completed step({...}) activities are checkpointed in the cluster journal, so when a replica dies mid-run, a surviving replica takes over the run and continues it from the last completed step — it replays the completed steps rather than re-running them. (On sqlite the engine is single-process — durable within one replica, no cross-replica failover.) Two requirements:

• Inject POD_IP (K8s downward API, fieldRef: status.podIP) or set VOLTRO_WORKFLOW_RUNNER_HOST. This is each replica's cluster identity — without a distinct value, every replica registers as the same runner and they stop distributing shards (and cross-pod resume degrades). The boot logs a warning if it sees localhost with SQL storage.
• Make step side effects idempotent. Failover is at-least-once at the step boundary: a crash between a side effect and its journal write re-runs that step. A step's own retry: does not change this — it's about the step you're inside, not the replica handoff.

Is polling the bottleneck? No — reclaim is a lease, not a poll. A crashed replica keeps its shards until its heartbeat goes stale; only then can a survivor claim them. So takeover latency is bounded by the lease TTL (~35s by default), not by any message-poll interval, and a push mechanism (LISTEN/NOTIFY) does not move it. Two knobs tune it:

VOLTRO_WORKFLOW_FAILOVER_LEASE=15       # seconds a dead replica's work stays locked (default 35)
VOLTRO_WORKFLOW_FAILOVER_HEARTBEAT=5    # lease-refresh cadence (default 10; keep ≈ lease/3)

Lower the lease for faster failover, at the cost of false-positive reclaims: if a healthy replica is paused longer than the lease by a GC pause or a DB-latency spike, another replica may briefly also claim its shards. Keep the heartbeat around a third of the lease so one slow refresh doesn't trip a reclaim. For crash detection that doesn't depend on the timeout at all, pair it with a K8s liveness probe so a dead pod is removed promptly.

(A separate concern is new-message pickup: a workflow triggered on the replica that owns its shard starts immediately, but one owned by ANOTHER replica is otherwise picked up on that replica's next storage poll — up to 10s. If you run a broadcast broker (Redis/NATS — which a multi-replica deployment already does for cross-replica reactivity), this is automatic and near-instant: a trigger pushes a "wake" over the bus and the shard owner re-polls at once, on any SQL dialect. Without a broker, the change stream does the same job wherever remote changes reach the spine — a Postgres-only multi-replica fleet (LISTEN/NOTIFY CDC) is the common case: a remote replica's signal, start context, or run transition arrives as a change event and triggers an immediate, coalesced re-poll, so signal/step latency stops being poll-bounded there too. Only with neither a broker nor CDC does the poll interval remain the bound — tune VOLTRO_WORKFLOW_POLL_INTERVAL=2 then. Unrelated to the failover path above.)

#Checklist

• VOLTRO_SESSION_SECRET set from voltro secret generate session, in a secrets manager
• Rotation window uses VOLTRO_SESSION_SECRET_PREVIOUS
• Liveness / readiness probes point at /internal/liveness + /internal/readiness
• Serving pods run voltro serve (not voltro dev), with VOLTRO_AUTO_MIGRATE=0
• Schema applied by a pre-deploy Job / initContainer (voltro db apply), not in the serving pod
• VOLTRO_MAX_RPC_BODY_BYTES + VOLTRO_MAX_BODY_BYTES (plugin routes/webhooks) sane; ingress caps body size + per-IP rate
• VOLTRO_TRUSTED_PROXIES set if you run behind an ingress AND rate-limit per IP
• VOLTRO_ALLOWED_ORIGINS set if the web app is on a different origin than the api
• Security headers reviewed (VOLTRO_SECURITY_HEADERS, VOLTRO_CSP); HSTS reaching the browser over https
• Every *.webhook.tsx declares its verification, and each signature-verified one has its VOLTRO_WEBHOOK_SECRET_<ID>
• auth.anonymous decided — 'refuse', 'no-tenant', { tenantId }, or 'header' only for public-per-tenant data; voltro serve refuses to boot without it
• OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME pointed at your collector
• VOLTRO_LOG_FORMAT=json; sentryPlugin() + SENTRY_DSN for errors
• terminationGracePeriodSeconds generous for graceful drain; VOLTRO_SHUTDOWN_GRACE_MS set just under it (minus the preStop sleep)
• CACHE_BACKEND / KV_BACKEND + a cross-replica change bus when running >1 replica
• DB_MAX_CONNECTIONS set so pool × replicaCount fits your database's limit — read the db pool: boot line before raising replicaCount
• POD_IP injected via the downward API on EVERY multi-replica deployment, not only for durable workflows
• VOLTRO_BROADCAST_NAMESPACE derived from the release namespace — staging and production do not separate themselves
• For durable workflows on >1 replica: SQL store + POD_IP injected; tune VOLTRO_WORKFLOW_FAILOVER_LEASE if a 35s takeover is too slow; step side effects idempotent

#Native dependencies in server bundles

Server bundles resolve native modules from the deployed application’s installed driver and plugin dependencies, including their transitive dependencies. This supports strict pnpm layouts, where SQLite’s native module is installed under the Effect SQLite driver rather than at the application root. Keep production dependencies with the deployed bundle. Resolution uses the deployed dependency tree, not absolute paths from the build machine. If a resolved native module fails to load, its original error is reported instead of trying another installed copy.