Clustering

Clustering is built in. There is no plugin to install and no config flag to flip: run more than one instance of your app against the same database and they form a cluster automatically. Durable workflows resume on any surviving instance, each cron fires exactly once across the whole fleet, and writes on one instance become reactive subscriptions on every other.

@effect/cluster is a core dependency of @voltro/workflow — the CLI wires it for you when voltro dev or voltro start boots an app on a SQL store. You scale out by running N instances; the coordination machinery is already there.

#How it's enabled

There is nothing to enable in code. The wiring is automatic and driven by the store dialect:

Store dialect	Runner storage	Horizontal scale-out
postgres, mysql, mariadb, mssql	sql	yes — workflows resume on any instance
sqlite	memory	no — durable replay within one process
memory (dev default)	—	single process

On a SQL store, the runtime composes the dialect's SqlClient → @effect/cluster SingleRunner → @effect/workflow ClusterWorkflowEngine into one layer and provides it to your program. sqlite falls back to in-process durable replay (no cross-instance scale-out) because @effect/cluster's runner-lock acquisition has no SQLite branch.

You do not call the cluster layer yourself — the CLI does. The boot log tells you which mode resolved:

[voltro:dev] workflow engine: cluster-sql, dialect=mariadb

Two settings tune this for demanding topologies: VOLTRO_WORKFLOW_SHARD_LOCK keeps cross-pod handoff correct on a Galera / Percona XtraDB cluster (see Galera and multi-primary clusters), and VOLTRO_WORKFLOW_RUNNER_STORAGE opts out of SQL runner storage when you don't need handoff (see Single-process runner storage).

#The deployment contract

A real multi-instance deployment has to give each pod a routable identity. This is the entire operational surface of clustering, and it is what the framework's Helm baseline injects:

• Routable runner host (POD_IP). When a workflow that started on a now-dead pod has to resume elsewhere, the surviving pod addresses the dead pod's runner by the address it advertised. The default is localhost, which is unreachable cross-pod. Inject the Kubernetes downward-API status.podIP as POD_IP (or set VOLTRO_WORKFLOW_RUNNER_HOST) so each pod advertises its real address. The runtime warns at boot if a SQL-backed runner is still on localhost.
• Fixed runner port. SingleRunner otherwise picks a random port, which breaks restart-resume because the runner table still references the previous process's port. Pin it via VOLTRO_WORKFLOW_RUNNER_PORT (default 34000) so a restarted pod inherits the same cluster identity.
• Unique, stable server_id per pod (MariaDB CDC). The binlog reader needs a server_id that is unique across the fleet and stable across reschedules — duplicate ids silently break binlog streams. The runtime derives it by hashing a per-pod replica id (POD_NAME → FNV-1a). With a StatefulSet the pod name is ordinal and deterministic.
• Replication user (MariaDB). The binlog CDC reader connects with a dedicated user holding REPLICATION SLAVE, REPLICATION CLIENT, against a server configured binlog_format=ROW, binlog_row_image=FULL, gtid_strict_mode=ON.

The relevant downward-API wiring, from the framework's Helm baseline (the AWB 4-replica MariaDB deployment is the worked example):

env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }   # → stable per-pod CDC server_id
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }    # → routable workflow-runner host
  - name: VOLTRO_WORKFLOW_RUNNER_HOST
    value: "$(POD_IP)"
  - name: VOLTRO_WORKFLOW_RUNNER_PORT
    value: "34000"
  - name: DB_DIALECT
    value: "mariadb"
  - name: CDC
    value: "1"

That is the whole contract: stable per-pod identity (POD_NAME → server_id), routable runner host (POD_IP), a pinned runner port, the shared DB_URL, and CDC on. No application code changes between one instance and four.

#Galera and multi-primary clusters

Multi-primary MariaDB (Galera) and Percona XtraDB are first-class and keep full cross-pod handoff — no proxy, no single-writer endpoint, no loss of durability. This is handled automatically; the paragraphs below are the why for operators who need it.

By default @effect/cluster coordinates which runner owns which shard with session advisory locks (GET_LOCK on MySQL/MariaDB, pg_advisory_lock on Postgres). Advisory locks are node-local — a lock held on one Galera node is invisible on the others, and Galera never replicates them. So on a Galera cluster whose app pods reach the database through a load-balancing Service (connections spread across nodes), advisory-lock coordination can't see itself: pods split-brain shard ownership, and the runner-storage bootstrap can wedge before it creates cluster_runners (the pod stays un-Ready while a shard-lock refresher errors against a cluster_runners table it thinks is missing).

Voltro resolves this with the VOLTRO_WORKFLOW_SHARD_LOCK setting (auto | row | advisory, default auto):

• auto — probe the live connection (@@wsrep_on) and switch to the row-lease path on a Galera / PXC cluster, keep advisory locks everywhere else. No operator config needed.
• row — force the row-lease path. SqlRunnerStorage coordinates shard ownership with a certified conditional upsert on the cluster_locks table (INSERT … ON DUPLICATE KEY UPDATE … WHERE acquired_at < <expiry>) instead of GET_LOCK. Galera certifies that write across all nodes — the same reason the cron advisoryLock tier uses a claims row, not a session lock — so shard ownership is coordinated correctly no matter which node a connection lands on. Cross-pod handoff works: a dead pod's lease ages out and a surviving pod takes over the shard.
• advisory — force the classic GET_LOCK path (single-primary only).

The boot log reports the resolved mode, and voltro cluster status shows it per instance (shard-lock=row):

[voltro:serve] shard-lock coordination: row-based (dialect=mariadb, mode=auto, wsrep/Galera cluster detected)

Two honest caveats. Row-lease failover is expiry-based, not instant — a dead pod's shards are reclaimed after the lease timeout (seconds), where advisory locks release the instant a connection drops. That's why auto keeps the faster advisory path on a single-primary server and only pays the timeout on Galera. And MySQL Group Replication (and other non-wsrep multi-primary topologies) aren't auto-detected — set VOLTRO_WORKFLOW_SHARD_LOCK=row explicitly there.

#Single-process runner storage

If a deployment genuinely does not need cross-pod handoff — dev, a single-region single-replica service, or one where each pod owning its own workflows is acceptable — set VOLTRO_WORKFLOW_RUNNER_STORAGE=memory to skip the SQL runner storage entirely:

VOLTRO_WORKFLOW_RUNNER_STORAGE=memory

This is the engine sqlite uses. It skips SqlRunnerStorage (no cluster_runners / cluster_locks, no advisory locks) while durability is unaffected — run/message/reply state still persists via SqlMessageStorage. The tradeoff is the point: no cross-pod handoff — a workflow started on a now-dead pod will not resume on a surviving one. Prefer the row-lease coordination above for multi-replica deployments that need handoff; reach for memory only when they don't.

VOLTRO_WORKFLOW_RUNNER_STORAGE=sql is rejected on sqlite / turso (no advisory-lock branch there), and an unrecognized VOLTRO_WORKFLOW_SHARD_LOCK value fails boot loudly — a misconfiguration never silently selects a broken engine.

#Durable workflows resume on any instance

Workflow durability lives in @effect/cluster's engine, not in any Voltro table. Each workflow's journal (steps, idempotency keys, signals) is persisted to the cluster's storage on the shared database. When a pod dies mid-step:

1. Its runner lease stops being renewed.
2. Another runner reclaims the dead runner's shards.
3. The reclaiming runner reads the workflow journal, fast-forwards past completed steps, and resumes from the unfinished step.

Because steps are journaled with idempotency keys, a re-executed step replays its cached result rather than running its side effects twice. This is why the POD_IP runner host matters: the reclaiming pod must be able to reach the workflow's runner address to take it over.

@effect/cluster is dialect-agnostic here — its SqlRunnerStorage and SqlMessageStorage dispatch internally via sql.onDialectOrElse({ mssql, mysql, sqlite, orElse: postgres }). MariaDB rides the mysql branch (GET_LOCK, ON DUPLICATE KEY UPDATE, native RETURNING since 10.5).

mssql only: @effect/cluster's mssql storage has two driver-level bugs the framework fixes with a patch that a plain install can't carry. Run voltro add mssql once, then pnpm install — see SQL Server → Workflow cluster. No-op on every other dialect.

#Crons fire once cluster-wide

Schedule coordination is app-wide. On a postgres store the default is advisoryLock; you can opt into cluster, or force single for a one-instance deployment with app.config.ts (scheduling: { coordination }). Individual *.cron.tsx files stay topology-agnostic.

coordination	Exactly-once mechanism	When
single	none — always fires	dev, single pod
advisoryLock	claims row in _voltro_schedule_claims	multi-instance, default on a SQL store
cluster	@effect/cluster shard ownership (ClusterCron)	when you already run the workflow engine and want one ownership model

advisoryLock is the lightweight tier. Every replica computes the same deterministic scheduledAt from the cron expression (not its own Date.now(), so clock skew is irrelevant), derives a claim key ${scheduleName}@${secondBucket}, and races to INSERT a row into _voltro_schedule_claims. The primary-key conflict makes exactly one replica win; the rest lose cleanly and skip. The second-precision bucket makes claims self-expiring — a crashed winner doesn't block the next firing, because the next firing is a new bucket and therefore a new key. This is deliberately a claims row, not a session-level lock, because a connection pool can hand a GET_LOCK/pg_advisory_lock connection to another query before the unlock.

cluster rides the same sharding machinery as durable workflows: ClusterCron fires on exactly one runner (the shard owner) and migrates ownership on runner death. The in-app timer is disarmed in this mode (@effect/cluster owns the clock); the firing is recorded with coordinationOutcome: 'cluster'. It needs a SQL store with cluster storage (everything except sqlite/memory), and degrades to single with a warning when that's unavailable.

Overlap policy (onOverlap: 'skip' | 'queue' | 'parallel') is also cluster-aware: skip checks for a status='running' row from any pod, not just the local one, so a long run on pod A suppresses a new firing on pod B.

#Cross-instance reactivity (per dialect)

A write on instance A has to become a reactive subscription delta on instance B. The transport is change-data-capture, and it is not Postgres-only — each dialect uses its native change feed:

Dialect	Transport	Latency
postgres	LISTEN/NOTIFY	fast
mariadb	binlog CDC (ROW image, GTID)	push-based, per-replica reader
mysql	inline emit only (no binlog CDC)	single-process
sqlite	in-process bus	single-process

On MariaDB the binlog is the message bus — each replica tails it itself, resuming from its own _voltro_cdc_offsets row keyed by replica id. There is no Redis, NATS, or external broker. CDC is on by default for any SQL dialect (CDC=1); set CDC=0 to fall back to single-process inline emit. The boot log reports the resolved transport:

[voltro:dev] sql dialect resolved: mariadb — CDC: binlog CDC (ROW), RETURNING: native (INSERT/DELETE); UPDATE then SELECT

#Observability

• Dashboard. Workflow runs, steps, and schedule runs stream live to the devtools/cloud dashboard.
• Tables. Introspection rows live on the shared database — query them directly:
  • _voltro_workflow_start_contexts — durable starter subject/trace/source plus parent execution id and parent-close policy, loaded by whichever runner first executes the workflow or child workflow.
  • _voltro_workflow_runs, _voltro_workflow_run_steps, _voltro_workflow_run_events — workflow execution history (these are fire-and-forget introspection; durability itself lives in the cluster engine's own tables).
  • _voltro_schedule_runs — every firing, with replicaId (which instance fired it) and coordinationOutcome (single / wonLock / cluster / …).
  • _voltro_schedule_claims — the live exactly-once claim rows for advisoryLock schedules.
• Logs and traces. voltro logs --tail 100 shows the resolved dialect, CDC flavor, coordinator, and runner identity at boot, plus every firing and resume. voltro traces correlates a workflow's steps across pods.

#voltro cluster status

voltro cluster status renders the clustering snapshot of every running api — discovered the same way as voltro logs / voltro traces (the runtime registry), then merged with the live coordination state each instance reads from the shared store:

voltro cluster status                 # pretty table of every instance
voltro cluster status --process myApi # restrict to one registered api
voltro cluster status --format json   # machine-readable, pipe to jq

For each instance it shows the replicaId, the runner address (host:port), the resolved dialect and clustering tier (full / inline-only / single-process / none), the CDC flavor, the coordination kind, and — on the MariaDB CDC path — the binlog server_id. A SQL-backed runner advertising localhost / 127.0.0.1 is flagged loudly: other pods can't reach it, so cross-pod workflow resume silently breaks (inject POD_IP via the K8s downward API or set VOLTRO_WORKFLOW_RUNNER_HOST). A footer summarises the shared coordination state — active replicas seen in recent runs, current schedule claims, and in-flight schedule + workflow runs. The same view is on the dashboard's Cluster panel.

Quick "which instance is running what" query:

SELECT "replicaId", count(*)
FROM "_voltro_schedule_runs"
WHERE status = 'running'
GROUP BY "replicaId";

#The multi-dialect story

Clustering works on every SQL dialect, with one capability gradient:

• postgres / mariadb / mssql — full clustering: durable workflow resume across instances, advisoryLock or cluster crons, native cross-instance reactivity (LISTEN/NOTIFY on postgres, binlog CDC on mariadb).
• mysql — durable workflow resume and schedule coordination work; cross-instance reactivity is inline-only (no binlog CDC path), so subscriptions don't fan out across instances.
• sqlite — durable workflow replay within a single process (runnerStorage: 'memory'); no horizontal scale-out.
• memory — single-process dev store; no clustering.

The same app code runs unchanged across all of them — the runtime resolves the right runner storage, coordinator, and CDC transport from the dialect at boot.

clusterPlugin() — deliberately not built. A convenience marker plugin (declaring an app "expects multi-instance", optionally pinning the runner host/port) was considered and rejected: clustering is already automatic for any SQL store with N instances, so there is no behaviour for such a plugin to toggle. Its only knobs — runner host/port — are env-driven (VOLTRO_WORKFLOW_RUNNER_HOST / VOLTRO_WORKFLOW_RUNNER_PORT) and injected by the deployment (K8s downward API), which is where pod identity belongs. The idea is on record but will not ship.

#Anti-patterns

• Assuming clustering is Postgres-only. It isn't. MariaDB is a first-class clustering target (binlog CDC), and the AWB 4-replica MariaDB deployment is the production reference. wal_level=logical is a Postgres detail, not a clustering requirement.
• Leaving the runner host on localhost in multi-pod deployments. Resume-on-another-pod silently breaks — pods can't reach a localhost runner. Inject POD_IP. The runtime warns about this at boot.
• Reusing a server_id across MariaDB pods. Duplicate server_id silently breaks binlog streams. Derive it from a stable per-pod POD_NAME (the framework does this for you).
• Session-level locks for cron coordination. advisoryLock uses a claims row, not pg_advisory_lock/GET_LOCK, precisely because pooled connections make session locks unreliable. Don't reach for session locks.