Clustering

Clustering is built in. There is no plugin to install and no config flag to flip: run more than one instance of your app against the same database and they form a cluster automatically. Durable workflows resume on any surviving instance, each cron fires exactly once across the whole fleet, and writes on one instance become reactive subscriptions on every other.

@effect/cluster is a core dependency of @voltro/workflow — the CLI wires it for you when voltro dev or voltro start boots an app on a SQL store. You scale out by running N instances; the coordination machinery is already there.

#How it's enabled

There is nothing to enable in code. The wiring is automatic and driven by the store dialect:

Store dialect	Runner storage	Horizontal scale-out
postgres, mysql, mariadb, mssql	sql	yes — workflows resume on any instance
sqlite	memory	no — durable replay within one process
memory (dev default)	—	single process

On a SQL store, the runtime composes the dialect's SqlClient → @effect/cluster SingleRunner → @effect/workflow ClusterWorkflowEngine into one layer and provides it to your program. sqlite falls back to in-process durable replay (no cross-instance scale-out) because @effect/cluster's runner-lock acquisition has no SQLite branch.

You do not call the cluster layer yourself — the CLI does. The boot log tells you which mode resolved:

[voltro:dev] workflow engine: cluster-sql, dialect=mariadb

Two settings tune this for demanding topologies: VOLTRO_WORKFLOW_SHARD_LOCK keeps cross-pod handoff correct on a Galera / Percona XtraDB cluster (see Galera and multi-primary clusters), and VOLTRO_WORKFLOW_RUNNER_STORAGE opts out of SQL runner storage when you don't need handoff (see Single-process runner storage).

#File-backed SQLite ownership

A local sqlite workflow engine owns its journal for the engine's lifetime. A second engine pointed at the same database now fails startup before consuming workflow messages, instead of executing the first engine's active work again. Stop the first engine before starting its replacement. For concurrent executing replicas, use a database with shared SQL runner coordination.

Ownership uses a separate <database>.voltro-workflow-lock SQLite file and the operating system's native lock lifecycle. Normal shutdown, boot failure and process death release the lock; the empty sidecar remains on disk. Do not delete that file while an engine is running. The guard does not hold a write transaction on the application database, so ordinary data reads, writes and dashboard observers remain possible.

This protection applies to local file-backed sqlite engines wired by normal dev/serve boot. Lower-level hosts receive the same guard when they pass dialectId: 'sqlite' to workflowEngineLayer. It does not establish distributed ownership for remote Turso, in-memory databases or other hosts that omit dialect metadata.

#The deployment contract

A real multi-instance deployment has to give each pod a routable identity. This is the entire operational surface of clustering, and it is what the framework's Helm baseline injects:

• Routable runner host (POD_IP). When a workflow that started on a now-dead pod has to resume elsewhere, the surviving pod addresses the dead pod's runner by the address it advertised. The default is localhost, which is unreachable cross-pod. Inject the Kubernetes downward-API status.podIP as POD_IP (or set VOLTRO_WORKFLOW_RUNNER_HOST) so each pod advertises its real address. The runtime warns at boot if a SQL-backed runner is still on localhost, GET /_voltro/inspect/cluster carries it as localhostRisk, and voltro cluster prints a red localhost runner line for the affected replica — so a fleet that lost the downward-API mapping shows it on every surface, not only in the boot log nobody re-reads.
• Fixed runner port. SingleRunner otherwise picks a random port, which breaks restart-resume because the runner table still references the previous process's port. Pin it via VOLTRO_WORKFLOW_RUNNER_PORT (default 34000) so a restarted pod inherits the same cluster identity. A value that is not a port (1..65535) is ignored with a warning and the default is used — including the tcp://… Kubernetes injects for a Service named voltro-workflow-runner.
• Unique, stable server_id per pod (MariaDB CDC). The binlog reader needs a server_id that is unique across the fleet and stable across reschedules — duplicate ids silently break binlog streams. The runtime derives it by hashing a per-pod replica id (POD_NAME → FNV-1a). With a StatefulSet the pod name is ordinal and deterministic.
• Replication user (MariaDB). The binlog CDC reader connects with a dedicated user holding REPLICATION SLAVE, REPLICATION CLIENT, against a server configured binlog_format=ROW, binlog_row_image=FULL, gtid_strict_mode=ON.

The relevant downward-API wiring, from the framework's Helm baseline:

env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }   # → stable per-pod CDC server_id
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }    # → routable workflow-runner host
  - name: VOLTRO_WORKFLOW_RUNNER_HOST
    value: "$(POD_IP)"
  - name: VOLTRO_WORKFLOW_RUNNER_PORT
    value: "34000"
  - name: DB_DIALECT
    value: "mariadb"
  - name: CDC
    value: "1"

That is the whole contract: stable per-pod identity (POD_NAME → server_id), routable runner host (POD_IP), a pinned runner port, the shared DB_URL, and CDC on. No application code changes between one instance and four.

#Galera and multi-primary clusters

Multi-primary MariaDB (Galera) and Percona XtraDB are first-class and keep full cross-pod handoff — no proxy, no single-writer endpoint, no loss of durability. This is handled automatically; the paragraphs below are the why for operators who need it.

By default @effect/cluster coordinates which runner owns which shard with session advisory locks (GET_LOCK on MySQL/MariaDB, pg_advisory_lock on Postgres). Advisory locks are node-local — a lock held on one Galera node is invisible on the others, and Galera never replicates them. So on a Galera cluster whose app pods reach the database through a load-balancing Service (connections spread across nodes), advisory-lock coordination can't see itself: pods split-brain shard ownership, and the runner-storage bootstrap can wedge before it creates cluster_runners (the pod stays un-Ready while a shard-lock refresher errors against a cluster_runners table it thinks is missing).

Voltro resolves this with the VOLTRO_WORKFLOW_SHARD_LOCK setting (auto | row | advisory, default auto):

• auto — probe the live connection (@@wsrep_on) and switch to the row-lease path on a Galera / PXC cluster, keep advisory locks everywhere else. No operator config needed.
• row — force the row-lease path. SqlRunnerStorage coordinates shard ownership with a certified conditional upsert on the cluster_locks table (INSERT … ON DUPLICATE KEY UPDATE … WHERE acquired_at < <expiry>) instead of GET_LOCK. Galera certifies that write across all nodes — the same reason the cron advisoryLock tier uses a claims row, not a session lock — so shard ownership is coordinated correctly no matter which node a connection lands on. Cross-pod handoff works: a dead pod's lease ages out and a surviving pod takes over the shard.
• advisory — force the classic GET_LOCK path (single-primary only).

The boot log reports the resolved mode, and voltro cluster status shows it per instance (shard-lock=row):

[voltro:serve] shard-lock coordination: row-based (dialect=mariadb, mode=auto, wsrep/Galera cluster detected)

Two honest caveats. Row-lease failover is expiry-based, not instant — a dead pod's shards are reclaimed after the lease timeout (seconds), where advisory locks release the instant a connection drops. That's why auto keeps the faster advisory path on a single-primary server and only pays the timeout on Galera. And MySQL Group Replication (and other non-wsrep multi-primary topologies) aren't auto-detected — set VOLTRO_WORKFLOW_SHARD_LOCK=row explicitly there.

#Single-process runner storage

If a deployment genuinely does not need cross-pod handoff — dev, a single-region single-replica service, or one where each pod owning its own workflows is acceptable — set VOLTRO_WORKFLOW_RUNNER_STORAGE=memory to skip the SQL runner storage entirely:

VOLTRO_WORKFLOW_RUNNER_STORAGE=memory

This is the engine sqlite uses. It skips SqlRunnerStorage (no cluster_runners / cluster_locks, no advisory locks) while durability is unaffected — run/message/reply state still persists via SqlMessageStorage. The tradeoff is the point: no cross-pod handoff — a workflow started on a now-dead pod will not resume on a surviving one. Prefer the row-lease coordination above for multi-replica deployments that need handoff; reach for memory only when they don't.

VOLTRO_WORKFLOW_RUNNER_STORAGE=sql is rejected for local or embedded SQLite-family connections. Direct remote Turso primaries use SQL runner storage with row locks automatically. An unrecognized VOLTRO_WORKFLOW_SHARD_LOCK value still fails boot; an advisory preference on a SQLite-family SQL runner resolves to its supported row-lock mechanism.

#Durable workflows resume on any instance

Workflow durability lives in @effect/cluster's engine, not in any Voltro table. Each workflow's journal (steps, idempotency keys, signals) is persisted to the cluster's storage on the shared database. When a pod dies mid-step:

1. Its runner lease stops being renewed.
2. Another runner reclaims the dead runner's shards.
3. The reclaiming runner reads the workflow journal, fast-forwards past completed steps, and resumes from the unfinished step.

A completed step replays its persisted result without running its side effects again. An interrupted step whose result was not persisted runs again on the surviving runner. Any side effects it performed before the crash can therefore repeat: make external writes idempotent, or use a transactional outbox for effects that must be coupled to database state. This is why the POD_IP runner host matters: the reclaiming pod must be able to reach the workflow's runner address to take it over.

A runner that is alive but cannot use its database gives its shards up the same way. Its heartbeat and its shard locks ride one reserved connection, apart from the app's pool, so a process whose pool has stopped handing out connections would otherwise keep heart-beating and keep every shard it owns — and a run started on one of those shards would sit at pending. Each runner therefore probes its pool on the lock-refresh cadence (VOLTRO_WORKFLOW_FAILOVER_HEARTBEAT, 10 s by default). When the pool has not served a query for three of those intervals, the runner releases every shard lock, stops its heartbeat and logs workflow runner: its connection pool has not served a query for …; another runner takes the shards once the lease runs out. The runner takes part again once its pool has answered for three intervals in a row; a single answer from a pool that frees a slot for a moment is not a recovery. Readiness reports the same pool through its SELECT 1 ping (see production hardening).

@effect/cluster is dialect-agnostic here — its SqlRunnerStorage and SqlMessageStorage dispatch internally via sql.onDialectOrElse({ mssql, mysql, sqlite, orElse: postgres }). MariaDB rides the mysql branch (GET_LOCK, ON DUPLICATE KEY UPDATE, native RETURNING since 10.5).

mssql only: @effect/cluster's mssql storage has two driver-level bugs the framework fixes with a patch that a plain install can't carry. Run voltro add mssql once, then pnpm install — see SQL Server → Workflow cluster. No-op on every other dialect.

#Crons fire once cluster-wide

Schedule coordination is app-wide. On a postgres store the default is advisoryLock; you can opt into cluster, or force single for a one-instance deployment with app.config.ts (scheduling: { coordination }). Individual *.cron.tsx files stay topology-agnostic.

coordination	Exactly-once mechanism	When
single	none — always fires	dev, single pod
advisoryLock	claims row in _voltro_schedule_claims	multi-instance, default on a SQL store
cluster	@effect/cluster shard ownership (ClusterCron)	when you already run the workflow engine and want one ownership model

advisoryLock is the lightweight tier. Every replica computes the same deterministic scheduledAt from the cron expression (not its own Date.now(), so clock skew is irrelevant), derives a claim key ${scheduleName}@${secondBucket}, and races to INSERT a row into _voltro_schedule_claims. The primary-key conflict makes exactly one replica win; the rest lose cleanly and skip. The second-precision bucket makes claims self-expiring — a crashed winner doesn't block the next firing, because the next firing is a new bucket and therefore a new key. This is deliberately a claims row, not a session-level lock, because a connection pool can hand a GET_LOCK/pg_advisory_lock connection to another query before the unlock.

cluster rides the same sharding machinery as durable workflows: ClusterCron fires on exactly one runner (the shard owner) and migrates ownership on runner death. The in-app timer is disarmed in this mode (@effect/cluster owns the clock); the firing is recorded with coordinationOutcome: 'cluster'. It needs a SQL store with cluster storage (everything except sqlite/memory), and degrades to single with a warning when that's unavailable.

Overlap policy (onOverlap: 'skip' | 'queue' | 'parallel') is also cluster-aware: skip checks for a status='running' row from any pod, not just the local one, so a long run on pod A suppresses a new firing on pod B.

#Cross-instance reactivity (per dialect)

A write on instance A has to become a reactive subscription delta on instance B. The transport is change-data-capture, and it is not Postgres-only — each dialect uses its native change feed:

Dialect	Transport	Latency
postgres	LISTEN/NOTIFY	fast
mariadb	binlog CDC (ROW image, GTID)	push-based, per-replica reader
mysql	inline emit only (no binlog CDC)	single-process
sqlite	in-process bus	single-process

On MariaDB the binlog is the message bus — each replica tails it itself, resuming from its own _voltro_cdc_offsets row keyed by replica id. There is no Redis, NATS, or external broker. CDC is on by default for any SQL dialect (CDC=1); set CDC=0 to fall back to single-process inline emit. The boot log reports the resolved transport:

[voltro:dev] sql dialect resolved: mariadb — CDC: binlog CDC (ROW), RETURNING: native (INSERT/DELETE); UPDATE then SELECT

#Observability

• Dashboard. Workflow runs, steps, and schedule runs stream live to the devtools/cloud dashboard.
• Tables. Introspection rows live on the shared database — query them directly:
  • _voltro_workflow_start_contexts — durable starter subject/trace/source plus parent execution id and parent-close policy, loaded by whichever runner first executes the workflow or child workflow.
  • _voltro_workflow_runs, _voltro_workflow_run_steps, _voltro_workflow_run_events — workflow execution history (these are fire-and-forget introspection; durability itself lives in the cluster engine's own tables).
  • _voltro_schedule_runs — every firing, with replicaId (which instance fired it) and coordinationOutcome (single / wonLock / cluster / …).
  • _voltro_schedule_claims — the live exactly-once claim rows for advisoryLock schedules.
• Logs and traces. voltro logs --tail 100 shows the resolved dialect, CDC flavor, coordinator, and runner identity at boot, plus every firing and resume. voltro traces correlates a workflow's steps across pods.

#voltro cluster status

voltro cluster status renders the clustering snapshot of every running api — discovered the same way as voltro logs / voltro traces (the runtime registry), then merged with the live coordination state each instance reads from the shared store:

voltro cluster status                 # pretty table of every instance
voltro cluster status --process myApi # restrict to one registered api
voltro cluster status --format json   # machine-readable, pipe to jq

For each instance it shows the replicaId, the runner address (host:port), the resolved dialect and clustering tier (full / inline-only / single-process / none), the CDC flavor, the coordination kind, and — on the MariaDB CDC path — the binlog server_id. A SQL-backed runner advertising localhost / 127.0.0.1 is flagged loudly: other pods can't reach it, so cross-pod workflow resume silently breaks (inject POD_IP via the K8s downward API or set VOLTRO_WORKFLOW_RUNNER_HOST). A footer summarises the shared coordination state — active replicas seen in recent runs, current schedule claims, and in-flight schedule + workflow runs. The same view is on the dashboard's Cluster panel.

Quick "which instance is running what" query:

SELECT "replicaId", count(*)
FROM "_voltro_schedule_runs"
WHERE status = 'running'
GROUP BY "replicaId";

#The multi-dialect story

Clustering works on every SQL dialect, with one capability gradient:

• postgres / mariadb / mssql — full clustering: durable workflow resume across instances, advisoryLock or cluster crons, native cross-instance reactivity (LISTEN/NOTIFY on postgres, binlog CDC on mariadb).
• mysql — durable workflow resume and schedule coordination work; cross-instance reactivity is inline-only (no binlog CDC path), so subscriptions don't fan out across instances.
• sqlite — durable workflow replay within a single process (runnerStorage: 'memory'); no horizontal scale-out.
• memory — single-process dev store; no clustering.

The same app code runs unchanged across all of them — the runtime resolves the right runner storage, coordinator, and CDC transport from the dialect at boot.

clusterPlugin() — deliberately not built. A convenience marker plugin (declaring an app "expects multi-instance", optionally pinning the runner host/port) was considered and rejected: clustering is already automatic for any SQL store with N instances, so there is no behaviour for such a plugin to toggle. Its only knobs — runner host/port — are env-driven (VOLTRO_WORKFLOW_RUNNER_HOST / VOLTRO_WORKFLOW_RUNNER_PORT) and injected by the deployment (K8s downward API), which is where pod identity belongs. The idea is on record but will not ship.

#Anti-patterns

• Assuming clustering is Postgres-only. It isn't. MariaDB is a first-class clustering target (binlog CDC), and multi-replica MariaDB deployments run it in production. wal_level=logical is a Postgres detail, not a clustering requirement.
• Leaving the runner host on localhost in multi-pod deployments. Resume-on-another-pod silently breaks — pods can't reach a localhost runner. Inject POD_IP. The runtime warns about this at boot.
• Reusing a server_id across MariaDB pods. Duplicate server_id silently breaks binlog streams. Derive it from a stable per-pod POD_NAME (the framework does this for you).
• Session-level locks for cron coordination. advisoryLock uses a claims row, not pg_advisory_lock/GET_LOCK, precisely because pooled connections make session locks unreliable. Don't reach for session locks.

#Remote Turso primaries

Both ordinary development and compiled production startup use the dialect's resolved connection capability. A direct remote Turso primary uses shared SQL runner registration and database-clock row leases. Replicas sharing that primary arbitrate workflow shards; after an executor is killed, an existing peer can resume unfinished work once ownership transfers. Completed steps remain recorded, while an unfinished step may run again: external effects still need idempotency.

Local Turso files and file-backed embedded replicas retain memory runner storage and single-process workflow limits. An embedded replica's sync URL does not make its local journal an authoritative shared runner store. VOLTRO_WORKFLOW_RUNNER_STORAGE=memory remains an explicit single-process override. The Cluster panel shows the actual runner storage and row lock mode rather than inferring memory storage from the Turso dialect name.

When upgrading an existing remote Turso deployment from memory runners, stop or drain those workers before starting SQL-runner workers against the same journal. Do not overlap the old independent memory owners with the new coordinated owners. The existing journal is retained; normal idempotency rules still apply to unfinished work.