CDC-out — declarative reverse-ETL

@voltro/plugin-cdc-out streams your table changes OUTWARD to an external sink. Declare { table → sink, map } and the plugin mirrors every change through a durable outbox in your app's own database — no separate Debezium / Fivetran pipeline. The data-team wedge: app data in the warehouse, declared in code, composed onto the change stream you already have.

#Wiring

// app.config.ts
import { cdcOutPlugin, webhookSink } from '@voltro/plugin-cdc-out'

export default {
  type: 'api' as const,
  name: 'api',
  plugins: [
    cdcOutPlugin({
      sinks: [
        {
          table: 'orders',
          sink:  webhookSink('https://warehouse.example.com/ingest'),
          // Map a row → the outbound record body (default: the row unchanged).
          map:   (row) => ({ id: row.id, total: row.total, status: row.status }),
          // Optional: only mirror some changes.
          filter: (event) => event.op !== 'delete',
          batchSize: 500,          // max records per delivered batch (default 100)
        },
      ],
      maxAttempts: 5,              // per-record delivery attempts before dead-letter
      backoffBaseMs: 200,          // first retry delay; doubles per attempt, jittered
      deliveryTimeoutMs: 10_000,   // per-attempt timeout — aborts the sink call
    }),
  ],
}

One sink config per table per instance — a duplicate table is a loud boot error. For multiple sinks on one table, wire a second instance with a name (cdcOutPlugin({ name: 'analytics', sinks: […] })); it suffixes the plugin name (@voltro/plugin-cdc-out#analytics) and the inspect mount (/_voltro/inspect/plugins/cdc-out--analytics/…).

#Sinks

• memorySink() — in-process, records every delivered batch. Dev + tests.
• webhookSink(url, { headers? }) — POSTs each batch as { records: [...] } JSON, honoring the engine's per-attempt abort signal. Its host is declared as a network:outbound:<host> permission automatically.
• Warehouse / Kafka — implement the CdcSink interface ({ name, deliver(batch, ctx), outboundHost? }). deliver may return a Promise or an Effect — both compose without wrapping. The engine is connector-agnostic; the sink is the only thing that changes.

#Delivery guarantees — exactly what holds

Every mirrored change becomes a row in _voltro_cdcout_outbox (contributed via extendSchema, migrated by voltro dev). The row's TypeID id is the record's deliveryKey — unique across replicas, stable across restarts and retries.

• Exactly-once ENQUEUE, fleet-wide. On changeScope: 'local' stores each replica enqueues only its OWN commits (injected cross-replica events are skipped). On 'fleet' stores (postgres changeStrategy: 'cdc' LISTEN/NOTIFY, mysql binlog) only the holder of the leader lease (_voltro_cdcout_leases, TTL-heartbeat) enqueues. Fail closed: no lease, no enqueue.
• At-least-once FROM ENQUEUE. The tap is post-commit — a crash in the narrow window between commit and the outbox insert loses that one event; the plugin does not claim better. From enqueue on, delivery survives restarts, retries with exponential backoff + jitter, and re-sends the SAME deliveryKeys — a sink that upserts on them dedupes safely.
• Ordered per pipe. One in-flight batch per pipe, consumed strictly in commit (id) order; the queue is head-blocking, so a batch waiting out its retry backoff is never overtaken.
• Dead-letter, never silent drop. A record that exhausts maxAttempts moves to status dead with its last error — queryable at GET /_voltro/inspect/plugins/cdc-out/dead-letter — and unblocks the pipe.
• Bounded storage. Delivered/dead rows are purged by the framework retention sweep after retentionHours (default 72, env CDCOUT_RETENTION_HOURS); pending rows are never purged.

#Multi-tenancy

The change tap fires for every tenant's rows, and a sink is app-global, developer-authored config: whatever map returns (default: the raw row, including tenantId) is mirrored to ONE external endpoint. Use filter / map to scope or strip tenant data, and only ever point sinks at operator-controlled URLs — never at a tenant-supplied one.

#Backfill

Seed a pipe with existing rows once — they flow through the SAME durable outbox as live changes (ordered, retried, dead-lettered, stable per-row deliveryKeys). Call from a *.startup.tsx or a CLI with the app's store:

import { enqueueBackfill } from '@voltro/plugin-cdc-out'
import { sinkConfig } from './cdc'

const n = await enqueueBackfill(store, sinkConfig, rows) // every current row, enqueued as an insert