Data (export / import / backup / restore)

voltro data moves your app's data and assets in and out. It has two families, because backup and portability are different jobs with different right answers:

• Logical — export / import: a portable, resumable, dialect-agnostic bundle. Use it for data takeout (GDPR), seeding staging from prod, or migrating across SQL dialects.
• Native — backup / restore: orchestrates the vendor tools (pg_dump, mysqldump, …) for a lossless, point-in-time same-dialect backup. Use it for disaster recovery.

Both stream — a table or a blob is never fully held in memory — and both survive interruptions.

#Logical export

voltro data export ./backup-2026-07-01
voltro data export ./out --tenant org_abc            # one tenant + its FK closure
voltro data export ./out --tables users,posts         # an explicit set
voltro data export ./out --assets                      # include stored blobs
voltro data export ./out --compression gzip            # zstd (default) | gzip | none
voltro data export ./out --consistency snapshot        # point-in-time (see below)
voltro data export ./out --target api --api-url https://api.example.com --token $SECRET  # export FROM a live instance (in-process)
voltro data export --target api --api-url https://api.example.com --token $SECRET --bundle-key backups/2026-07-01.vbundle  # instance exports straight to storage (scale)

The output is a directory bundle, not a zip — each table is an independent file, so an interrupted run resumes cleanly and per-table compression stays effective (a single zip would fight both). Over --target api this same bundle is streamed as one framed .vbundle archive (a zero-dependency concatenation, not a zip) and unpacked back into the directory on the other end.

backup-2026-07-01/
  README.md                  # human summary: date, source host, dialect, scope,
                             # table/row counts, and a LOUD real-vs-masked banner
  manifest.json              # format version, source dialect + schema fingerprint,
                             # per-table column types + row counts + checksums
  data/
    users.ndjson.zst         # one typed-encoded row per line, framed-compressed
    posts.ndjson.zst
  assets/
    index.ndjson             # key → sha256 → size → content-type
    9f86d081…                # blobs, content-addressed by sha256 (auto-deduped)
  .ledger.json               # resume checkpoint (which tables/assets are done)

README.md is a human-readable sidecar written on every export — provenance (when, from which host, which dialect, scope) plus a prominent data-sensitivity banner: it says outright whether the bundle holds real (unmasked) data or was masked at the source (and which columns). It's ignored on import (the importer only reads files listed in manifest.json) and never contains a connection string or credentials. It rides inside the .vbundle archive too.

#Single-file bundles (.vbundle) — one artifact, any size

Give the export/import a path ending in .vbundle and you get (or read) one self-contained file instead of a directory — everything (tables + blobs + manifest + README) in a single artifact you can copy A→B:

voltro data export ./snapshot.vbundle --assets            # one file, blobs included
voltro data import ./snapshot.vbundle --assets            # stream it back in
voltro data inspect ./snapshot.vbundle                    # peek at the metadata WITHOUT extracting
voltro data unpack ./snapshot.vbundle ./out               # explode a file to a dir to look inside

voltro data inspect prints the bundle's README.md (date, source host, dialect, scope, table/row counts, and the real-vs-masked banner) — read via an early-stop peek that stops after the metadata, so it stays fast even on a 200 GB bundle (add --json for the machine-readable manifest.json, --passphrase for an encrypted bundle). voltro data unpack fully extracts a file to a directory (blobs materialised under assets/), decrypting with --passphrase when needed.

It's built to work at any size (200 GB+) without needing that much scratch space. The pipeline streams end-to-end: on export, blobs are pulled from object storage straight into the file (nothing staged locally — peak local disk is just the tables); on import, blobs stream straight from the file to the destination's object storage (never unpacked to a temp dir first). The file is a framed archive (.vbundle, a zero-dependency streamable format — not a zip, whose central-directory-at-the-end design would force downloading the whole thing before reading entry one). Blobs are named by storage key with a per-blob sha footer, so integrity is per-entry and a resumed transfer skips blobs already present at the destination.

#Encryption at rest (--encrypt)

A .vbundle is plaintext by default. For a copy that must be confidential at rest (a backup in a bucket, a file on a laptop/CI, an unmasked dump — which contains your .encrypted() columns as plaintext), encrypt it:

voltro data export ./backup.vbundle --assets --encrypt --passphrase "$BUNDLE_KEY"
voltro data import ./backup.vbundle --assets --passphrase "$BUNDLE_KEY"
voltro data unpack ./backup.vbundle ./inspect --passphrase "$BUNDLE_KEY"

• A dedicated key, NOT the transfer secret. The passphrase comes from --passphrase or VOLTRO_BUNDLE_KEY and is independent of VOLTRO_DATA_TRANSFER_SECRET (auth ≠ encryption; different instances have different transfer secrets; a local export has none). It's stretched with scrypt (salt + params in the file header).
• Streaming, authenticated AES-256-GCM. The bundle is encrypted in chunks (the age/Tink STREAM construction), so it stays streaming + resumable at 200 GB, and it's tamper- and truncation-evident: a wrong passphrase, a flipped byte, or a dropped final chunk all fail decryption loudly — never a silent partial import.
• Complementary to masking. Masking makes the data safe for a lower environment (strips PII); encryption makes the artifact confidential. A DR backup wants encryption (full real data); a prod→dev copy wants masking (and can add encryption too).

#Why NDJSON, and why it's exact

Rows are written as newline-delimited JSON — streamable, resumable by line, and human-inspectable. A naïve JSON.stringify would corrupt data, so a typed codec (driven by the column types in the manifest) handles the values JSON can't:

• bigint → preserved exactly (never truncated to a float)
• NaN / Infinity → preserved (JSON would turn them into null)
• bytes → base64; timestamp/date → ISO-8601 → Date on import
• json / array / vector → structured, verbatim

#Scope

Flag	Selects
(none)	Every table (the default).
--tenant <id>	Every tenant()-scoped table filtered to that tenant, plus the FK closure in BOTH directions: (1) the transitive FK-parent closure of those rows — closure-pulled shared tables (a global users / reference table) are row-subset to the ids the tenant's rows actually reference, never exported in full; and (2) the child closure — rows that reference the tenant's rows (the comments on the tenant's projects) come along too, each scoped to the ids that actually point into the tenant set. The child walk is anchored on the tenant() tables, so a row that references only a shared parent (a global users another tenant also references) is not pulled — that would be a cross-tenant leak. A --tenant bundle therefore carries the tenant's parents AND children and no other tenant's rows — that's what makes it safe as a GDPR / per-tenant takeout.
--tables a,b	An explicit set (you own referential integrity; the importer's deferred-FK resolution covers load-order dangles, see below).

Tenant-scope details:

• A tenant scope without a tenant id refuses loudly (ScopeError): pass --tenant <id> (CLI) or scope.tenantId (API/profile). It never falls back to an unfiltered export.
• Which tables count as tenant-scoped comes from the tenant() mixin metadata when the CLI / admin endpoint can read the declared schema (authoritative — a table can carry a tenantId column without being tenant-scoped, e.g. a global users table's active-org pointer). Without that metadata the exporter falls back to a documented heuristic: any table with a tenantId column.
• A cross-tenant reference (a tenant-A row pointing at a tenant-B row) is never followed — the bundle stays tenant-clean and the reference dangles; importing such a bundle reports it loudly (DanglingReferenceError) unless the target already has the row.

#Consistency: live vs snapshot

• live (default) — each table is read in short keyset-paginated chunks. Resilient and easy on the database, but the tables are read at slightly different instants (a concurrent write can leave a child whose parent you already passed; the importer's deferred-FK resolution handles the dangling reference — see below).
• snapshot — every table is read inside one transaction pinned to a single MVCC snapshot (per-dialect isolation prelude), so the whole export is a consistent instant. The trade: that transaction is held open for the export's duration.

#Logical import

voltro data import ./backup-2026-07-01
voltro data import ./out --mode append --on-conflict skip   # insert-only
voltro data import ./out --mode replace --atomic            # full refresh, all-or-nothing
voltro data import ./out --target api --api-url https://api.example.com --token $SECRET  # upload to a live instance
voltro data import --target api --api-url https://api.example.com --token $SECRET --bundle-key backups/2026-07-01.vbundle  # instance pulls from storage (scale)
voltro data import ./out --assets            # also restore blobs
voltro data import ./out --no-verify         # skip checksum/row-count verification
voltro data import ./out --force             # import despite schema drift AND cross-dialect warnings

Import is integrity-checked (each table's checksum + row count verified as it decodes; each asset re-hashed against its content address), applies tables in FK-parent-first order, and a resumed run skips already-applied tables via the ledger.

#Schema-drift pre-flight

Every bundle records the schema fingerprint of the source it was exported from (a stable hash of the schema shape — the same fingerprint prod boot uses to detect drift). Before touching the target, import recomputes the target app's fingerprint the same way and compares it to the bundle's. On a mismatch it refuses, fail-closed, before any row lands — a drifted target (a missing column, a renamed table) would otherwise fail mid-load with a raw database error after some rows already committed:

✗ schema drift: the target schema (fingerprint d6ba94b0…) differs from the bundle's
  source (f67942f6…). Refusing before the table phase.
    items.extra: in the target, not in the bundle
  Re-export against the current schema, or pass --force to import anyway.

The error names the drift — the tables/columns that moved (down to per-column granularity for the tables the bundle carries) — so you can see exactly what changed. --force downgrades the refusal to a loud warning and proceeds (the operator has accepted that the shapes differ). Over the --target api path the same check runs on the instance against its declared schema and returns 409 schema drift with the fingerprints + diff; --force sends x-import-force: 1 to override.

The pre-flight only runs when the importer has a target schema (the CLI introspects it; the API endpoint uses the instance's declared schema). Importing into a fresh/empty database with no comparable schema simply skips the check.

#Deferred-FK resolution

A row whose write fails on a foreign-key constraint — a forward reference from a live-consistency export, a genuine FK cycle between tables (which the exporter orders by breaking the closing edge), or an intra-table self-reference to a later row — does not fail the import. It is held and resolved after every table has streamed:

1. Retry to a fixpoint — forward references resolve once the later tables landed.
2. FK-shedding — rows still stuck are written with their FK-bearing columns set to NULL (possible wherever those columns are nullable), which breaks row cycles on every dialect without session-level constraint toggles.
3. Patch pass — shed rows are re-written with the full bundle row, restoring the FK values.

Anything still unresolvable — the parent row exists in neither the bundle nor the target, or a NOT NULL FK cycle — fails with a typed DanglingReferenceError naming each offending table + primary key + the database's own reason. Held rows are the exception set, not the data set: memory is bounded by how many rows dangle at load time.

#Write modes (--mode)

Mode	Behaviour	Use it for	Conflict
upsert (default)	INSERT-or-UPDATE by primary key	sync / idempotent re-import	overwrites per row
append	INSERT only	additive data (event log, new seed)	--on-conflict skip (default) or fail
replace	TRUNCATE the target tables (children-first) then INSERT	full refresh — target ends up exactly the bundle	—

replace refuses a partial bundle (a subset / tenant / table scope): truncating would delete rows the bundle never carried. Re-export with full scope, or use upsert. (append --on-conflict fail throws on a duplicate primary key only where the store enforces the constraint — every SQL dialect does; the in-memory dev store overwrites.)

#Importing into a LIVE instance

A plain import connects straight to the database (not through the running app), so it is an uncoordinated concurrent writer — readers can see partial state, the reactive layer either storms (CDC dialects) or goes stale (others), and rows race with live writes. So import / restore refuse by default when a live instance is detected (via the local runtime registry, or an explicit --api-url probe). Two ways forward:

Target	How	Guarantees	Use for
direct (default)	writes straight to the DB	none while live — guarded; pass --allow-live to override	a stopped target, a replica, a dev/staging DB not serving traffic
direct + --atomic	wraps truncate+load in ONE transaction	MVCC readers see the import all-or-nothing (old until commit, new after) — the live-safe replace	replacing a live target's data with no partial-state window
in-process (--target api)	voltro data import <dir> --target api --api-url <url> --token <secret> uploads the packed bundle (or, with --bundle-key, has the instance pull it from storage) to the instance's secret-gated admin endpoint, which imports it in its own process through its store	full app pipeline — validation, mixins, field encryption, hooks — AND automatic reactivity (in-process writes emit change events, so subscriptions update; no separate resync)	a live merge (incl. PROD) that must honour app invariants

--atomic is the recommended live replace: one transaction, so live reads never see a half-loaded table. (It holds a write transaction for the load duration — writes to those tables block, readers don't. A shadow-table rename would shorten that lock window, but for a full replace concurrent writes are discarded on swap anyway, so it isn't the default.)

#The --target api data-transfer endpoints (prod-safe)

--target api moves data in/out of a running production instance without direct DB access. It does NOT reuse the dev data-viewer surface — two dedicated endpoints, POST /_voltro/admin/export and POST /_voltro/admin/import, both built to be safe on prod and both gated by the same secret:

• Secure by default — no secret, no endpoints. The instance mounts BOTH routes only when a data-transfer secret is configured: VOLTRO_DATA_TRANSFER_SECRET=<≥16 chars> (or serveApi({ dataTransferSecret })). Unset (or shorter than 16 chars) → the routes return 404. There is no "on" switch that leaves them open.
• Secret gate, constant-time. Every request must present that secret as a Bearer token (Authorization: Bearer <secret>), compared in constant time over SHA-256 digests (neither length nor content leaks via timing). The gate is a framework-owned secret independent of app RBAC — enabling data transfer can never accidentally ride on a user role. The CLI sends it via --token / VOLTRO_DATA_TRANSFER_SECRET (it must equal the server's secret). Wrong/absent token → 401.
• One secret, both directions. The same secret gates export (read) and import (write). If you need to grant export without import (e.g. a backup job that must never overwrite), split it into a per-operation credential deliberately — the default is one credential for the whole surface.

Import (POST /_voltro/admin/import) applies a bundle in the instance's own process through its store — full app pipeline (validation, mixins, field encryption, hooks) AND automatic reactivity. Two transports, neither needs a server-readable path:

• upload (default) — the CLI packs the bundle and uploads the bytes; the server unpacks to its own temp dir, imports in-process, cleans up. Buffered → small/moderate bundles.
• storage-pull (--bundle-key <key>) — the CLI sends only { "bundleKey": "<key>" }; the instance streams that archive from its configured object storage (S3 / Azure / MinIO / filesystem, resolved from the storage env — STORAGE_PROVIDER, S3_*, AZURE_*, …), no body buffering → arbitrarily large bundles.

Export (POST /_voltro/admin/export) reads the instance's data in-process (so field-decryption + hooks apply) and packs a bundle. ⚠️ This is a data-exfiltration surface — it can read all prod data — which is exactly why it sits behind the same off-by-default secret. It is also the right place for source-side masking on a live box: name a server-side profile and its scope/subset/masking apply before any byte leaves the instance (fail-closed — see the prod→dev section below). Two transports:

• download (default) — the response body is the packed bundle; the CLI writes it to <outDir>. Buffered → small/moderate exports.
• storage-push (--bundle-key <key>) — the instance exports straight to its object storage under that key and returns { bundleKey, tables, rows }, no response buffering → the scale path. Then pull it elsewhere (e.g. import --bundle-key).

The masking policy for a --target api export is named by --profile <name> and loaded on the server (data-profiles/<name>.ts) — version-controlled on the instance, never supplied by the client. No profile → a raw export (the secret-holder is trusted to read prod).

Assets over the API — full parity with the file path. With --assets, an API export streams the instance's blobs from its own object storage into the bundle (single-pass, nothing staged); an API import streams the bundle's blobs straight to the destination instance's storage as the archive arrives. So voltro data export prod.vbundle --assets (from a running instance via --target api) → voltro data import prod.vbundle --target api --api-url <dev> moves data and blobs at any size. (Uploading a directory bundle that has materialised assets over --target api transfers data only — pack a single-file .vbundle for assets; the CLI warns if you try.)

• Audited — every export/import logs the transport, table + row counts, and (export) how many columns were masked.

For a live merge (import) that must run app invariants + drive reactivity, this in-process path needs no quiesce/resync — in-process writes are reactive automatically.

#Cross-dialect

You can import a bundle into a different dialect than it came from — but only for the framework's portable DSL types. A cross-dialect import runs a portability lint first:

• Blocked (refused unless --force): raw() columns (verbatim source-dialect SQL), and vector columns targeting a dialect with no vector type.
• Warned (imported, represented differently): array / interval on a non-postgres target.
• Portable everywhere: text, integer, real, boolean, timestamp, date, json, bytes, reference, enum, id.

The lint refuses loudly rather than silently coercing — a blocked import tells you exactly which columns are the problem.

#Reliability: chunking, retry, resume

A production export can run for hours over millions of rows and gigabytes of blobs. It is built so a dropped connection, a pool blip, or an outright crash never means starting over — and so a huge table never blows up memory or knocks the source database over.

#Chunking — flat memory, gentle on the DB

Every table is read with a keyset cursor, not OFFSET: WHERE pk > :last ORDER BY pk LIMIT n (default 1000 rows/page, tune with chunkSize). Consequences:

• Flat memory — one bounded page is in memory at a time, regardless of table size (a 20 M-row table streams in 1000-row pages).
• O(1) per page on the pk index — OFFSET n re-scans and skips n rows every page (O(n²) over a full walk); keyset reads each row exactly once.
• Backpressure — a page is fetched only when the sink (encode → compress → disk) is ready to take it, so a slow disk throttles the DB reads instead of overrunning memory or hammering the server.

live consistency keeps each read short (no long-held transaction → vacuum-friendly); snapshot trades that for one pinned transaction held for the export's duration (see above).

#Retry — a blip doesn't kill the run

Each page read is retried on a transient failure (dropped connection, pooled-backend hiccup) with exponential backoff + jitter, up to 5 attempts by default (retryTimes). The retry is per page, and the keyset cursor is preserved — so a reconnected page resumes at exactly the row it stopped on, never re-emitting or skipping. A genuinely-broken read still surfaces after the attempts are exhausted rather than hanging.

#Resume — re-run and it continues

Both export and import checkpoint into a small .ledger.json and can be re-run to continue where they stopped:

• Export writes each table with an atomic temp-file + rename, and records the table (and each asset) in the ledger only once it's fully written. A crash mid-table leaves no half-written file (the temp is discarded); re-running skips the completed tables/assets and redoes only the unfinished one. Assets are content-addressed, so resume is per-asset — a blob whose hash is already in the bundle is skipped.
• Import keeps its own ledger (.import.ledger.json) and skips already-applied tables on a re-run. Correctness never depends on the ledger, though: every row is upserted by primary key, so redoing an in-flight table is always safe, and each table's checksum + row count and each asset's hash are verified as they load — a truncated or corrupted bundle fails loudly instead of importing garbage.

Net: interrupt an export or import at any point — network drop, Ctrl-C, OOM-killed pod — and re-running the same command finishes the job without duplicating work or corrupting the target.

#Progress & observability

A multi-hour job is not a black box. Both pipelines signal per table — never per row, so the reporting never slows the hot path:

• voltro data export|import prints a line per finished table — export [7/23] users · 1.2k rows — so you watch the run progress instead of staring at a silent terminal. A ledger-resumed table prints once with its recorded count.
• Programmatically, runExport / runImport take an optional onProgress callback. It fires a start then a done event per table, in FK-parent-first order, carrying the table name, its 0-based index, the tableCount, rowsDone, and total (the table's row count when known ahead of time — import reads it from the manifest; a live export learns it only once the table drains). A throwing progress renderer never aborts the job.
• Traces + metrics are always on, no wiring. Each table runs inside an Effect.withSpan('data-transfer.export.table' | 'data-transfer.import.table') (attributes table / index / phase), so if the app has tracing enabled the export/import shows per-table phase timing in the trace. Two metrics record throughput: voltro_data_transfer_rows (counter, tagged by phase + table) and voltro_data_transfer_table_seconds (histogram, tagged by phase).

import { runImport, type ProgressEvent } from '@voltro/data-transfer'

yield* runImport({
  store: target,
  bundleDir: './backup',
  onProgress: (e: ProgressEvent) => {
    if (e.event === 'done') console.log(`[${e.index + 1}/${e.tableCount}] ${e.table}: ${e.rowsDone} rows`)
  },
})

#Typed errors — CLI-catchable AND rpc-declarable

Every pipeline failure is a tagged error, caught by tag with Effect.catchTag(...). The errors that appear on the runExport / runImport error channels — the wire errors, and therefore exactly what the --target api admin endpoints surface — are Schema.TaggedError, so a handler can declare them on an rpc procedure's error: schema and the rpc encoder marshals them across the wire round-trip-safely (no hand-rolled JSON per tag):

BundleError, CodecError, IntegrityError, CrossDialectError, ImportModeError, DanglingReferenceError, SchemaDriftError, MaskingError, ScopeError.

The two internal errors — NativeToolError (native backup/restore) and CompressionError (folded into BundleError by the pipelines) — never cross the wire, so they stay plain Data.TaggedError: still catchable by tag, just no Schema surface.

import { Effect } from 'effect'
import { runImport, type SchemaDriftError } from '@voltro/data-transfer'

yield* runImport({ store: target, bundleDir: './backup', targetSnapshot }).pipe(
  Effect.catchTag('SchemaDriftError', (e: SchemaDriftError) => Effect.log(`refusing: schema drifted — ${e.diff.join('; ')}`)),
)

#Native backup / restore

voltro data backup ./backups/2026-07-01     # pg_dump --format=custom / mysqldump --single-transaction / …
voltro data restore ./backups/2026-07-01    # pg_restore / mysql / …

These shell out to the vendor tools resolved from your DB_DIALECT + connection env. They produce a dialect-native artifact (db.dump, db.sql, db.sqlite, db.bacpac) that is lossless and point-in-time consistent for same-dialect restore — the right tool for disaster recovery. Secrets are passed via the tools' environment variables (PGPASSWORD, MYSQL_PWD), never on the command line, where the tool supports it. The named tool must be installed and on PATH.

#Masking (prod → dev/stage safely)

Cloning prod into a lower environment must not carry real user data. voltro data export does this with masking: PII is replaced by realistic, referentially-consistent fakes at the source — before a row is ever written — so real values never reach the bundle, transit, or a developer's machine.

voltro data export ./out --profile dev

Masking is driven by two layers:

1. Classification in the schema — .sensitive(class) / .safe() on each column. This says what kind of data a column holds. It lives in the schema because the data's meaning is a property of the schema, not of one export.
2. A per-environment masking POLICY — how each class/column is transformed for this target. It lives in a profile so a target environment's whole recipe is one reviewable, version-controlled file.

// data-profiles/dev.profile.ts
import { defineDataProfile } from '@voltro/data-transfer'
export default defineDataProfile({
  subset: { seeds: { users: undefined } },
  masking: {
    seed: process.env.MASK_SEED!,
    // classes override the built-in defaults; columns override per-column
    classes: { freeText: 'redact' },
    columns: { '_voltro_mail_outbox.to': { fake: 'email' } },
    onUnclassified: 'error', // fail-closed (default)
  },
  consistency: 'snapshot',
  assets: true,
})

#Deterministic + seed-keyed

Every transform is deterministic and keyed on a secret seed:

• Same input → same fake, everywhere. One email becomes the SAME fake in every table it appears in, so joins survive; and it stays stable across re-runs, so dev data doesn't churn. Keeping the seed stable is pseudonymisation.
• Rotate or discard the seed → the mapping is irrecoverable. That makes the result anonymisation.

seed is REQUIRED — masking without one is a bug (pass it from the environment, never commit it). Transforms are also format-preserving (a fake email is a valid email) and null-preserving (a null stays null — nullability holds).

#Actions

An action is what a column's value becomes. Set them per class (classes) or per column (columns, which wins):

Action	Effect
keep	copy verbatim
null	set to null
redact	fixed placeholder ([redacted] for text, null otherwise)
hash	deterministic opaque hex (stable, non-reversible without the seed)
dateShift	shift a date by a seed-derived offset (relative intervals + ordering preserved)
{ fake: '<class>' }	a format-preserving fake of that class
{ custom: (input) => … }	your own transform (input = { value, table, column, columnType, seed })

#Class → action defaults

A minimal policy is just a seed — every known class has a default action:

Class	Default action
email fullName firstName lastName username phone address company url ip creditCard	{ fake: '<class>' }
date	dateShift
secret	null
freeText	redact

A custom class with no entry in classes falls back to redact. Resolution order for any column is: columns[table.column] → the class's action (classes → built-in default) → .safe() keeps → PK/FK keeps → onUnclassified.

#Fail-closed

Masking is fail-closed. A column that is neither .sensitive() nor .safe() (and isn't a PK/FK, and has no columns override) refuses the export and is named in the error — so a newly-added column can never silently leak PII to dev. Classify it, or override it in the policy.

The opt-out is onUnclassified: 'keep' (or 'null'), which makes masking fail-open for unclassified columns. Discouraged — it defeats the guarantee; prefer classifying the column. See why fail-closed.

#--dry-run — preview without writing

Preview a masking export without writing a bundle — the trust surface before real data moves. It reads a small sample per table and reports, per masked column, the before→after; a leak scan that warns when a KEPT column still LOOKS like PII (catches a misclassification — a .safe() on something that isn't); and the fail-closed list.

voltro data export ./out --profile dev --dry-run

  users
    email  [fake:email]  "ada@corp.com" → "grace.hopper1847@example.com"
    name   [fake:fullName]  "Ada Byron" → "Linus Torvalds"
    ssn    [null]  "078-05-1120" → null

  ⚠ possible leaks in KEPT columns:
    users.nickname looks like email: "ada@corp.com"

  ✗ unclassified (would BLOCK a real export — add .sensitive()/.safe() or a policy override):
    users.bio

  ✓ every exported column is classified — safe to run.

Nothing is written. --dry-run exits non-zero when the unclassified list is non-empty, so it doubles as a CI gate for classification coverage. (It requires a profile with a masking policy — there is nothing to preview otherwise.)

#Audit

A masking export records what it changed in the bundle's manifest.json, under masking:

"masking": {
  "transformed": ["users.email", "users.name", "users.ssn"]
}

masking.transformed lists every table.column that was pseudonymised/anonymised (i.e. every column whose action was not keep), so a reviewer can verify the copy was masked as intended — without diffing the data. A policy id is recorded alongside it when the policy sets one.

#Subsetting

subset exports a referentially-correct SLICE instead of whole tables. You give seed rows per table; the exporter adds their transitive FK-parent closure — every row the seeds (and their parents, recursively) point at — so the slice imports with no dangling references.

subset: {
  seeds: {
    users: undefined,                          // every user (no predicate)
    orders: eq('status', 'open'),              // only open orders (a Predicate)
  },
}
// exports those rows PLUS every parent row they reference (users an order points at, etc.)

A subset replaces scope when both are set — only the seeded tables and their parent closure are exported. Contrast the two:

• scope (--tenant / --tables / all) selects tables (a tenant's rows plus the row-subset FK parents they reference, or an explicit table set). Use it to move a tenant or named tables.
• subset selects specific rows and pulls in exactly the parents they need. Use it to carve a small, self-consistent slice ("these 1000 users and everything they reference").

Child closure. By default the closure follows parents only — a seeded user brings the org it belongs to, but not that user's posts. Set children to also pull in the rows that reference the seeded set, each scoped to the ids that actually point into it (never a whole child table). The child rows' own parents fold back through the parent closure, so the slice stays referentially complete:

subset: {
  seeds:    { orgs: eq('id', 'org_abc') },
  children: { roots: ['orgs'] },  // pull the org's users → their posts → those posts' comments
}

children.roots anchors the walk: a child comes along only when it references a roots row (or a child already pulled in this walk), so the walk stays tight — a row that references only a shared parent outside the anchor is not dragged in. (This is exactly what keeps a --tenant export tenant-tight: its child walk is anchored on the tenant() tables.) Use children: true for an unanchored org-slice takeout where there is no tenant boundary to respect.

Subsetting reads the selected rows into memory to collect id sets — it's built for small slices, not for halving a huge table (use scope/--tenant for that).

#Profiles

A profile bundles a target environment's whole recipe — scope/subset, masking, consistency, compression, assets — into one default-exported object, so voltro data export --profile dev is a single, reviewable, version-controlled description of how prod becomes dev.

import { defineDataProfile } from '@voltro/data-transfer'

export default defineDataProfile({
  scope:       { kind: 'all' },        // or omit and use --tenant/--tables
  subset:      { seeds: { users: undefined } }, // replaces scope when set
  masking:     { seed: process.env.MASK_SEED! },
  consistency: 'snapshot',             // 'live' | 'snapshot'
  compression: 'zstd',                 // 'zstd' | 'gzip' | 'none'
  assets:      true,                   // include stored blobs
})

--profile <name> resolves, in order: ./<name>, ./<name>.profile.ts, ./data-profiles/<name>.ts, ./data-profiles/<name>.profile.ts. So --profile dev finds ./dev.profile.ts or ./data-profiles/dev.ts. The module's default export must be the profile (use defineDataProfile for full type-checking).

CLI flags override profile values. --assets, --compression, --consistency, --tenant, --tables all win over what the profile sets — so a profile is the default and a flag is the one-off override.

#Which do I use?

Goal	Command
Disaster recovery / scheduled backups	voltro data backup (native)
Clone prod → staging (masked, no real PII)	voltro data export --profile <env> (masking + subset)
Preview a masked export before running it	voltro data export --profile <env> --dry-run
Clone prod → staging (same dialect, unmasked)	voltro data export then import (or native backup/restore)
GDPR / per-tenant takeout	voltro data export --tenant <id> --assets
A small self-consistent slice of the data	voltro data export --profile <env> with a subset
Move Postgres → MySQL	voltro data export then import (cross-dialect lint applies)