Data (export / import / backup / restore)

voltro data moves your app's data and assets in and out. It has two families, because backup and portability are different jobs with different right answers:

• Logical — export / import: a portable, resumable, dialect-agnostic bundle. Use it for data takeout (GDPR), seeding staging from prod, or migrating across SQL dialects.
• Native — backup / restore: orchestrates the vendor tools (pg_dump, mysqldump, …) for a lossless, point-in-time same-dialect backup. Use it for disaster recovery.

Both stream — a table or a blob is never fully held in memory — and both survive interruptions.

#Logical export

voltro data export ./backup-2026-07-01
voltro data export ./out --tenant org_abc            # one tenant + its FK closure
voltro data export ./out --tables users,posts         # an explicit set
voltro data export ./out --exclude cluster_locks       # everything else
voltro data export ./out --assets                      # include stored blobs
voltro data export ./out --compression gzip            # zstd (default) | gzip | none
voltro data export ./out --consistency snapshot        # point-in-time (see below)
voltro data export ./out --target api --api-url https://api.example.com --token $SECRET  # export FROM a live instance (in-process)
voltro data export --target api --api-url https://api.example.com --token $SECRET --bundle-key backups/2026-07-01.vbundle  # instance exports straight to storage (scale)

The output is a directory bundle, not a zip — each table is an independent file, so an interrupted run resumes cleanly and per-table compression stays effective (a single zip would fight both). Over --target api this same bundle is streamed as one framed .vbundle archive (a zero-dependency concatenation, not a zip) and unpacked back into the directory on the other end.

backup-2026-07-01/
  README.md                  # human summary: date, source host, dialect, scope,
                             # table/row counts, and a LOUD real-vs-masked banner
  manifest.json              # format version, source dialect + schema fingerprint,
                             # per-table column types + row counts + checksums
  data/
    users.ndjson.zst         # one typed-encoded row per line, framed-compressed
    posts.ndjson.zst
  assets/
    index.ndjson             # key → sha256 → size → content-type
    9f86d081…                # blobs, content-addressed by sha256 (auto-deduped)
  .ledger.json               # resume checkpoint (which tables/assets are done)

README.md is a human-readable sidecar written on every export — provenance (when, from which host, which dialect, scope) plus a prominent data-sensitivity banner: it says outright whether the bundle holds real (unmasked) data or was masked at the source (and which columns). It's ignored on import (the importer only reads files listed in manifest.json) and never contains a connection string or credentials. It rides inside the .vbundle archive too.

#Single-file bundles (.vbundle) — one artifact, any size

Give the export/import a path ending in .vbundle and you get (or read) one self-contained file instead of a directory — everything (tables + blobs + manifest + README) in a single artifact you can copy A→B:

voltro data export ./snapshot.vbundle --assets            # one file, blobs included
voltro data import ./snapshot.vbundle --assets            # stream it back in
voltro data inspect ./snapshot.vbundle                    # peek at the metadata WITHOUT extracting
voltro data unpack ./snapshot.vbundle ./out               # explode a file to a dir to look inside

voltro data inspect prints the bundle's README.md (date, source host, dialect, scope, table/row counts, and the real-vs-masked banner) — read via an early-stop peek that stops after the metadata, so it stays fast even on a 200 GB bundle (add --json for the machine-readable manifest.json, --passphrase for an encrypted bundle). voltro data unpack fully extracts a file to a directory (blobs materialised under assets/), decrypting with --passphrase when needed.

It's built to work at any size (200 GB+) without needing that much scratch space. The pipeline streams end-to-end: on export, blobs are pulled from object storage straight into the file (nothing staged locally — peak local disk is just the tables); on import, blobs stream straight from the file to the destination's object storage (never unpacked to a temp dir first). The file is a framed archive (.vbundle, a zero-dependency streamable format — not a zip, whose central-directory-at-the-end design would force downloading the whole thing before reading entry one). Blobs are named by storage key with a per-blob sha footer, so integrity is per-entry and a resumed transfer skips blobs already present at the destination.

#Encryption at rest (--encrypt)

A .vbundle is plaintext by default. For a copy that must be confidential at rest (a backup in a bucket, a file on a laptop/CI, an unmasked dump — which contains your .encrypted() columns as plaintext), encrypt it:

voltro data export ./backup.vbundle --assets --encrypt --passphrase "$BUNDLE_KEY"
voltro data import ./backup.vbundle --assets --passphrase "$BUNDLE_KEY"
voltro data unpack ./backup.vbundle ./inspect --passphrase "$BUNDLE_KEY"

• A dedicated key, NOT the transfer secret. The passphrase comes from --passphrase or VOLTRO_BUNDLE_KEY and is independent of VOLTRO_DATA_TRANSFER_SECRET (auth ≠ encryption; different instances have different transfer secrets; a local export has none). It's stretched with scrypt (salt + params in the file header).
• Streaming, authenticated AES-256-GCM. The bundle is encrypted in chunks (the age/Tink STREAM construction), so it stays streaming + resumable at 200 GB, and it's tamper- and truncation-evident: a wrong passphrase, a flipped byte, or a dropped final chunk all fail decryption loudly — never a silent partial import.
• Complementary to masking. Masking makes the data safe for a lower environment (strips PII); encryption makes the artifact confidential. A DR backup wants encryption (full real data); a prod→dev copy wants masking (and can add encryption too).

#Why NDJSON, and why it's exact

Rows are written as newline-delimited JSON — streamable, resumable by line, and human-inspectable. A naïve JSON.stringify would corrupt data, so a typed codec (driven by the column types in the manifest) handles the values JSON can't:

• bigint → preserved exactly (never truncated to a float)
• NaN / Infinity → preserved (JSON would turn them into null)
• bytes → base64; timestamp/date → ISO-8601 → Date on import
• json / array / vector → structured, verbatim

#Scope

Flag	Selects
(none)	Every table the app has DATA in (the default) — see the note below on the handful that describe a deployment rather than filling it.
--tenant <id>	Every tenant()-scoped table filtered to that tenant, plus the FK closure in BOTH directions: (1) the transitive FK-parent closure of those rows — closure-pulled shared tables (a global users / reference table) are row-subset to the ids the tenant's rows actually reference, never exported in full; and (2) the child closure — rows that reference the tenant's rows (the comments on the tenant's projects) come along too, each scoped to the ids that actually point into the tenant set. The child walk is anchored on the tenant() tables, so a row that references only a shared parent (a global users another tenant also references) is not pulled — that would be a cross-tenant leak. A --tenant bundle therefore carries the tenant's parents AND children and no other tenant's rows — that's what makes it safe as a GDPR / per-tenant takeout.
--tables a,b	An explicit set (you own referential integrity; the importer's deferred-FK resolution covers load-order dangles, see below).
--exclude a,b	Everything EXCEPT these. The scope stays all and records what was left out — so the deployment-describing tables are still filtered, and replace still accepts the bundle. Works on both targets (the instance resolves it against its own live list). Cannot be combined with --tables or --tenant; a name that does not exist is refused, because an exclusion that excludes nothing leaves the run looking like it worked.

#What all deliberately leaves out

A number of framework tables hold rows about a deployment rather than an app's data. The test is: would a row from elsewhere make this target act — send, run, admit, refuse, skip a delivery — or assert something untrue about its own history? That covers its migration ledger, file-migration and seed records, CDC offsets and change log, schedule claims, wakeups and firing history, workflow watermarks / pending starts / admissions / pauses / delivered events, its outbox and delivery attempts, its idempotency keys, its storage grants, its spend and usage accounting, and its own traces and undo log. They are dropped from all, skipped on import, and never emptied by a replace, and the run says which ones and why.

Every framework table is classified one way or the other, and a new one fails the build until somebody decides — the classification used to live in two places, the module and a hand-kept copy in its test, and the two disagreed about exactly the tables that later caused trouble.

The reason is worth one paragraph, because it cost a real environment ninety minutes. A scope: all bundle carried _voltro_migration_plans, replace wrote it, and the target's next boot refused:

auto-migrate: SCHEMA FINGERPRINT MISMATCH — declared=6e2c61081a9ed80c  live=28af9a54414f22f1

The refusal was right. That fingerprint is computed over the declared table set, so the imported row was not out of date, it was foreign: it stated a schema decision made somewhere else. (The declared set could also differ per environment then, because NODE_ENV decided two of the tables. It no longer does — but that removes one way for two deployments to differ, not the reason a foreign ledger row is wrong.) Two of the others would have made the target act — a pending workflow start runs a workflow somebody queued elsewhere, a pause silently stops one here.

all is the only scope filtered. Name one of these in --tables and you get it — an explicit name is an expectation, and this command refuses to drop those silently.

Every table needs a single-column primary key. The export is keyset-paginated, so it orders by one column and resumes from the last value on the next page. That column comes from the real primary key — a declared id() where there is one, otherwise the table's actual PK, whatever it is called.

A table with a composite primary key, or none at all, is refused by name rather than exported. Ordering by one column of a composite key splits equal values across page boundaries, which drops or duplicates rows into a bundle that reports success — and a short backup is discovered at the restore. Leave such a table out with --exclude.

Tenant-scope details:

• A tenant scope without a tenant id refuses loudly (ScopeError): pass --tenant <id> (CLI) or scope.tenantId (API/profile). It never falls back to an unfiltered export.
• Which tables count as tenant-scoped comes from the tenant() mixin metadata when the CLI / admin endpoint can read the declared schema (authoritative — a table can carry a tenantId column without being tenant-scoped, e.g. a global users table's active-org pointer). Without that metadata the exporter falls back to a documented heuristic: any table with a tenantId column.
• A cross-tenant reference (a tenant-A row pointing at a tenant-B row) is never followed — the bundle stays tenant-clean and the reference dangles; importing such a bundle reports it loudly (RowsRefusedError) unless the target already has the row.

#Consistency: live vs snapshot

• live (default) — each table is read in short keyset-paginated chunks. Resilient and easy on the database, but the tables are read at slightly different instants (a concurrent write can leave a child whose parent you already passed; the importer's deferred-FK resolution handles the dangling reference — see below).
• snapshot — every table is read inside one transaction pinned to a single MVCC snapshot (per-dialect isolation prelude), so the whole export is a consistent instant. The trade: that transaction is held open for the export's duration.

#Logical import

voltro data import ./backup-2026-07-01
voltro data import ./out --mode append --on-conflict skip   # insert-only
voltro data import ./out --mode replace --atomic            # full refresh, all-or-nothing
voltro data import ./out --target api --api-url https://api.example.com --token $SECRET  # upload to a live instance
voltro data import --target api --api-url https://api.example.com --token $SECRET --bundle-key backups/2026-07-01.vbundle  # instance pulls from storage (scale)
voltro data import ./out --assets            # also restore blobs
voltro data import ./out --no-verify         # skip checksum/row-count verification
voltro data import ./out --force             # import despite schema drift AND cross-dialect warnings
voltro data import ./out --tables users,teams   # load only these tables out of the bundle
voltro data import ./out --dry-run              # report what would move; write nothing

--dry-run and --tables work on BOTH targets, including --target api. On the api path they travel as x-import-dry-run / x-import-tables, and the response echoes { "dryRun": true, "wrote": false } so a preview is never mistaken for a write.

A dry run reaches every verdict a real run reaches — schema fit, cross-dialect portability, mode legality, the table selection — and stops at the first line that would write. It does NOT read per-table checksums (those stream during the load) and cannot see a conflict that depends on the target's current rows; the command says both out loud, because a preview over-read is worse than no preview.

--tables names tables the BUNDLE carries. A name it does not carry is refused, listing what it does — a silently-ignored table name is how a run scoped to one table writes the whole bundle.

#Every import option, in one place

Option	Default	What it does
--mode upsert|append|replace	upsert	How rows are written. See the table below.
--on-conflict skip|fail	skip	append only: what to do when the primary key already exists.
--atomic	on for replace, off otherwise	Wrap the whole table phase in ONE transaction — readers see the import all-or-nothing.
--no-atomic	—	Opt out of that. See the trade below.
--tables a,b	every table	Import only these tables from the bundle.
--dry-run	off	Run every pre-flight and report what WOULD move; write nothing.
--force	off	Proceed despite schema drift AND cross-dialect warnings.
--no-verify	off	Skip per-table checksum verification (direct target only).
--allow-live	off	Override the refusal to write directly into a database a live instance is serving (direct target only).
--assets	off	Restore blobs as well as rows.
--passphrase <s>	—	Decrypt an --encrypted bundle (direct target only; over the api the CLI decrypts before sending).
--target direct|api	direct	Write straight to the database, or through a running instance's admin endpoint.
--api-url <url> / --token <secret>	—	Required by --target api. The token must equal the instance's data-transfer secret.
--bundle-key <key>	—	--target api: have the instance PULL the archive from object storage instead of uploading it.
--chunk-size <mb>	16	--target api: bytes per upload chunk. Chunking engages automatically above one chunk.
--timeout <seconds>	none	--target api: give up waiting for the instance. Default is to wait as long as the import takes.
--json	off	Machine-readable result.

A flag this subcommand does not read is refused, not ignored — see below.

#A flag this command does not read is an ERROR

Every voltro data subcommand declares the flags it reads, per target, and refuses anything else instead of ignoring it:

✗ voltro data import: --no-verify is not read with --target api (it is a --target direct
  flag). Remove it, or change --target.
✗ voltro data export: --mode is not a `export` flag (it belongs to `voltro data import`).
✗ voltro data import: unknown flag --drynrun. Run `voltro data --help` for the flags this
  command reads.

The refusal happens before anything boots, so a mistyped flag costs you a message rather than a run. On a command whose job is moving data into a live system, silence is the wrong default: an accepted-and-ignored flag turns a typo into a no-op whose only evidence of working is that nothing complained.

Import is integrity-checked (each table's checksum + row count verified as it decodes; each asset re-hashed against its content address), applies tables in FK-parent-first order, and a resumed run skips already-applied tables via the ledger.

#Postgres targets bulk-load via COPY

On a postgres target, the direct import switches to COPY … FROM STDIN wherever plain-INSERT semantics provably hold: --mode replace (the tables were just truncated) and the default upsert into a table that is empty at import time — the fresh-target shape every cross-dialect migration (mysql → postgres, sqlite → postgres, …) lands in. Measured on a 7-column table (text/int/bool/jsonb/timestamptz) with 50 000 rows against a local postgres: row-by-row 12.8 s (~3.9 k rows/s) vs COPY 0.59 s (~84.6 k rows/s) — 21.7× faster. Your factor depends on row width and network latency; COPY's advantage grows with per-row round-trip cost.

Everything else keeps the per-row DataStore writes: upsert into a non-empty table (COPY cannot upsert), append (per-row conflict handling), --atomic (the COPY connection would sit outside the transaction), and every other dialect. A refused COPY batch (an FK the deferred pass repairs later, a value COPY text can't carry) is atomic — nothing landed — so the importer replays exactly that batch through the per-row path and continues; semantics are identical, only the speed differs.

#Schema-drift pre-flight

The question the pre-flight asks is "will the rows this bundle carries fit this target?" — not "are these two schemas identical". It compares the intersection: for every table the bundle carries, the columns and types must exist on the target, and a column the target REQUIRES (NOT NULL, no default) that the bundle carries no value for is refused too. On a problem it refuses, fail-closed, before any row lands:

✗ schema drift: refusing before the table phase.
    table 'items' is in the bundle but MISSING from the target — its rows have nowhere to go
    orders.total: type 'integer' (bundle) vs 'text' (target)
    users.region: the target requires it (NOT NULL, no default) and the bundle carries no
      value — every row of this table would fail

Every line is something that would break the load. Tables and columns the target has and the bundle does not are untouched by definition and are never reported — that is the normal shape of any cross-environment seed, and refusing on it would make --force the routine way to run an import and take the protection with it.

Both fingerprints are still reported (they are what you paste when asking for help), and an identical pair is a fast path that skips the comparison. But differing fingerprints are not on their own a refusal: a bundle's fingerprint covers its SOURCE schema regardless of export scope, so a one-table export out of a 75-table database carries the 75-table fingerprint.

--force downgrades the refusal to a loud warning and proceeds. Over the --target api path the same check runs on the instance against its declared schema and returns 409 schema drift with the fingerprints + diff; --force sends x-import-force: 1 to override.

The pre-flight only runs when the importer has a target schema (the CLI introspects it; the API endpoint uses the instance's declared schema). Importing into a fresh/empty database with no comparable schema simply skips the check.

#replace writes down what it is about to destroy

Before the first delete, a replace exports the target's current rows for exactly the tables it is going to empty, as an ordinary bundle beside yours:

rollback capture: 240172 row(s) across 75 table(s) → ./out.rollback-2026-08-21T09-10-11-000Z
  If this run does not finish, restore with:  voltro data import ./out.rollback-… --mode replace

It is on disk before anything is destroyed, so it does not depend on a transaction, or on the process being alive to roll one back. That distinction is the whole reason it exists: a deployment lost 240 172 rows to a replace whose api pod disappeared nine minutes in, and recovered from an export they had taken twenty minutes earlier out of habit. This is that habit, made into the tool's behaviour.

It is fail-closed. A capture that cannot be taken stops the import before it starts, and the target is untouched. A safety net you believe in and do not have is worse than none — the belief is what stops you taking your own export.

--no-rollback turns it off, and --rollback-dir <path> puts it somewhere else. It is only taken for replace: upsert and append do not destroy, so there is no moment where the old state has silently become unreachable.

Over --target api it goes to the instance's object storage, because there the process that would roll a transaction back IS the instance — a capture in the pod's filesystem would go away with the failure it exists for. Name a key:

voltro data import ./out --target api --api-url <url> --token $SECRET --mode replace --rollback-key backups/before.vbundle

The instance writes the capture there before the first delete, and refuses the run (409) if it has no storage configured — asking for a capture and being served without one is the answer that removes your own precaution while looking like agreement. A replace that names no key still runs, and says what it did not keep.

#replace loads somewhere else first

The target keeps its rows until the load stands. replace creates a staging table per table, loads the bundle into those, and then swaps the CONTENT across in one short transaction of server-side SQL. It says so when it does:

staging 114 table(s) before the swap — the target keeps its rows until the load stands.
  The destructive step is one server-side transaction at the end, not the whole load.

The difference is what a dead process costs. Loading straight into the target holds the destructive transaction open for the whole load — minutes, for a large bundle — and the target only survives because the database rolls that transaction back. With staging, a process that dies during the load leaves the target exactly as it was, because nothing has been deleted yet.

Not every run can take it, and a run that cannot says why rather than quietly taking the slower path:

• a store the framework cannot send DDL to (the in-memory store).
• a write recorder on any table in the set — rowHistoryPlugin({ timing: 'in-transaction' }) and friends. A recorder is keyed by table name, so a staged write would find none and the recorder would silently not run. Its promise is "if the change committed, the entry is there", so the run keeps the path that can keep it.

--no-atomic stages too, and that is where it changes the most. The flag exists for resumability on a large bundle, and it used to be the mode with the worst failure: the target emptied and partially refilled, in neither state. Staged, the ledger keeps its exact meaning — a recorded table is one fully loaded, it just lands in staging — while the target stays untouched until the swap. Resumable and all-or-nothing at once, which the two could not be before.

A resumed run continues from what it already staged rather than reloading it, and a run whose swap has not happened keeps its staged rows and says so:

the staged rows are KEPT so a re-run can continue from them rather than reload.
  If you are not going to re-run this bundle, drop them: voltro data clear-staging --yes

A failed swap names every offending row, not the first. Staging carries no foreign keys — a staged row whose parent has not been staged yet must not be refused — so a dangling reference surfaces at the swap, where the database reports one constraint. The importer then asks staging the same question and lists every row that fails it:

the swap could not run: 2 row(s) in the bundle reference a row the bundle does not carry.
    tasks.t_41: ownerId = "u_9" — no such row in users
    tasks.t_88: ownerId = "u_12" — no such row in users
  The target is UNCHANGED — the swap runs in one transaction and none of it committed.

#Staging tables a dead run left behind

A staged run RECORDS the scratch tables it creates, in the same _voltro_replace_in_progress table an interrupted destructive replace writes to — with one difference that matters: a staging record never refuses a boot. Nothing was destroyed, so there is nothing to refuse over. The boot reports instead:

staged data-import leftovers:
  - 3 staging table(s) from a `replace` over api, last active 74 minute(s) ago — DROPPED: the run
    is not resumable and has been silent long enough that nothing is loading into them.
  The target of a staged `replace` is untouched until one short swap at the end, so none of this is
  a reason to refuse the boot — it is a reason to know the disk is holding a copy of a bundle.

The run refreshes a heartbeat on that record every couple of seconds while rows land, which is what lets a boot tell the three cases apart:

what the record says	what the boot does
silent past the threshold, started without --no-atomic	drops the tables it names
still beating	leaves them — an import is loading into them right now, here or on another replica
started --no-atomic	leaves them — its staging IS the resume point

The threshold is 30 minutes by default. It is deliberately generous: the cost of collecting too early is that an in-flight import's swap fails with a missing table and you re-run it — the target is untouched either way — but the cost is still a re-run.

Declare a different one for a deployment whose imports routinely pause longer than that, waiting on an upstream export or a maintenance window:

// app.config.ts
export default {
  dataTransfer: {
    stagingStaleMinutes: 90,
  },
}

VOLTRO_STAGING_STALE_MINUTES overrides the declaration in turn — an operator acting on a running deployment outranks what the project declared. Note that the threshold decides only what a boot DROPS: leftover staging tables are named in the boot log either way.

What the boot does NOT collect, you can:

voltro data clear-staging --yes

It now labels each table with what its own run says, so a resume point is distinguishable from a leftover before you drop it:

2 staging table(s) from an earlier `--mode replace`:
    _voltro_staging_tasks — RESUMABLE: a `--no-atomic` re-run continues from it, last active 4 min ago
    _voltro_staging_notes — no run claims it (an orphan, or from before the marker)

A cycle in the bundle's foreign keys is detected before the load, not after it. The swap inserts parents first, so two tables referencing each other cannot both be satisfied by a bulk copy on postgres, sqlite or SQL Server — SET CONSTRAINTS ALL DEFERRED does not help, because postgres only defers a constraint declared DEFERRABLE. Such a run says so and takes the row-by-row path, whose deferred-FK pass exists for exactly that shape. A table referencing ITSELF is not a cycle: one statement carries the whole table.

#A replace does not write per-row history

Write recorders — rowHistoryPlugin({ timing: 'in-transaction' }) and anything else registered through the same seam — are suspended for a replace. A replace sets a state; it does not change rows, so a per-row history entry would describe something that did not happen. On a large bundle that is not a detail: one import wrote 242 950 history rows for a deployment, doubling the write load of their most expensive run.

The reasoning is not the cost. This path already writes through the raw store — no tenant scoping, no row filter, no audit() stamping — and the recorder fired anyway because it sits one layer below. Suspending it makes the layers agree.

upsert and append still record: those CHANGE existing state, which is what a recorder is for. Every suspended run says so, and the suspension is scoped to the run rather than the process, so requests served alongside it keep recording.

Instead of per-row history, the operation records itself. One row in _voltro_data_transfers per run — in either direction: the mode, the transport, the bundle, the source deployment's schema fingerprint, the counts — and the failure, for the run you are usually looking for. A trail that only records successes goes quiet exactly when it is needed.

It is best-effort, unlike the interrupted-replace marker: that one is a safety interlock and a run which cannot write it must not proceed, while this is history. A target whose schema is not migrated yet still imports, and says the trace could not be written.

#Every request fits a budget

Some environments cap a single request: a job runner that kills a client after ten minutes, an ingress with a 30-second ceiling, a CI step with a deadline. A whole transfer may still take an hour — it just has to do so as a series of short requests, each individually abandonable and individually retryable.

The rule the endpoints follow:

A request that carries bytes never runs a transfer. A request that starts a transfer never carries bytes.

# Nothing here may take longer than 30 seconds per request.
voltro data import ./bundle --target api --api-url https://app.example \
  --mode replace --max-request-seconds 30

# Or: start it and walk away. The outcome is NOT known when this returns.
voltro data import ./bundle --target api --api-url https://app.example --detach

Both staging areas — the uploaded bundle on its way in, the produced one on its way out — sit in the system temp directory by default and move with VOLTRO_IMPORT_UPLOAD_DIR / VOLTRO_EXPORT_ARTIFACT_DIR. Worth setting on a container: a bundle is the size your database is, and a default /tmp is frequently a small tmpfs.

--max-request-seconds is DECLARED rather than probed. The thing that kills a request is a policy on your side, and only you know it — in the one environment this was measured against, the ingress would happily hold a connection for an hour and the caller's own toolchain killed the client at ten minutes.

An import is three steps. The bundle goes up (in 16 MiB chunks when it is big, resumable, one plain request when it is small); import/start begins the run and answers as soon as the run is recorded; then the client watches the record. start is idempotent — retrying it under a budget returns the run that is already going rather than beginning a second destructive one.

An export is three steps too. Ask, wait for the record, then fetch the bytes in ranges from GET /_voltro/admin/export/download. The archive used to arrive in the response body, which made the request last as long as reading your whole database.

--detach returns once the run has started and says so plainly: exit 0 there means "it started", not "it worked". Attached, the exit code comes from the run's record — 0 finished, 1 failed, 2 still going when you stopped watching. Ctrl-C loses the watching and never the run.

#Watching a transfer you cannot see

Over --target api the run happens INSIDE the instance. Everything it decides — whether it stages, whether recorders are suspended, how far it has got — is printed in the pod's log, and someone reaching for --target api is by construction someone who cannot reach the database directly and usually cannot read that log either. The run is also deliberately decoupled from the caller: killing the client does not stop it, which is what keeps a dead client from leaving a half-emptied target. Both properties are right, and together they used to mean an operator could neither see the run nor learn how it ended.

Two things close that, and neither is a streamed response on the upload connection. That connection is the thing an operator is most likely to lose: a client killed by a job timeout while the import runs on inside the instance is the ordinary case, and a stream would go quiet at exactly the moment somebody needs to know what happened. A poll can be run from a different machine than the one that started the import.

Ask before you upload. A replace over the api asks the instance what it is going to do, before a byte of the bundle goes up. The instance answers from the same function the run itself calls, so the answer cannot drift from the run:

$ voltro data import ./bundle --target api --api-url https://app.example --mode replace
api replace: the instance WILL stage — 104 table(s) load into copies and the
  target keeps its rows until one short swap at the end. Interrupting the load leaves the target intact.
  importing — 41200 row(s) across 18 table(s) (staged; the target still holds its own rows)
  done — 228866 row(s) across 104 table(s)

With --bundle-key the client never holds the bundle, so it sends the key and the instance reads the table list out of the archive itself — the caller about to have an instance empty its own database is the last one who should be told to check a log they cannot read.

If it will not stage, the line says so and names the reason — a bundle table the target does not have, a foreign-key cycle, a dialect this build cannot stage on. That is the difference between "interrupting this is safe" and "interrupting this empties the target", which is exactly the decision an operator is making while they watch.

Read the run from anywhere. _voltro_data_transfers carries one row per run in EITHER direction: opened before the first table, advanced every couple of seconds as tables land, closed with the outcome — so polling it IS the progress feed:

$ voltro data transfers --target api --api-url https://app.example
1 import(s) IN FLIGHT — re-run this to watch the counters move
  2026-08-23T09:04:01.000Z · replace — started and never reported finishing · via api · staged (target untouched until the swap) · from /tmp/b
  2026-08-23T08:00:00.000Z · export all — 228866 row(s) across 104 table(s) · via api · from backups/nightly.vbundle

Same data over GET /_voltro/admin/transfers?limit=20, behind the same data-transfer secret (the row names bundles and schema fingerprints). It works from a different machine than the one that started the import, and through anything that forwards a GET.

The staged part is a separate claim from the preflight's, and both are needed. The preflight says what the instance WILL do; this says what it did. Without it the two could only be closed by reading the pod's log, which is the one place a --target api caller cannot reach.

#An interrupted replace cannot be silent

The capture only helps if somebody knows to reach for it. A half-replaced database is indistinguishable from an empty one from the inside — every table exists, every constraint holds, every query returns nothing without erroring — so a run that emptied a target and disappeared can be served over for hours before anyone asks the right question.

So a replace writes one row before the first delete and removes it after the last insert. Finding it at boot is a refusal, not a warning:

refusing to start: a destructive import did not finish.
  - a `replace` over api began emptying 114 table(s) 12 minute(s) ago and never reported finishing.
    The target's previous rows were captured first:
      ./out.rollback-2026-08-21T09-10-11-000Z
    Restore them with:  voltro data import ./out.rollback-… --mode replace

The row lives in the same transaction as the emptying, so it is present exactly when the emptying is: a run that rolls back cleanly takes the marker with it, and a boot over a database nothing happened to is not refused. A completed replace clears its own marker and any older one — so the recovery import both restores the data and silences the alarm, in one command.

Nothing expires. A half-replaced database does not become whole with time, so clearing it is a decision:

voltro data clear-replace-marker --yes

The devtools import panel takes the same two precautions as the command line — a capture before the first delete, and the marker — because a button is easier to press than a command is to type.

#Deferred-FK resolution

A row whose write fails on a foreign-key constraint — a forward reference from a live-consistency export, a genuine FK cycle between tables (which the exporter orders by breaking the closing edge), or an intra-table self-reference to a later row — does not fail the import. It is held and resolved after every table has streamed:

1. Retry to a fixpoint — forward references resolve once the later tables landed.
2. FK-shedding — rows still stuck are written with their FK-bearing columns set to NULL (possible wherever those columns are nullable), which breaks row cycles on every dialect without session-level constraint toggles.
3. Patch pass — shed rows are re-written with the full bundle row, restoring the FK values.

Anything still unresolvable — the parent row exists in neither the bundle nor the target, a NOT NULL FK cycle, or a row the target refuses for a reason of its own — fails with a typed RowsRefusedError. Held rows are the exception set, not the data set: memory is bounded by how many rows dangle at load time.

#Reading a refusal

The error carries three things, and they answer different questions:

Field	What it is
totalCount / primaryCount	how many rows were refused, and how many of those are the actual failures. A row is derived when one of its reference columns points at another row that also failed — it could not have landed whatever it contained, so it says nothing about itself. In an FK-dense bundle these dominate.
byTable	complete per-table counts (refused, primary), worst first.
rows	up to 20 refused rows — primary ones first — each with table, id, reason.

rows is capped and byTable is not, and that distinction is worth one sentence: tallying the tables in the printed list answers "how big is the cap", not "which tables failed". The CLI prints the byTable line above the list and says so when the list is truncated:

import refused 35 row(s), of which 35 are the actual failures — the rest could not land because a row they reference did not.
  by table: vacations 26, weeklyUpdates 9
  vacations v1  not-null constraint userId: the column requires a value [ER_BAD_NULL_ERROR/1048]
  …
  (the list above is capped at 20 of 35 — the counts by table are complete)

A refused import ends with a non-zero exit status and that report — not with a stack trace. A refusal is a condition with a named cause, not a framework defect, so it is not dressed as one. The report is the same on both transports: over --target api the refusal crosses as a 500 whose message embeds it, and the CLI renders that rather than printing the body raw.

RowsRefusedError keeps its tag in every mode, --atomic included. That is worth stating because it is the mode replace uses by default: the whole table phase runs in one transaction, and rolling that transaction back needs a rejection — but the rejection carries the typed error, not a rendering of it. So Effect.catchTag('RowsRefusedError', …) works on the default path, which is the one most likely to raise one.

One trap if you consume runImport yourself: what Effect.runPromise rejects with is a FiberFailure, and err._tag on one of those reads undefined however good the error inside is. Use Effect.catchTag on the effect, or asImportError(err) (exported from @voltro/data-transfer) on the rejection — reading the tag off the caught value takes the "not my error" branch every time.

A row's reason is stated as precisely as the driver allows, in three tiers:

1. the rule, in your schema's vocabulary — unique constraint PRIMARY: a row with this value already exists [ER_DUP_ENTRY/1062], foreign key teams_ibfk_1: the referenced row does not exist (import it first, or check the bundle's table order) [ER_NO_REFERENCED_ROW_2/1452]. The driver's own code is appended, because that is what you grep a log for;
2. the driver's own message + code, for anything it refused that is not one of the five integrity rules — Data too long for column 'v' at row 1 [ER_DATA_TOO_LONG/1406];
3. that there was nothing, when no driver detail is reachable at all: the database refused the write and the driver gave no detail — plus (via …), the chain of wrappers, whenever that chain has more than one layer to name. Read this tier as "not a constraint": a guard, a row filter, or a failure whose words did not survive. When it fires, the run also logs the full rendering of the first three such rows (table, primary key, and the whole error as it rendered) and counts the rest. That log line is never returned over the wire — it carries our stack frames, and on some engines a driver's sentence carries row data — so on --target api you read it in the instance's log.

Unlike the rpc wire, this string DOES include the driver's own words. The audience is the difference: it is read only by whoever ran voltro data import — the holder of the data-transfer secret, who supplied the rows and can export the whole target anyway.

#Write modes (--mode)

Mode	Behaviour	Use it for	Conflict
upsert (default)	INSERT-or-UPDATE by primary key	sync / idempotent re-import	overwrites per row
append	INSERT only	additive data (event log, new seed)	--on-conflict skip (default) or fail
replace	capture the target, empty it, then INSERT	full refresh — target ends up exactly the bundle	—

replace refuses a partial bundle (a subset / tenant / table scope): emptying would delete rows the bundle never carried. Re-export with full scope, or use upsert. (append --on-conflict fail throws on a duplicate primary key only where the store enforces the constraint — every SQL dialect does; the in-memory dev store overwrites.)

#How replace empties the target, and what it refuses

The bundle's tables are emptied as one unit, in one transaction, with referential integrity suspended for the duration — not table by table. Both halves matter:

• All or nothing — the emptying AND the load. --mode replace runs the whole table phase in ONE transaction by default, so a run that cannot finish leaves the target exactly as it found it. That default is a correction: the guarantee used to cover only the emptying, and a replace that died partway through the load left the target emptied of its old rows and holding part of the new ones. There is no useful state for a replace to stop in, which is why it is the default rather than a flag you have to know about.

  One transaction also closes a window that is easy to miss. Between the delete and the load the target is empty, and if the database is being served, the application writes into that gap — a row it creates on demand is then a primary-key conflict against the same row arriving from the bundle. With one transaction the concurrent writer waits instead of racing.

  --no-atomic opts out, and the trade is real: every write to those tables waits for the load, so on a bundle that takes minutes, so does the wait. On postgres it also re-enables the bulk COPY loader, which cannot join a transaction it does not own — an atomic run says so once rather than being quietly slower.

• No ordering can replace the suspension. MySQL, MariaDB and SQL Server check a foreign key as each row is deleted, so a table that references itself cannot be emptied at all by ordering tables — the conflict is between two rows of one of them. createdBy → actors on the actors table is exactly that shape, and it is what an audit mixin on an actor table produces. (Postgres needs no suspension: a multi-table TRUNCATE covers the whole set at once.)

What it will not do is reach outside the bundle. If a table the bundle does not carry holds rows that reference one it does, replace refuses before deleting anything and names them:

replace cannot empty this target: 1 table(s) OUTSIDE the bundle hold rows that reference
tables INSIDE it. Emptying the bundle's tables would leave those rows pointing at nothing,
and the bundle cannot restore them.
  webhookDeliveries (4021 rows): createdBy → actors, updatedBy → actors
Nothing has been deleted. Either re-export with a scope that includes those tables, or use
--mode upsert if those rows are meant to survive.

An empty table outside the bundle blocks nothing — a schema always carries tables an environment has never written to.

One consequence of emptying in bulk: the delete itself emits no change events (one per row is not affordable at bundle scale). Over --target api that is covered — an import through the instance's own process asks every live subscription to re-read once it lands, the same coarse refresh the framework uses after a broadcast gap. Over --target direct there is no live instance to tell, which is the whole reason that path is guarded.

#Which mode for which job

You want	Mode	Why
Keep a staging database in step with a production export, re-runnable	upsert	Idempotent per row: run it as often as you like, the result is the same.
Add an event log / new seed data without touching what is there	append --on-conflict skip	Insert-only, and a row that already exists is left alone rather than overwritten.
Catch a duplicate instead of silently skipping it	append --on-conflict fail	The import stops and names the row. Use when a duplicate means the bundle is wrong.
Make a target become exactly a bundle — replace an environment	replace (full-scope bundle)	Empties the bundle's tables first, so rows the bundle does not carry are gone.
The same, against a database that is being served right now	replace --atomic	One transaction: readers see the old state until commit, the new one after. Never a half-loaded table.
The same, against a live instance, honouring app invariants	replace --target api	Runs in the instance's process: validation, mixins, field encryption, hooks, and live subscriptions re-read afterwards.

Two things replace will refuse, both before writing anything: a partial bundle (subset / tenant / table scope), because emptying would delete rows the bundle never carried; and a target where a table outside the bundle holds rows referencing one inside it, because those rows cannot be put back. Both refusals name what to do instead.

#Recipe: replace a dev environment from your local database

# 1. Full-scope export of the source, locally.
voltro data export ./dev-refresh

# 2. Preview against the target first — every pre-flight runs, nothing is written.
voltro data import ./dev-refresh --target api \
  --api-url https://dev.example --token "$VOLTRO_DATA_TRANSFER_SECRET" \
  --mode replace --dry-run

# 3. Do it. A big bundle chunks itself; a failed run can be re-run and resumes.
voltro data import ./dev-refresh --target api \
  --api-url https://dev.example --token "$VOLTRO_DATA_TRANSFER_SECRET" \
  --mode replace --atomic

If step 3 dies halfway — a dropped connection, a laptop closing — run exactly the same command again. The upload continues where it stopped, and the import is a single all-or-nothing step, so the target is either the old state or the new one.

#Importing into a LIVE instance

A plain import connects straight to the database (not through the running app), so it is an uncoordinated concurrent writer — readers can see partial state, the reactive layer either storms (CDC dialects) or goes stale (others), and rows race with live writes. So import / restore refuse by default when a live instance is detected (via the local runtime registry, or an explicit --api-url probe). Two ways forward:

Target	How	Guarantees	Use for
direct (default)	writes straight to the DB	none while live — guarded; pass --allow-live to override	a stopped target, a replica, a dev/staging DB not serving traffic
direct + --atomic (the default for replace)	wraps empty+load in ONE transaction	MVCC readers see the import all-or-nothing (old until commit, new after) — the live-safe replace	replacing a live target's data with no partial-state window
in-process (--target api)	voltro data import <dir> --target api --api-url <url> --token <secret> uploads the packed bundle (or, with --bundle-key, has the instance pull it from storage) to the instance's secret-gated admin endpoint, which imports it in its own process through its store	full app pipeline — validation, mixins, field encryption, hooks — AND automatic reactivity (in-process writes emit change events, so subscriptions update; no separate resync)	a live merge (incl. PROD) that must honour app invariants

--atomic is what replace does by default (see above): one transaction, so live reads never see a half-loaded table. (It holds a write transaction for the load duration — writes to those tables block, readers don't. A shadow-table rename would shorten that lock window, but for a full replace concurrent writes are discarded on swap anyway, so it isn't the default.)

#The --target api data-transfer endpoints (prod-safe)

--target api moves data in/out of a running production instance without direct DB access. It does NOT reuse the dev data-viewer surface — two dedicated endpoints, POST /_voltro/admin/export and POST /_voltro/admin/import, both built to be safe on prod and both gated by the same secret:

• Secure by default — no secret, no endpoints. The instance mounts BOTH routes only when a data-transfer secret is configured: VOLTRO_DATA_TRANSFER_SECRET=<≥16 chars> (or serveApi({ dataTransferSecret })). Unset (or shorter than 16 chars) → the routes return 404. There is no "on" switch that leaves them open.
• Secret gate, constant-time. Every request must present that secret as a Bearer token (Authorization: Bearer <secret>), compared in constant time over SHA-256 digests (neither length nor content leaks via timing). The gate is a framework-owned secret independent of app RBAC — enabling data transfer can never accidentally ride on a user role. The CLI sends it via --token / VOLTRO_DATA_TRANSFER_SECRET (it must equal the server's secret). Wrong/absent token → 401.
• One secret, both directions. The same secret gates export (read) and import (write). If you need to grant export without import (e.g. a backup job that must never overwrite), split it into a per-operation credential deliberately — the default is one credential for the whole surface.

#A big bundle goes up in chunks, and resumes (--chunk-size)

A bundle bigger than one chunk is uploaded as a series of short requests instead of one long one, and the switch is automatic: the packer's stream is buffered one chunk ahead, so a bundle that fits inside that buffer is sent exactly as before — one request, no protocol — and a bigger one chunks itself. You never have to know in advance which table is the big one.

Each chunk is its own request, so a proxy body cap or an ingress read timeout has nothing large to choke on. The import still runs once, at the end, over the whole bundle — same modes, same deferred-FK repair, same all-or-nothing emptying. Only the transport changes.

Resume is real. The upload id is derived from the bundle itself, so re-running the same command after a failure asks the instance how far it got and continues from there:

api import: resuming a chunked upload the instance already holds { bytes: 50331648 }
api import: uploaded in chunks { chunks: 7, bytes: 62914560, resumedFrom: 50331648 }

That is safe because packBundle over a bundle directory is byte-identical across runs, and because each chunk carries the bundle's key (the hash of its manifest). Uploading a different bundle under the same id is refused rather than spliced into the partial one, and a chunk that does not start exactly where the instance left off is refused with the offset it does expect.

--chunk-size <mb> overrides the default of 16 MiB. Unfinished uploads are discarded by the instance after 24 hours.

#The chunk protocol, for when you are reading proxy logs

Worth knowing if an ingress sits in the way, because these requests are what it will show you. All of them are POST to the same /_voltro/admin/import path, gated by the same Bearer secret:

Request	Headers	Answer
probe	x-import-upload-id, x-import-probe: 1, empty body	200 {uploadId, bytes} — how much the instance already holds
chunk	x-import-upload-id, x-import-upload-key, x-import-chunk-offset	202 {uploadId, bytes} — accepted, nothing imported yet
final chunk	the same plus x-import-chunk-final: 1	200 with the import's own result — this is the long one

A 409 means the instance refused the chunk and says which of three things happened: offset-mismatch (with the expectedOffset to continue from), key-mismatch (a different bundle under this upload id — use a new id), or bad-id. None of them is retryable by simply repeating the request, which is why each one names the fix.

Only the FINAL request runs the import, so only that one is long. If it is the request your proxy times out on, that is the one to raise proxy_read_timeout for — or use --bundle-key and have the instance pull the archive from object storage instead of receiving it.

#How long an --target api call may take (--timeout)

Both api-target calls wait as long as the instance needs. There is no default deadline, and that is deliberate: the response arrives only when the import (or export) has finished, so any fixed bound is really a bound on the size of your database. A full bundle of a grown database routinely takes longer than five minutes to apply.

Pass --timeout <seconds> when you want one. If it is hit, the message says what it means — the upload finished long ago and the instance is very probably still importing:

voltro data import ./out --target api --api-url https://api.example --token $SECRET --timeout 900

Note the asymmetry when a deadline is hit: re-running an upsert after the first run has finished is safe (it is idempotent); re-running a replace while the first is still mid-flight would empty the target under it. Check the instance's log before deciding — the message says so too.

An instance that is not answering at all is a different message, and it names the instance rather than a database:

voltro: api import: the instance at api.example.com is not answering (ETIMEDOUT).
  That address came from --api-url (or VOLTRO_API_URL) — it is the running instance, not the database.
  Nothing was sent, so nothing was imported. …

Worth one line because the alternative was measured: a connect failure carries an address, a port and an errno, which is the same shape a database driver's carries — so without this the api host was reported as an unreachable database, attributed to DB_URL, a variable not in play on that run. Every word after the address was wrong, and the address being right is what made it convincing.

Import (POST /_voltro/admin/import) applies a bundle in the instance's own process through its store — full app pipeline (validation, mixins, field encryption, hooks) AND automatic reactivity. Two transports, neither needs a server-readable path:

• upload (default) — the CLI packs the bundle and uploads the bytes; the server unpacks to its own temp dir, imports in-process, cleans up. Buffered → small/moderate bundles.
• storage-pull (--bundle-key <key>) — the CLI sends only { "bundleKey": "<key>" }; the instance streams that archive from its configured object storage (S3 / Azure / MinIO / filesystem, resolved from the storage env — STORAGE_PROVIDER, S3_*, AZURE_*, …), no body buffering → arbitrarily large bundles.

Export (POST /_voltro/admin/export) reads the instance's data in-process (so field-decryption + hooks apply) and packs a bundle. ⚠️ This is a data-exfiltration surface — it can read all prod data — which is exactly why it sits behind the same off-by-default secret. It is also the right place for source-side masking on a live box: name a server-side profile and its scope/subset/masking apply before any byte leaves the instance (fail-closed — see the prod→dev section below). Two transports:

• download (default) — the response body is the packed bundle; the CLI writes it to <outDir>. Buffered → small/moderate exports.
• storage-push (--bundle-key <key>) — the instance exports straight to its object storage under that key and returns { bundleKey, tables, rows }, no response buffering → the scale path. Then pull it elsewhere (e.g. import --bundle-key).

The masking policy for a --target api export is named by --profile <name> and loaded on the server (data-profiles/<name>.ts) — version-controlled on the instance, never supplied by the client. No profile → a raw export (the secret-holder is trusted to read prod).

Assets over the API — full parity with the file path. With --assets, an API export streams the instance's blobs from its own object storage into the bundle (single-pass, nothing staged); an API import streams the bundle's blobs straight to the destination instance's storage as the archive arrives. So voltro data export prod.vbundle --assets (from a running instance via --target api) → voltro data import prod.vbundle --target api --api-url <dev> moves data and blobs at any size. (Uploading a directory bundle that has materialised assets over --target api transfers data only — pack a single-file .vbundle for assets; the CLI warns if you try.)

• Audited — every export/import logs the transport, table + row counts, and (export) how many columns were masked.

For a live merge (import) that must run app invariants + drive reactivity, this in-process path needs no quiesce/resync — in-process writes are reactive automatically.

#Cross-dialect

You can import a bundle into a different dialect than it came from — but only for the framework's portable DSL types. A cross-dialect import runs a portability lint first:

• Blocked (refused unless --force): raw() columns (verbatim source-dialect SQL), and vector columns targeting a dialect with no vector type.
• Warned (imported, represented differently): array / interval on a non-postgres target.
• Portable everywhere: text, integer, real, boolean, timestamp, date, json, bytes, reference, enum, id.

The lint refuses loudly rather than silently coercing — a blocked import tells you exactly which columns are the problem.

#Reliability: chunking, retry, resume

A production export can run for hours over millions of rows and gigabytes of blobs. It is built so a dropped connection, a pool blip, or an outright crash never means starting over — and so a huge table never blows up memory or knocks the source database over.

#Chunking — flat memory, gentle on the DB

Every table is read with a keyset cursor, not OFFSET: WHERE pk > :last ORDER BY pk LIMIT n (default 1000 rows/page, tune with chunkSize). Consequences:

• Flat memory — one bounded page is in memory at a time, regardless of table size (a 20 M-row table streams in 1000-row pages).
• O(1) per page on the pk index — OFFSET n re-scans and skips n rows every page (O(n²) over a full walk); keyset reads each row exactly once.
• Backpressure — a page is fetched only when the sink (encode → compress → disk) is ready to take it, so a slow disk throttles the DB reads instead of overrunning memory or hammering the server.

live consistency keeps each read short (no long-held transaction → vacuum-friendly); snapshot trades that for one pinned transaction held for the export's duration (see above).

#Retry — a blip doesn't kill the run

Each page read is retried on a transient failure (dropped connection, pooled-backend hiccup) with exponential backoff + jitter, up to 5 attempts by default (retryTimes). The retry is per page, and the keyset cursor is preserved — so a reconnected page resumes at exactly the row it stopped on, never re-emitting or skipping. A genuinely-broken read still surfaces after the attempts are exhausted rather than hanging.

#Resume — re-run and it continues

Both export and import checkpoint into a small .ledger.json and can be re-run to continue where they stopped:

• Export writes each table with an atomic temp-file + rename, and records the table (and each asset) in the ledger only once it's fully written. A crash mid-table leaves no half-written file (the temp is discarded); re-running skips the completed tables/assets and redoes only the unfinished one. Assets are content-addressed, so resume is per-asset — a blob whose hash is already in the bundle is skipped.
• Import keeps its own ledger (.import.ledger.json) and skips already-applied tables on a re-run. Correctness never depends on the ledger, though: every row is upserted by primary key, so redoing an in-flight table is always safe, and each table's checksum + row count and each asset's hash are verified as they load — a truncated or corrupted bundle fails loudly instead of importing garbage.

Net: interrupt an export or import at any point — network drop, Ctrl-C, OOM-killed pod — and re-running the same command finishes the job without duplicating work or corrupting the target.

A re-run of a COMPLETED import writes nothing, and says so. The ledger lives in the bundle directory, so re-importing a bundle whose tables are all recorded skips every one of them. That is resume working — but the returned report counts the BUNDLE's rows either way, so import complete … 10593 rows would otherwise print over a target you had just truncated. The import warns instead:

⚠ resume: 5 of 5 table(s) were already applied by an earlier run of THIS bundle
  directory, so this run wrote NO rows for them — that is every table in the
  bundle, so nothing was written at all. Tables: tenants, teams, projects,
  actors, auditLogs. Delete ./out/.import.ledger.json to force a full re-import.

The --target api path is unaffected: the instance unpacks each upload into a fresh temp directory, so it never carries a ledger between runs.

#What the summary line counts

import complete … rows: N counts the rows this run wrote, not the rows the bundle carries. The two differ more often than you would think, and the case where they differ most used to read as a success:

import complete — 0 rows written, every table was already applied by an earlier run
of this bundle directory { rows: 0, carried: 242950, skippedTables: 114, skippedRows: 242950 }

The resume ledger lives inside the bundle directory, so copying a bundle copies its ledger, and the copy then imports nothing — correctly, and with a warning that says so and names the file to delete. But the warning is not the last line, and an operator piping the output through tail -1 sees only the last line. So the last line now tells the truth on its own: rows is what was written, carried is what the bundle holds, and skippedRows is what an earlier run had already applied.

#Progress & observability

A multi-hour job is not a black box. Both pipelines signal per table — never per row, so the reporting never slows the hot path:

• voltro data export|import prints a line per finished table — export [7/23] users · 1.2k rows — so you watch the run progress instead of staring at a silent terminal. A ledger-resumed table prints once with its recorded count.
• Programmatically, runExport / runImport take an optional onProgress callback. It fires a start then a done event per table, in FK-parent-first order, carrying the table name, its 0-based index, the tableCount, rowsDone, and total (the table's row count when known ahead of time — import reads it from the manifest; a live export learns it only once the table drains). A throwing progress renderer never aborts the job.
• Traces + metrics are always on, no wiring. Each table runs inside an Effect.withSpan('data-transfer.export.table' | 'data-transfer.import.table') (attributes table / index / phase), so if the app has tracing enabled the export/import shows per-table phase timing in the trace. Two metrics record throughput: voltro_data_transfer_rows (counter, tagged by phase + table) and voltro_data_transfer_table_seconds (histogram, tagged by phase).

import { runImport, type ProgressEvent } from '@voltro/data-transfer'

yield* runImport({
  store: target,
  bundleDir: './backup',
  onProgress: (e: ProgressEvent) => {
    if (e.event === 'done') console.log(`[${e.index + 1}/${e.tableCount}] ${e.table}: ${e.rowsDone} rows`)
  },
})

#Typed errors — CLI-catchable AND rpc-declarable

Every pipeline failure is a tagged error, caught by tag with Effect.catchTag(...). The errors that appear on the runExport / runImport error channels — the wire errors, and therefore exactly what the --target api admin endpoints surface — are Schema.TaggedError, so a handler can declare them on an rpc procedure's error: schema and the rpc encoder marshals them across the wire round-trip-safely (no hand-rolled JSON per tag):

BundleError, CodecError, IntegrityError, CrossDialectError, ImportModeError, RowsRefusedError, SchemaDriftError, MaskingError, ScopeError.

The two internal errors — NativeToolError (native backup/restore) and CompressionError (folded into BundleError by the pipelines) — never cross the wire, so they stay plain Data.TaggedError: still catchable by tag, just no Schema surface.

import { Effect } from 'effect'
import { runImport, type SchemaDriftError } from '@voltro/data-transfer'

yield* runImport({ store: target, bundleDir: './backup', targetSnapshot }).pipe(
  Effect.catchTag('SchemaDriftError', (e: SchemaDriftError) => Effect.log(`refusing: schema drifted — ${e.diff.join('; ')}`)),
)

#Native backup / restore

voltro data backup ./backups/2026-07-01              # pg_dump --format=custom / mariadb-dump --single-transaction / …
voltro data backup ./backups/2026-07-01 --assets     # rows AND the stored blobs
voltro data restore ./backups/2026-07-01 --assets    # pg_restore / mariadb / … + the blobs

These shell out to the vendor tools resolved from your DB_DIALECT + connection env. They produce a dialect-native artifact (db.dump, db.sql, db.sqlite, db.bacpac) that is lossless and point-in-time consistent for same-dialect restore — the right tool for disaster recovery. Secrets are passed via the tools' environment variables (PGPASSWORD, MYSQL_PWD), never on the command line, where the tool supports it. The named tool must be installed and on PATH.

On mariadb the MariaDB-named binaries (mariadb-dump, mariadb) are preferred and Oracle's (mysqldump, mysql) are the fallback — with the reason carried into the failure, because the error that fallback produces (Unknown table 'COLUMN_STATISTICS' in information_schema, 1109) names a table nobody asked for. Two things worth knowing before you go and install the MariaDB client package:

• Do not reach for --column-statistics=0. That flag does not exist on mariadb-dump, so it patches the wrong client and breaks the right one.
• A MariaDB 12.x client requires TLS by default. Running it by hand against a server without TLS fails with TLS/SSL error: SSL is required (2026) and needs --skip-ssl. These commands are not affected — it is the first manual call after installing that trips.

#--assets — the blobs are not in the dump

A vendor dump contains rows. Your blobs are in object storage, and no pg_dump has ever seen them. So a rows-only backup restores a database whose rows reference objects that are not there — and the reference and the object are checked at different times, which is why that state is discovered by a user, months later, rather than by the restore.

The backend it reads is the one your app configured — storagePlugin({ provider: s3(…) }) if you installed it, the STORAGE_* env otherwise. (The commands used to resolve the env default unconditionally, so an app that configured its provider in code had its export, import and backup reading a different backend than the rest of it.)

--assets captures them alongside the dump, through the same content-addressed pipeline voltro data export --assets uses: each blob is streamed (never buffered whole), stored under assets/<sha256> so identical content is stored once, and listed in assets/index.ndjson. restore --assets streams them back and re-hashes on the way, so a corrupted artifact can never silently overwrite good bytes.

backups/2026-07-01/
├─ db.dump                       # the vendor artifact (rows)
├─ voltro-backup-stamp.json      # provenance, incl. what --assets captured
├─ assets/index.ndjson           # key → sha256 → size → contentType
└─ assets/<sha256>               # the blob bodies, deduped by content

Three refusals, each for a belief that is otherwise acted on silently:

• backup --assets with no storage provider configured → refused. There is nothing to capture, and a flag that is accepted and ignored lets you build a rollback story on an artifact that does not contain what you asked for. "Configured" means one of the two things a person actually did: installed storagePlugin(...), or set STORAGE_PROVIDER. An in-memory provider nobody asked for is not a decision — and until recently it was what this check saw, which is why the refusal never fired and --assets wrote artifacts stamped as carrying blobs that held none.
• restore --assets on a rows-only backup → refused. You believe the blobs are in there. Restoring the rows anyway produces exactly the dangling state this exists to prevent.
• restore without --assets on a backup that HAS them → warned, not refused. Restoring rows without blobs is legitimate (a schema drill, a lower environment), and refusing it would push people at --force.

A reference the provider cannot resolve is reported, not fatal. A row in _voltro_storage_refs can point at an object that was deleted, or that never arrived because an earlier import ran without --assets. That is a fact about your data, and no backup can put back bytes that are not there — so the capture records the key, steps over it, and the run says how many:

warn  177 of 178 blob reference(s) point at objects the storage provider does not have;
      they are NOT in this backup and no restore can bring them back.

Aborting on the first one made --assets unusable for exactly the deployment that needed it: 178 references, one resolvable, and the run stopped at the second — leaving an assets/ directory with a single blob, no voltro-backup-stamp.json (the writer never got that far), and nothing anywhere saying 177 objects had been skipped. The stamp is now written on every path, including the one where the asset phase fails, because it describes the dump and the dump is already on disk and correct. Without it, restore greeted an artifact this tool had written minutes earlier with "an older/handmade backup. Cannot verify dialect or schema version."

Only a genuine not found is treated this way. A 403 from a rotated credential or a 5xx from a backend outage still fails the capture — calling those "the object is gone" would turn a recoverable outage into a backup that quietly contains nothing.

Three numbers, because they answer three questions. _voltro_storage_refs holds one row per reference, several of which legitimately name one key, and the content-addressed store keeps one body per distinct object:

57 reference(s) → 16 key(s) → 16 object(s), 65476 byte(s) under assets/

The stamp carries all three (references, count, objects, with totalBytes and objectBytes beside them). It used to carry only the reference count under the name count, so a stamp read 57 over a directory holding 16 files — and anyone answering "are all the blobs there?" after a restore compared the two and found a 3.5× gap that was not one. A key named by several references is also fetched once now, rather than downloaded and hashed once per row.

Resume is per blob key, so re-running a --assets capture that was interrupted transfers only what is missing. The dump itself has no resume — a vendor artifact is one opaque file with no offset to restart from. If you need a resumable, chunkable, observable transfer, that is the logical path (export / import), and it is why the logical path exists.

#The stamp's skew warning compares the backup against the TARGET

restore reads voltro-backup-stamp.json before touching anything and warns when the backup's schema fingerprint differs from the target's. That warning used to say the difference was against "what this code declares", and it was not — the value it compares against is the target database's live schema, read by introspection at restore time. Bringing a target to the backup's shape makes the warning disappear while the declared fingerprint is a third value entirely, which is how the mislabel was caught. The comparison was always the useful one; only the sentence was wrong, and it sent readers looking for a code change where a database differed.

The same distinction shows up in voltro db plan, which prints live … · declared … rather than from → to for the same reason: a hash of a live database never equals the hash of the declaration it came from. Introspection cannot recover everything a declaration carries — generated expressions, maxLength, sensitivity markers — so the two are not comparable and are not meant to match. The plan's operation list is what says whether they agree; 0 operations under two different fingerprints means they do.

#A restore that is interrupted refuses the next boot

restore writes one row into _voltro_replace_in_progress before the first destructive statement and removes it after the last write — the blobs included. Its presence at boot is a refusal naming the artifact that was going in.

This is the counterpart to --allow-live, and it guards from the other side: --allow-live asks you not to restore over a running instance, and this says this database is mid-restore, do not serve it. A half-restored database looks exactly like a normal one from the inside — every query answers, nothing errors.

The restore artifact can erase the marker, and this table said otherwise. It read "postgres, mysql, mariadb: drops only the objects the dump names — the marker survives". The reasoning is right and the premise was wrong: a native dump names the whole database, _voltro_replace_in_progress included, and a mysql-family restore writes DROP TABLE IF EXISTS in front of each table. The table sorts early, so the guard was removed near the start of the window it covers. Measured downstream: one row before the restore, zero after, twice.

Two changes, covering different dumps:

• A backup taken by voltro data backup excludes the marker table (--exclude-table / --ignore-table). It can no longer carry the thing that erases the guard on the way back in.
• restore writes the marker back after the tool exits, on the failing path as well as the succeeding one. That covers dumps taken before this version and dumps made by hand. If it was removed and rewritten you get a warning saying so; if it could not be rewritten you get an error, because the guard is then off for that run and nothing will stop the next boot.

A restore that cannot write the marker at all is refused. Two things can prevent it — the bookkeeping store will not open (wrong credentials, an unreachable database, a missing env var, no app.config.ts from here), or the table is not there yet — and both mean the same thing to you: this restore would run with no guard. The refusal names which one it was:

✗ refusing to restore: the in-progress marker cannot be written.
    reason: bookkeeping is unavailable: connect ECONNREFUSED 127.0.0.1:5432

This used to be a silent hole rather than a refusal, and worse than silent. The failure to open the store was caught and discarded, and the discarded value guarded every branch below it — including the refusal that would have reported the guard missing. So a restore ran on and, over a database with zero marker rows, printed "the next boot will REFUSE, by design". The next boot did not refuse, and voltro data clear-replace-marker had nothing to clear. A restore is the operation you run against a target that is already unwell, so the precaution was falling away exactly when it was needed.

--no-marker restores without the guard, deliberately. It warns every time and names the reason the marker was unavailable. It exists because the accidental way did: if going unguarded is ever right, it should be something you typed.

dialect	shape	effect
postgres, mysql, mariadb	the dump names the whole database, so the restore drops the marker table too	our backups exclude it; for any other dump the marker is written back after the tool exits
sqlite, turso	whole-file replacement — made atomic (temp file + rename)	there is no half-restored state to catch; a killed restore leaves the live file untouched
mssql	sqlpackage /Action:Import replaces the database	a failed import is the one case not covered here — verify with --drill

Clear a marker deliberately with voltro data clear-replace-marker --yes once you have decided the current state is correct.

_voltro_data_transfers is excluded for the same reason, one table over. The restore opens its own run row there before the tool starts; a dump that carried the table dropped it mid-flight, and the update recording the outcome then wrote into a table that no longer held the row. The visible result was that a failed native restore did not appear in voltro data transfers at all — only the backup row the dump had brought over from the source database. The command that answers "did the restore finish" could not see the run asking.

Exactly those two tables are excluded, and the line is deliberate: a native restore into the same deployment should bring the migration ledger, the stored plans, the CDC offsets and the schedule claims — they describe the data being restored. These two describe the restore, and a record of an operation must not be overwritten by the operation it records.

#Both directions are in the history

backup and restore write a row to the same _voltro_data_transfers record import and export use, so voltro data transfers answers "did last night's backup finish" from the instance that ran it:

2026-08-23T02:00:00.000Z · backup native postgres + assets — finished (412 blob(s)) · via cli · from ./backups/2026-08-23/db.dump
2026-08-22T09:14:02.000Z · FAILED restore native mariadb — mariadb-dump: exited with code 2 · via cli · from ./backups/2026-08-21/db.sql

A native run reports blobs, not rows: the vendor tool reports no row count we can trust, and printing 0 row(s) over a pg_dump that worked would be a measurement, and a wrong one. A target with no _voltro_data_transfers table still gets its backup — the closing line says it was not recorded, rather than implying it was.

#The provenance stamp — a restore that refuses the wrong DB

A native dump is opaque: it doesn't say which dialect made it, which schema shape it carries, or when. backup writes a sidecar voltro-backup-stamp.json next to the artifact recording exactly that — dialect, the @voltro/cli version, the timestamp, and two schema fingerprints.

Two, because they are different facts and only one of them is a claim about the artifact:

• schemaFingerprint — the source database's whole live schema at backup time. This is what the skew warning below compares against a target.
• dumpFingerprint — the schema the artifact carries: that same snapshot minus the tables the dump excludes. On postgres and the mysql family those are _voltro_replace_in_progress and _voltro_data_transfers (see above); on sqlite, turso and mssql nothing is excluded and the two values are equal.

The distinction is not bookkeeping. voltro data backup opens its own run row in _voltro_data_transfers before it dumps, so on any database the framework has run against, the artifact is two tables short of the live schema it was taken from. Anything comparing a restored schema against a stamped one has to compare against dumpFingerprint — the drill did not, and failed every healthy backup with "the artifact is inconsistent."

restore reads the stamp before touching the DB and acts on two failures that are otherwise silent until they corrupt:

• Cross-dialect restore → refused. Restoring a postgres dump while DB_DIALECT=mysql is never valid; it stops with an error instead of half-loading. Override with --force only if you genuinely know better.
• Schema/code skew → warned. If the backup's schema fingerprint differs from what the running code declares, restore prints a warning to run voltro db apply afterwards — the dump's shape predates (or postdates) this deploy's code. (Production boot already refuses on a fingerprint mismatch; the stamp surfaces it at restore time, before the boot.)

A backup with no stamp (older, or hand-made) restores with a caution rather than a hard stop.

#The restore drill — prove the backup, don't assume it

voltro data restore ./backups/2026-07-01 --drill --drill-url postgres://…/scratch
# or set DRILL_DB_URL and just: voltro data restore ./backups/2026-07-01 --drill

--drill restores the artifact into a throwaway database (from --drill-url / DRILL_DB_URL) and verifies it — without ever touching the live DB. It refuses a drill target that resolves to your live connection (a drill that --cleans production is the disaster it exists to rehearse against). After the restore it introspects the throwaway DB and probes its migration ledger:

• zero tables restored → FAIL (the dump is empty or unreadable — this backup would not recover you),
• schema fingerprint disagrees with the stamp's dumpFingerprint → FAIL (the restore didn't reproduce what was backed up),
• _voltro_migration_plans restored EMPTY → FAIL (see below),
• tables + matching fingerprint + a populated or absent ledger → PASS.

A stamp too old to carry a dumpFingerprint gives a PASS (partial) that says the shape could not be cross-checked. It does not fall back to schemaFingerprint: that is the comparison that fails a healthy backup, and a check that is red on every real input gets switched off — taking its genuine failures with it.

It exits non-zero on any FAIL, so a scheduled CI job turns a silently-broken backup into a red build. Run it against your latest artifact on a cron — a backup you've never restored is a hypothesis, and this is how you keep it a fact.

#The ledger check — the one thing a schema comparison cannot see

A fingerprint answers "is the shape right?". A drill's real question is "would my app come up against this?", and the gap between them is content — a framework table that restored with the right columns and the wrong rows.

voltro serve's boot gate reads the newest row of _voltro_migration_plans and refuses with prod-mismatch when there is none. So a ledger table that restores with exactly the right columns and zero rows is a database no source tree can boot, and its schema fingerprint is identical to a healthy one's. The drill fails that, and names it:

FAIL — restored 30 table(s) with the right shape, but `_voltro_migration_plans`
       came back EMPTY.
       `voltro serve` reads the newest row of that table as its boot gate and
       refuses with `prod-mismatch` when there is none.

A restored database with no ledger table at all is not a voltro-managed schema (a hand-made dump, someone else's database) — the drill says so and claims nothing about booting it, rather than failing it.

What the drill deliberately does not judge is a ledger whose fingerprint differs from what your code declares. It has no way to know which commit you will deploy next to this database, and voltro db apply clears that state anyway; failing a backup for it would make the drill red for a reason that is not about the backup.

#Why there is no full app boot

Booting a real app against the restored database sounds like the stronger check, and it would be a weaker one. There is no app in the drill's path — it would have to boot a fixture, and a fixture booting says nothing about whether your app boots. It moves the drill from "proves your backup" to "proves our fixture" while reading as the bigger claim.

The part worth having does not need a process: the boot gate is a comparison, not a startup sequence, so the one boot-fatal condition that holds regardless of which code you deploy is reachable with a SELECT. That is the ledger check above.

#Point-in-time recovery (PITR) is your database's job, not the framework's

backup is a point-in-time snapshot. "Restore to 14:32, just before the bad deploy" (PITR) needs continuous WAL/binlog archiving, which lives at the database/provider layer — pg's archive_command + a base backup (pgBackRest / WAL-G), a managed provider's continuous backup (RDS, Cloud SQL, Neon, PlanetScale). The framework deliberately does not reimplement it: layer PITR under these native snapshots at the infra layer. A weekly voltro data backup + provider PITR together give you both a portable artifact and a fine-grained restore point.

Test your backups. A backup you've never restored is a hypothesis. Restore your latest artifact into a throwaway database and boot the app against it on a schedule — the stamp's dialect/fingerprint checks turn a silently-broken backup into a loud one, but only an actual restore proves the bytes are good.

#Masking (prod → dev/stage safely)

Cloning prod into a lower environment must not carry real user data. voltro data export does this with masking: PII is replaced by realistic, referentially-consistent fakes at the source — before a row is ever written — so real values never reach the bundle, transit, or a developer's machine.

voltro data export ./out --profile dev

Masking is driven by two layers:

1. Classification in the schema — .sensitive(class) / .safe() on each column. This says what kind of data a column holds. It lives in the schema because the data's meaning is a property of the schema, not of one export.
2. A per-environment masking POLICY — how each class/column is transformed for this target. It lives in a profile so a target environment's whole recipe is one reviewable, version-controlled file.

// data-profiles/dev.profile.ts
import { defineDataProfile } from '@voltro/data-transfer'
export default defineDataProfile({
  subset: { seeds: { users: undefined } },
  masking: {
    seed: process.env.MASK_SEED!,
    // classes override the built-in defaults; columns override per-column
    classes: { freeText: 'redact' },
    columns: { '_voltro_mail_outbox.to': { fake: 'email' } },
    onUnclassified: 'error', // fail-closed (default)
  },
  consistency: 'snapshot',
  assets: true,
})

#Deterministic + seed-keyed

Every transform is deterministic and keyed on a secret seed:

• Same input → same fake, everywhere. One email becomes the SAME fake in every table it appears in, so joins survive; and it stays stable across re-runs, so dev data doesn't churn. Keeping the seed stable is pseudonymisation.
• Rotate or discard the seed → the mapping is irrecoverable. That makes the result anonymisation.

seed is REQUIRED — masking without one is a bug (pass it from the environment, never commit it). Transforms are also format-preserving (a fake email is a valid email) and null-preserving (a null stays null — nullability holds).

#Actions

An action is what a column's value becomes. Set them per class (classes) or per column (columns, which wins):

Action	Effect
keep	copy verbatim
null	set to null
redact	fixed placeholder ([redacted] for text, null otherwise)
hash	deterministic opaque hex (stable, non-reversible without the seed)
dateShift	shift a date by a seed-derived offset (relative intervals + ordering preserved)
{ fake: '<class>' }	a format-preserving fake of that class
{ custom: (input) => … }	your own transform (input = { value, table, column, columnType, seed })

#Class → action defaults

A minimal policy is just a seed — every known class has a default action:

Class	Default action
email fullName firstName lastName username phone address company url ip creditCard	{ fake: '<class>' }
date	dateShift
secret	null
freeText	redact

A custom class with no entry in classes falls back to redact. Resolution order for any column is: columns[table.column] → the class's action (classes → built-in default) → .safe() keeps → PK/FK keeps → onUnclassified.

#Fail-closed

Masking is fail-closed. A column that is neither .sensitive() nor .safe() (and isn't a PK/FK, and has no columns override) refuses the export and is named in the error — so a newly-added column can never silently leak PII to dev. Classify it, or override it in the policy.

The opt-out is onUnclassified: 'keep' (or 'null'), which makes masking fail-open for unclassified columns. Discouraged — it defeats the guarantee; prefer classifying the column. See why fail-closed.

A policy this build cannot carry out is refused too, before a row is read. An action of a shape the applier does not understand — a typo like { action: 'fake', kind: 'email' } where the shape is { fake: 'email' } — used to fall through to "copy the value", so the export succeeded and shipped the raw column while the audit line counted it as masked. It is now a MaskingError with invalidActions, reported separately from unclassified because the two have different fixes: one needs a classification, the other needs the policy corrected.

#--dry-run — preview without writing

Preview a masking export without writing a bundle — the trust surface before real data moves. It reads a small sample per table and reports, per masked column, the before→after; a leak scan that warns when a KEPT column still LOOKS like PII (catches a misclassification — a .safe() on something that isn't); and the fail-closed list.

voltro data export ./out --profile dev --dry-run

  users
    email  [fake:email]  "ada@corp.com" → "grace.hopper1847@example.com"
    name   [fake:fullName]  "Ada Byron" → "Linus Torvalds"
    ssn    [null]  "078-05-1120" → null

  ⚠ possible leaks in KEPT columns:
    users.nickname looks like email: "ada@corp.com"

  ✗ unclassified (would BLOCK a real export — add .sensitive()/.safe() or a policy override):
    users.bio

  ✓ every exported column is classified — safe to run.

Nothing is written. --dry-run exits non-zero when the unclassified list is non-empty, so it doubles as a CI gate for classification coverage. (It requires a profile with a masking policy — there is nothing to preview otherwise.)

#Audit

A masking export records what it changed in the bundle's manifest.json, under masking:

"masking": {
  "transformed": ["users.email", "users.name", "users.ssn"]
}

masking.transformed lists every table.column that was pseudonymised/anonymised (i.e. every column whose action was not keep), so a reviewer can verify the copy was masked as intended — without diffing the data. A policy id is recorded alongside it when the policy sets one.

#Subsetting

subset exports a referentially-correct SLICE instead of whole tables. You give seed rows per table; the exporter adds their transitive FK-parent closure — every row the seeds (and their parents, recursively) point at — so the slice imports with no dangling references.

subset: {
  seeds: {
    users: undefined,                          // every user (no predicate)
    orders: eq('status', 'open'),              // only open orders (a Predicate)
  },
}
// exports those rows PLUS every parent row they reference (users an order points at, etc.)

A subset replaces scope when both are set — only the seeded tables and their parent closure are exported. Contrast the two:

• scope (--tenant / --tables / all) selects tables (a tenant's rows plus the row-subset FK parents they reference, or an explicit table set). Use it to move a tenant or named tables.
• subset selects specific rows and pulls in exactly the parents they need. Use it to carve a small, self-consistent slice ("these 1000 users and everything they reference").

Child closure. By default the closure follows parents only — a seeded user brings the org it belongs to, but not that user's posts. Set children to also pull in the rows that reference the seeded set, each scoped to the ids that actually point into it (never a whole child table). The child rows' own parents fold back through the parent closure, so the slice stays referentially complete:

subset: {
  seeds:    { orgs: eq('id', 'org_abc') },
  children: { roots: ['orgs'] },  // pull the org's users → their posts → those posts' comments
}

children.roots anchors the walk: a child comes along only when it references a roots row (or a child already pulled in this walk), so the walk stays tight — a row that references only a shared parent outside the anchor is not dragged in. (This is exactly what keeps a --tenant export tenant-tight: its child walk is anchored on the tenant() tables.) Use children: true for an unanchored org-slice takeout where there is no tenant boundary to respect.

Subsetting reads the selected rows into memory to collect id sets — it's built for small slices, not for halving a huge table (use scope/--tenant for that).

#Profiles

A profile bundles a target environment's whole recipe — scope/subset, masking, consistency, compression, assets — into one default-exported object, so voltro data export --profile dev is a single, reviewable, version-controlled description of how prod becomes dev.

import { defineDataProfile } from '@voltro/data-transfer'

export default defineDataProfile({
  scope:       { kind: 'all' },        // or omit and use --tenant/--tables
  subset:      { seeds: { users: undefined } }, // replaces scope when set
  masking:     { seed: process.env.MASK_SEED! },
  consistency: 'snapshot',             // 'live' | 'snapshot'
  compression: 'zstd',                 // 'zstd' | 'gzip' | 'none'
  assets:      true,                   // include stored blobs
})

--profile <name> resolves, in order: ./<name>, ./<name>.profile.ts, ./data-profiles/<name>.ts, ./data-profiles/<name>.profile.ts. So --profile dev finds ./dev.profile.ts or ./data-profiles/dev.ts. The module's default export must be the profile (use defineDataProfile for full type-checking).

CLI flags override profile values. --assets, --compression, --consistency, --tenant, --tables all win over what the profile sets — so a profile is the default and a flag is the one-off override.

#Which do I use?

Goal	Command
Disaster recovery / scheduled backups	voltro data backup (native)
Clone prod → staging (masked, no real PII)	voltro data export --profile <env> (masking + subset)
Preview a masked export before running it	voltro data export --profile <env> --dry-run
Clone prod → staging (same dialect, unmasked)	voltro data export then import (or native backup/restore)
GDPR / per-tenant takeout	voltro data export --tenant <id> --assets
A small self-consistent slice of the data	voltro data export --profile <env> with a subset
Move Postgres → MySQL	voltro data export then import (cross-dialect lint applies)
Replace a dev/staging environment with your local state	voltro data import <dir> --target api --mode replace --atomic
Seed a fresh cluster through the running app (invariants + encryption)	voltro data import <dir> --target api (default upsert)
Move a bundle too big for one request	nothing extra — the upload chunks itself; --chunk-size only if you need a different size
Resume an upload that died	re-run the identical command