Inspect & test

When something's wrong, these are the tools. Live inspection of a running app happens over an HTTP surface (and the dashboard that consumes it), not a dedicated CLI verb. The shell-facing debugging commands are voltro logs and voltro traces; the harness commands are voltro test and voltro e2e.

#voltro inspect

voltro inspect <subcommand> is the ergonomic CLI wrapper over the HTTP surface below. It discovers every running api via ~/.voltro/runtime-registry.json, fans the matching GET/POST out to each, and renders the merged result — no curl + jq needed.

voltro inspect app                              # manifest meta (kind, name, store, …)
voltro inspect routes                           # web page tree (web apps only)
voltro inspect rpc                              # procedures + workflow descriptors (api only)
voltro inspect metrics                          # rolling rpc latency buckets
voltro inspect cache                            # web data-cache stats (web apps only)
voltro inspect schedules                        # cron registrations + coordination mode
voltro inspect schedules --failing              # only broken crons — exits 1 if any (see below)
voltro inspect aggregates                       # materialised aggregate views
voltro inspect invoke --tag users.list --input '{}'   # call a procedure over HTTP

Flags on every subcommand: --process <name> narrows to one api; --format pretty|json (default pretty). invoke additionally takes --tag <procedureTag> and --input <json>. With no subcommand it prints the endpoint map + the live processes it can reach.

#schedules --failing — is any cron actually broken?

A schedule fires unattended: there is no user watching it fail. The plain schedules listing answers which crons exist and when they fire next — never whether they work. --failing rolls each schedule's recent runs (/_voltro/inspect/schedules/runs) into a verdict and prints only the broken ones:

$ voltro inspect schedules --failing
# schedules @api
  sprint.report  0 2 * * *  FAILING x87
    every recorded run failed (last 20)
    Workflow "sprint.report" was started with an invalid payload. missing required field(s): teamId

It exits 1 when anything is failing, so it works as a post-deploy gate and not only as something someone remembers to run:

voltro inspect schedules --failing || echo "broken cron — do not promote"

A trailing success ends a streak (a recovered job is not reported), and skipped / missed runs are ignored — those are coordination outcomes (another pod took the tick, the process was down), not handler verdicts. A schedule that has never run is not "failing".

Pair it with voltro logs --level error: a failing handler now logs at error level, so the two surfaces agree.

#voltro schedule run <name> — fire one job, now

voltro schedule run nightly-reconcile
voltro schedule run nightly-reconcile --process billing --format json
voltro schedule run nightly-reconcile --url https://api.example.com   # a deployed app

For the normal case: a nightly job that corrects business data, and you want to run it once and watch. It reports the run id, and voltro inspect schedules shows the outcome.

A run id of null is not a failure and is reported as its own outcome: the run was coordinated away — another replica holds the lock, or the previous run is still going and this schedule's onOverlap is 'skip'. Printing "ok" there would claim work that never started.

--trigger external records the run as externally triggered instead of manual, for schedules that are normally fired by an outside scheduler.

This works against voltro serve as well as voltro dev. It did not before — production mounted no inspect surface at all, which also meant the post-deploy gate below could only ever be run against a dev server.

#Targeting a deployed app

Every command in this family resolves its target from the local runtime registry — the apps running on this machine. Pass --url to point one at a deployed app instead:

voltro inspect app     --url https://api.example.com --token "$TOKEN"
voltro logs   --tail 100 --url https://api.example.com
voltro traces --errors   --url https://api.example.com
voltro check             --url https://api.example.com

--token (or VOLTRO_INSPECT_TOKEN) supplies the bearer; VOLTRO_INSPECT_URL sets a default target so you can drop the flag. Works for inspect, logs, traces, workflows, cluster and check.

#logs and traces need a ring on the target — off by default

A deployed app runs voltro serve, and voltro serve keeps no in-process log ring or span buffer unless you ask for one. Without it both commands answer:

voltro logs: NOTHING WAS SEARCHED — the target could not be read: HTTP 404
endpoint not enabled on this instance — This instance keeps no in-process logs ring…

That headline is the point: "no records matched the given filters" would have sent you off widening --tail forever. Turn a ring on per app:

// app.config.ts
export default defineAppConfig({
  inspect: {
    logs:   true,              // default size
    traces: { size: 2_000 },   // or name it
  },
})

or per deployment, without a rebuild: VOLTRO_INSPECT_LOGS=2000, VOLTRO_INSPECT_TRACES=on, =off to override a config that declared one.

Why off by default. A ring is memory on every replica, forever, for data most deployments already collect from stdout through their platform's pipeline. That pipeline stays the main road; the ring answers the different question — what has THIS instance been doing in the last few minutes — and it answers it for a hosted app, where reading a pod's stdout is not an option.

What a ring does not do. It holds a bounded number of records on ONE replica. Against a fleet you are reading 1/N, and it is not retention: a restart empties it. voltro dev keeps both rings unconditionally — this switch is about what production does.

#voltro probe access — is a declared guard actually enforced?

voltro check reads an app's manifest and reports a procedure with no access decision. It cannot tell you whether the decisions that ARE declared are enforced. voltro probe access asks the running app:

voltro probe access                 # every live api
voltro probe access --url https://api.example.com --strict

It calls every procedure that declares a guard with no credentials at all and reports any that answer anyway:

api: probed 14 guarded procedure(s) with NO credentials
  ✗ orders.export — ANSWERED an unauthenticated call
  ? billing.invoice — answered 'ParseError' — not an access refusal, so the guard was not reached
  ✓ 12 refused · 1 inconclusive · 1 admitted

Three verdicts, and the third is what keeps the command honest:

Verdict	Meaning
refused	The call came back as an access refusal. The declaration is enforced.
admitted	The call SUCCEEDED without credentials. This is the finding.
inconclusive	The call failed for a reason that is not an access refusal — usually payload validation running before the guard. Not a pass.

Exit code is non-zero on any admitted. --strict also fails on inconclusive, which is what you want in CI: "I could not tell" should block.

What it deliberately does not do. It probes ANONYMOUSLY, so it cannot tell orders:read from orders:write — it answers exactly one question, and the alternative (minting a subject per guard) would put credential minting into a command that can be pointed at production. Procedures declared openAccess: are skipped; probing them would report every deliberately-public route as a finding and bury the real ones.

#Securing the local surface

voltro dev mints a per-project VOLTRO_INSPECT_TOKEN into .env.local, so the inspect surface is authenticated from the first boot — the dev server listens on every interface, and without a token anyone on the same network could read your rows, schema and logs. You don't have to wire it anywhere: the CLI picks the token up from the runtime registry (so the commands work from any directory), and the dashboard's server-side proxy supplies it for same-machine targets. Setting VOLTRO_INSPECT_TOKEN yourself always wins.

#The inspect HTTP surface

Every voltro dev / voltro start instance exposes an introspection surface under /_voltro/inspect/*.

It is not read-only. The core endpoints are reads, but installed plugins mount their own — and some are POSTs that DO things: plugin-governance mounts /erase (an irreversible GDPR right-to-be-forgotten deletion) and /export (a full personal-data dump); plugin-storage mounts /share and /revoke.

So a mutating method needs a second credential. VOLTRO_INSPECT_WRITE_TOKEN, sent as the x-voltro-inspect-write header ON TOP of the bearer — an additional factor, not an alternative: the read token still has to be correct. GET / HEAD / OPTIONS are unaffected. Unset, those endpoints are refused.

curl -H "authorization: Bearer $VOLTRO_INSPECT_TOKEN" \
     -H "x-voltro-inspect-write: $VOLTRO_INSPECT_WRITE_TOKEN" \
     -X POST http://localhost:4000/_voltro/inspect/plugins/governance/erase

voltro dev mints it per project like the read token, and the dashboard proxy injects it for loopback targets, so the dev loop is unchanged. Nothing mints it for serve / start — in production a destructive endpoint should take a deliberate act to enable. Use a DIFFERENT value from the read token; reusing it gives the split no meaning.

A plugin mounting a non-GET inspect endpoint must also declare the inspect:write permission, and the boot audit refuses it otherwise. That governs what a PLUGIN may mount; the write credential governs who may call it. The Voltro Dashboard consumes it to render the route sitemap, RPC list, subscription panel, workflow runs, and metrics. You can also hit the endpoints directly with curl (the voltro inspect subcommands above are the thin wrapper over exactly these).

PORT=4000   # from the app's app.config.ts

curl -s localhost:$PORT/_voltro/inspect/routes  | jq    # web: page tree + render-mode flags
curl -s localhost:$PORT/_voltro/inspect/rpc     | jq    # api: every query / mutation / action / workflow
curl -s localhost:$PORT/_voltro/inspect/metrics | jq    # rolling per-tag latency + invocation count

There is no /_voltro/inspect/queries endpoint. The registered GET surface is app, routes (web-only), cache (web-only), rpc (api-only), metrics, subscriptions (api-only), checks (api-only) and agent/tools (api-only) — rpc is the procedure list, routes is the web page tree.

#Invariant checks + the agent-tool surface

curl -s localhost:$PORT/_voltro/inspect/checks      | jq   # browser-safety, procedure-access, convergence, serverOnly
curl -s localhost:$PORT/_voltro/inspect/agent/tools | jq   # the exposeAsTool procedures an agent may run

checks runs the framework's own invariant checks and answers pass | fail | unavailable per check — unavailable means THIS process cannot answer it (a deployed voltro serve has no source tree to walk) and is never a pass. agent/tools lists the policy-admitted agent tools, and its sibling POST /_voltro/inspect/agent/call executes one; both are off until agents: { mcp: true }, and the call additionally needs the write credential plus an app credential on x-voltro-agent-authorization. See the MCP server page for the full gate list.

What the surface reads:

• The endpoints serve in-process state from the running instance — no separate daemon.
• The default target is http://localhost:<port> based on the cwd's app.config.ts.
• GET responses are JSON; pipe into jq.

#Routes & RPC

curl -s localhost:$PORT/_voltro/inspect/routes | jq    # web: page tree + render-mode flags
curl -s localhost:$PORT/_voltro/inspect/rpc    | jq    # api: query / mutation / action / workflow list

#Subscriptions panel

curl -s localhost:$PORT/_voltro/inspect/subscriptions | jq

Shows every active subscriber with:

• Query name + input
• Subject (who's subscribed)
• Read set (which tables / rows are tracked)
• Frame buffer depth (backpressure indicator)
• Connection age

For "why isn't this updating?" — check the read set. If your mutation writes to a table not in the read set, the subscription doesn't invalidate.

#Workflow runs — voltro workflows

voltro workflows is the primary surface for inspecting + operating runs:

voltro workflows list                          # recent runs
voltro workflows show <runId>                  # one run's steps + events
voltro workflows retry <runId>
voltro workflows cancel <runId>
voltro workflows suspend <runId>
voltro workflows resume <runId>
voltro workflows signal <runId> --name approval        # inject a named signal
voltro workflows update <runId> --name …
voltro workflows children <parentExecutionId>
voltro workflows flow                          # the admission queue + ledger
voltro workflows pause|unpause <workflowName>  # stop/restart admission fleet-wide
voltro workflows cancel-many --reason "…"      # DRY RUN until --commit
voltro workflows replay-many --mode redrive    # DRY RUN until --commit
voltro workflows inferences                    # offloaded model calls in flight

inferences shows what nothing else can: a run parked on an offloaded model call reads suspended in the run list with no step row yet, so during a slow provider — the moment you would look — the run list has nothing to say.

Underneath, workflow state lives in the _voltro_workflow_runs + _voltro_workflow_run_steps tables and is surfaced live by the dashboard's Workflows panel. The same data is reachable over HTTP:

curl -s localhost:$PORT/_voltro/inspect/workflows/runs | jq            # recent runs
curl -s "localhost:$PORT/_voltro/inspect/workflows/runs/<runId>/steps" | jq   # step-by-step
curl -s "localhost:$PORT/_voltro/inspect/workflows/runs/<runId>/events" | jq  # the run's event log
curl -s "localhost:$PORT/_voltro/inspect/workflows/stats?hours=24" | jq       # bucketed run activity (the dashboard chart)

The runs endpoint filters server-side, so a triage query over a large run history costs one narrow page instead of the whole table:

# multi-status + tag search + source + id-prefix + time range — all composable
curl -s "localhost:$PORT/_voltro/inspect/workflows/runs?statuses=failed,cancelled&q=orders&source=workflow-rpc&idPrefix=wfrun_&from=2026-08-01T00:00:00Z&to=2026-08-08T00:00:00Z" | jq

statuses is a comma list; q is a case-insensitive tag substring; idPrefix matches the run id or the execution id (you never have to know which kind your log line carried); from/to bound startedAt. The dashboard's filter bar sends exactly these params.

/workflows/stats returns ~48 buckets over a trailing window (hours, default 24, max 168; optional tag), each with started / succeeded / failed / cancelled counts, plus per-workflow totals. When the window held more runs than the scan cap, the response says truncated: true — the chart renders that as a warning, because a silently-truncated chart shows throughput dropping at exactly the moment it spiked.

Each run row carries ID, name, status (running / succeeded / failed / dead), step count, last completed step, and duration. The per-run action endpoint matches …/workflows/runs/<runId>/<action> for cancel / retry / suspend / resume / signal — the voltro workflows subcommands and the dashboard's run-detail buttons both POST to these:

curl -s -X POST localhost:$PORT/_voltro/inspect/workflows/runs/<runId>/retry
curl -s -X POST localhost:$PORT/_voltro/inspect/workflows/runs/<runId>/cancel

For dead-letter triage, filter runs by status, or open the dashboard's Workflows tab. See Workflows / Debugging.

#voltro logs

The fastest way to see what a running instance is doing. Buffers the last 2000 server-side log records plus every browser console line the dev console bridge forwarded.

voltro logs                                # last 100 from every running process
voltro logs --tail 50 --level error        # only errors, last 50
voltro logs --since 30s                    # last 30 seconds
voltro logs --filter 'notes.summarise'     # message substring
voltro logs --trace <traceId>              # the WHOLE causal chain for one request
voltro logs --format json | jq '.records'  # machine-parseable

Run this BEFORE grepping source — the buffer carries the real error, stack, and rpc tag. The full flag set (--scope, --source, --process, --no-color, …) is documented in Traces & logs from the shell — the canonical reference for both commands.

#voltro traces

Mirror of voltro logs for the distributed-trace buffer.

voltro traces                              # 20 newest traces, pretty
voltro traces --errors                     # only traces containing an errored span
voltro traces --id <traceId>               # one trace, span waterfall
voltro traces --errors --format json | jq '.traces[]'

Workflow: voltro traces --errors --format json to find a failure, then voltro logs --trace <id> --format json for the full chain (frontend → api → api, in order). The full flag set (--min-duration, --status, --process, …) lives in Traces & logs from the shell.

#voltro cluster

voltro cluster status gives a clustering snapshot of every running api — one row per instance with its replicaId, runner address, dialect, CDC flavour, coordination mode, and server_id, plus a flag for any SQL runner stuck on localhost (a common misconfiguration that silently breaks cross-instance work). Use it to confirm a multi-instance deployment actually formed a cluster rather than N isolated nodes.

voltro cluster status                 # pretty table across every running api
voltro cluster status --json          # machine-readable (also --format json)
voltro cluster status --process api   # narrow to one named process

Like the rest of the inspect family it reads the live /_voltro/inspect/* surface, so an api has to be running.

#voltro test

voltro test
voltro test path/to/file.test.ts
voltro test --filter notes

Runs Vitest with the framework's preset:

• Auto-loaded global setup (test context, mock providers).
• STORE=memory by default for fast isolation.
• Plays well with @voltro/testing helpers (mock stores).

#Every vitest flag is forwarded

Anything the command does not interpret itself goes straight to vitest — parsed by vitest's own CLI parser, not a list this wrapper maintains:

voltro test --coverage
voltro test --reporter=junit --outputFile=reports/junit.xml
voltro test --coverage --reporter=junit --outputFile=reports/junit.xml

That covers coverage numbers and a JUnit report for a merge-request widget, which is what most pipelines want beyond the exit code.

--coverage needs a provider package. vitest ships coverage providers as optional peer dependencies, so nothing installs one for you. Every app scaffolded by voltro create-project / voltro add-app already declares @vitest/coverage-v8 beside vitest; an older project adds it once:

pnpm add -D @vitest/coverage-v8      # or --coverage.provider=istanbul → @vitest/coverage-istanbul

voltro test checks for it before booting vitest and refuses with that install command, because vitest's own failure (Cannot find dependency '@vitest/coverage-v8') names neither the flag nor the fix.

The framework keeps three decisions for itself and they win over a forwarded flag: the root (a positional that is an existing directory, which vitest would otherwise read as a filter), --watch, and passWithNoTests — an explicit filter that matches no file is an error, which vitest cannot decide because it does not know which positional was treated as a root.

An unrecognised flag is ignored rather than fatal, and a flag vitest cannot parse at all degrades to "run without the extra flags" with a warning instead of taking the run down.

The actual test runner is Vitest; this command is a thin wrapper that injects the framework's config. You can run vitest directly if you prefer:

pnpm vitest

#voltro e2e

voltro e2e            # current dir
voltro e2e apps/web   # explicit web app directory

voltro e2e takes only an optional path to the web app; it parses no other flags. Boots:

1. voltro dev for the api app.
2. voltro dev for the web app.
3. Runs every file matching e2e/**/*.spec.ts, one process each, as a plain tsx script (node --import tsx <file>).
4. Tear down: stops the boot processes.

There is no test runner and no browser driver here. A spec is an ordinary TypeScript program: it runs top to bottom, and a non-zero exit code (an uncaught throw, process.exit(1), a failed node:assert) is a failed file. The framework ships no describe/it, no page fixture, no reporter, no sharding, and no browser — because what voltro e2e actually contributes is the lifecycle, and the lifecycle is the same whichever driver you pick.

Two environment variables are handed to every spec:

Variable	Value
WEB_URL	http://localhost:<webPort> — the booted web app
API_URL	http://localhost:<apiPort> — the booted api app

A spec that only needs the API is just fetch plus node:assert:

// apps/web/e2e/signup.spec.ts
import assert from 'node:assert/strict'

const res = await fetch(`${process.env.API_URL}/v1/signup`, {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ email: 'a@b.com', password: 'correct horse battery staple' }),
})

assert.equal(res.status, 200)
console.log('✓ signup accepted')

To drive a real browser, bring your own driver and launch it inside the spec — the framework does not choose one for you, and does not install one:

// apps/web/e2e/signup-browser.spec.ts
import assert from 'node:assert/strict'
import { chromium } from 'playwright-core'   // your dependency, not the framework's

const browser = await chromium.launch()
const page = await browser.newPage()
await page.goto(`${process.env.WEB_URL}/signup`)
await page.fill('[name=email]', 'a@b.com')
await page.fill('[name=password]', 'correct horse battery staple')
await page.click('button[type=submit]')
await page.waitForURL(/\/dashboard/)
assert.ok(page.url().includes('/dashboard'))
await browser.close()

Configure the boot in app.config.ts:

export default {
  type: 'web' as const,
  name: 'web',
  e2e: {
    apiDir: '../api',            // relative to the web app root
    pattern: 'e2e/**/*.spec.ts',
    apiPort: 4000,
    webPort: 5173,
  },
}

Anything below the browser — a handler, a guard, a REST route, an Idempotency-Key replay, the x-tenant header — is faster and more precise from a request-level test, which needs no booted process at all. Reach for voltro e2e when the thing under test is the two processes talking to each other.

#Securing the inspect surface

The surface is fail-closed: with no VOLTRO_INSPECT_TOKEN configured, every endpoint answers 401. The absence of a secret is not consent — a check you were configured not to perform is a refusal, not a pass.

That is why voltro dev mints one for you (above) and why nothing mints outside dev: in production a missing secret must stay a boot-time decision rather than an invented value. So a production voltro start serves nothing here until you set the token yourself:

VOLTRO_INSPECT_TOKEN=$(openssl rand -hex 32)
voltro start apps/api

Then every request must carry the token:

curl -s -H "Authorization: Bearer $VOLTRO_INSPECT_TOKEN" \
  localhost:4000/_voltro/inspect/rpc | jq

Or disable the surface entirely: VOLTRO_INSPECT=off. The dashboard then can't introspect the production instance — that's intentional.

#Anti-patterns

• Assuming the surface is open because you didn't configure it. It is the reverse: no token means 401, not "everyone". If a production dashboard suddenly stops introspecting, the missing VOLTRO_INSPECT_TOKEN is the first thing to check — not a network problem.
• voltro test against STORE=postgres by default. Slower + flaky (test isolation harder). Use postgres only for integration tests that NEED it.
• Skipping voltro e2e because "it's slow". It catches integration bugs that unit tests miss. Run it in CI on every PR; locally for changes that touch queries.

#See also

• Workflows / Debugging — the dashboard's workflow panel
• Self-hosting — production observability setup