Distributed tracing

Trace continuity is automatic and on by default in voltro dev — a buffer-only tracer is always installed (set VOLTRO_TRACING_BUFFER=off to disable). You don't wire anything. A single request gets one traceId that flows frontend → api → api, and every log line of that request carries it.

#How propagation works

• Frontend → server is automatic. useMutation / useSubscription wrap each rpc call in a client span; @effect/rpc propagates that span's W3C trace id to the server, so the handler span — and everything it calls — shares the same traceId. In any handler, ctx.request.traceId is that id.
• api → api is automatic over HTTP. An action or handler that calls another api via the framework HttpClient sends a W3C traceparent header; an inbound *.webhook.tsx route (an incoming HTTP handler) continues the caller's trace. So frontend → api Y → api Z (via webhook / HTTP) is one trace.
• Every log line carries it. Handlers stamp fields.traceId onto each log line they emit, so voltro logs --trace <id> returns the whole chain across hops, in order.

client.mutation.placeOrder        traceId=4bf92f35…
└─ mutation.placeOrder            traceId=4bf92f35…   (api Y, ctx.request.traceId)
   └─ webhook/fulfilment          traceId=4bf92f35…   (api Z, continued via traceparent)

#Seeing traces

• DevTools "Traces" panel (dev) — a waterfall per request, failed hops tinted red. Backed by the in-memory ring buffer.
• Cloud dashboard's per-app "Traces" page — same waterfall, with history kept in _voltro_traces (postgres) when durable persistence is enabled (dev: on by default, interesting-only; prod: off by default → use OTLP). The in-memory ring is always the live source; see Durable trace volume below.

#Durable trace volume (postgres)

The in-memory ring always keeps EVERY span for live debugging. SEPARATELY, on postgres, the framework can durably mirror a SUBSET of spans into _voltro_traces so the in-app Traces dashboard survives restarts. This is a dev convenience — at real volume durable tracing belongs in an OTLP backend (Tempo / Honeycomb / Datadog), not your OLTP postgres (set OTEL_EXPORTER_OTLP_ENDPOINT). A row-per-span firehose into postgres is what turned this table into ~85% of a production DB. So persistence is environment-aware + fully configurable:

• Default OFF in production, interesting in dev. voltro dev (not NODE_ENV=production) persists only "interesting" spans. voltro serve / voltro start default NODE_ENV=production when it's unset, so they persist NOTHING and _voltro_traces is not even created — use OTLP in prod. An explicit NODE_ENV is never overridden.
• VOLTRO_TRACING_PERSIST = off | errors | interesting | all (overrides the env default everywhere). interesting = error + slow + per-trace root spans, MINUS the high-volume subscription delivery (snapshot/delta) spans. errors = error spans only. all = every non-delivery span. off = no persistence and no table.
• VOLTRO_TRACING_SLOW_MS (default 500) — the "slow" threshold used by interesting.
• VOLTRO_TRACING_PERSIST_DELIVERY (default off) — also persist the subscription delivery spans (the firehose; rarely wanted).
• VOLTRO_TRACING_SAMPLE (0–1, default 1) — per-trace sampling (a kept trace keeps all its eligible spans; errors are never sampled out). =0 with no explicit mode means off.
• VOLTRO_TRACING_TTL_HOURS (default 24) — the GC prunes rows past the TTL; it loops until the whole over-TTL backlog drains each pass (batched set-based delete), so it always catches up. It WARNs if a pass errors or is shifting very high volume — a silently-failing GC is no longer silent.
• VOLTRO_TRACING_BUFFER=off — disables the durable flusher + the in-memory tracer entirely (and drops the table).

_voltro_traces has no foreign keys — TRUNCATE TABLE _voltro_traces reclaims the space instantly if it ever grew under old defaults. To get the in-app Traces history in prod anyway, set VOLTRO_TRACING_PERSIST=interesting (keeps it small) — but prefer OTLP for production-grade tracing.

#Dev error-surfacing toast

In dev, if any hop of a trace you triggered errors — even when the frontend call itself returned OK (a fire-and-forget emit, a detached workflow, a swallowed downstream error) — an auto-toast appears in the browser pointing at the failing hop and at voltro logs --trace <id>.

This catches the class of failure that's normally invisible: the UI shows success, but a downstream hop quietly failed. The toast surfaces it immediately with the exact traceId to chase.

#The debugging loop

When you debug a failure:

1. Get the traceId — from the dev error toast, an error line's fields.traceId, or /_voltro/inspect/traces?onlyErrors=1.
2. Run voltro logs --trace <id> — that's the end-to-end causal chain, every log line of the request across hops, in order.

This is the fastest way to answer "where did this error come from?" — far faster than inferring it from source or from the user's description.

#Shipping traces to a vendor (Sentry / Datadog)

The same trace continuity flows to a vendor APM when you install a deep-observability plugin — no OTEL_* env. Each contributes an OTel span-processor to the framework's tracer via contributeObservability, so the vendor runs as a consumer of the existing tracer (the in-app Traces panel keeps working) and receives the identical spans:

• @voltro/plugin-sentry — every mutation/query/action error becomes a Sentry issue correlated to its trace_id + span_id, with the request's log lines as breadcrumbs; opt-in performance traces (traces: true).
• @voltro/plugin-datadog — framework spans → the Datadog Agent's OTLP receiver (traces: true), plus log forwarding with dd.trace_id correlation.

See Observability › Routing traces to a vendor for the contribution surface + the sampler caveat.

#"Missing peer" warnings on install (benign)

pnpm install may print missing-peer warnings for @opentelemetry/sdk-logs and @opentelemetry/sdk-trace-web — these are optional peers of @effect/opentelemetry (pulled in transitively), used only if you export logs/browser traces to OTLP. Boot and the in-memory trace ring work without them, so the warnings are safe to ignore. To silence them, either add the two packages to your app, or add a pnpm rule:

// package.json
"pnpm": { "peerDependencyRules": { "ignoreMissing": ["@opentelemetry/sdk-logs", "@opentelemetry/sdk-trace-web"] } }