Distributed tracing

Trace continuity is automatic and on by default in voltro dev — a buffer-only tracer is always installed (set VOLTRO_TRACING_BUFFER=off to disable). You don't wire anything. A single request gets one traceId that flows frontend → api → api, and every log line of that request carries it.

#How propagation works

• Database runtime boundaries keep their parent. Awaited work dispatched through another Effect runtime sharing the server's tracing layer keeps the active parent span, including SQL-store calls. Explicit parent spans and explicit new roots take precedence; independent tracing layers do not share context.

• Frontend → server is automatic. useMutation / useSubscription wrap each rpc call in a client span; @effect/rpc propagates that span's W3C trace id to the server, so the handler span — and everything it calls — shares the same traceId. In any handler, ctx.request.traceId is that id.

• api → api is automatic over HTTP. An action or handler that calls another api via the framework HttpClient sends a W3C traceparent header; an inbound *.webhook.tsx route (an incoming HTTP handler) continues the caller's trace. So frontend → api Y → api Z (via webhook / HTTP) is one trace.

• Every log line carries it. Handlers stamp fields.traceId onto each log line they emit, so voltro logs --trace <id> returns the whole chain across hops, in order.

The client span exists before the transport proves delivery. If a connection fails, its traceId can therefore have no matching server span — or the server may have completed the handler and lost only the response. In rpc-error policy, use RpcError.kind, never the mere presence of traceId, to distinguish transport failures from handler failures.

client.mutation.placeOrder        traceId=4bf92f35…
└─ mutation.placeOrder            traceId=4bf92f35…   (api Y, ctx.request.traceId)
   └─ webhook/fulfilment          traceId=4bf92f35…   (api Z, continued via traceparent)

#Seeing traces

• DevTools "Traces" panel (dev) — a waterfall per request, failed hops tinted red. Backed by the in-memory ring buffer.
• Cloud dashboard's per-app "Traces" page — same waterfall, with history kept in _voltro_traces (postgres) when durable persistence is enabled (dev: on by default, interesting-only; prod: off by default → use OTLP). The in-memory ring is always the live source; see Durable trace volume below.

#Reading a trace

The recent grouped view omits successful traces made entirely of recognized framework housekeeping spans, such as scheduler-claim polling. It filters those groups before applying the trace limit, so they do not crowd out application requests. Errors and traces that also contain application work remain visible. In the embedded Traces scope, Show background traces includes the housekeeping groups; the same option is available as /_voltro/inspect/traces?includeBackground=1. The bounded raw ring and lookup by trace ID remain unchanged.

A trace explains one operation's timing and failures. Each span is a timed step: the handler, a database call or an outbound HTTP request. Start with the operation name and total duration, then expand a slow or failed step to inspect its recorded attributes and error. Attributes can identify the RPC tag, SQL operation or HTTP target; only information actually recorded by the instrumentation is shown.

For example, a refresh taking two seconds with a 1 ms database span probably spends its time elsewhere. A single sql.execute span under 1 ms only tells you that one database operation was fast. It does not identify the button that caused it, and it does not prove the whole refresh succeeded. If the caller was not captured, the panel says so rather than inventing a request relationship. Trace durations overlap when child work runs concurrently, so do not sum the bars as total request time.

The embedded console refreshes its recent traces every four seconds. Selecting a trace keeps its latest captured version and selected row even after the bounded recent feed evicts it. The detail marks this as a saved selection; selecting another entry releases it. While the trace remains in the live feed, newly received spans still update the selection.

#Durable trace volume (postgres)

The in-memory ring always keeps EVERY span for live debugging. SEPARATELY, on postgres, the framework can durably mirror a SUBSET of spans into _voltro_traces so the in-app Traces dashboard survives restarts. This is a dev convenience — at real volume durable tracing belongs in an OTLP backend (Tempo / Honeycomb / Datadog), not your OLTP postgres (set OTEL_EXPORTER_OTLP_ENDPOINT). A row-per-span firehose into postgres is what turned this table into ~85% of a production DB. So persistence is environment-aware + fully configurable:

• Default OFF in production, interesting in dev. voltro dev (not NODE_ENV=production) persists only "interesting" spans. voltro serve / voltro start default NODE_ENV=production when it's unset, so they persist NOTHING and _voltro_traces is not even created — use OTLP in prod. An explicit NODE_ENV is never overridden.

• VOLTRO_TRACING_PERSIST = off | errors | interesting | all (overrides the env default everywhere). interesting = error + slow + per-trace root spans, MINUS the high-volume subscription delivery (snapshot/delta) spans, MINUS the framework's own background-task reads, MINUS a fast query or transaction that no operation wraps (sql.execute, store.* as a root). errors = error spans only. all = every non-delivery span. off = no persistence and no table.

  Why a bare query is not a root worth keeping. A query or transaction that runs outside any request, job or schedule span — a poller's, a background drain's — has no parent, so it counts as a root. It is not the entry of anything a person asked for. A demo measured 5.3 million spans in 24 hours, 5.28 million of them such bare sql.execute / store.transactional roots: 3.3 GB of _voltro_traces, each span one more write on the database it describes. They are kept when slow or failing, like any other span.

  Why background tasks are excluded from "slow". The framework polls four of its own tables on a timer (_voltro_schedule_claims, _voltro_workflow_pending, _voltro_workflow_pauses, _voltro_ai_inferences). Those reads are slow exactly when the database is under pressure — which is when persisting them costs the most. A deployment measured the loop closing on itself: _voltro_traces at 476 571 rows / 335 MB, writing ~11 INSERTs/s onto the same 15-slot pooler the app read through, 99 % of it framework poller spans. Pool pressure makes the spans slow, slow spans are "interesting", persisting them costs pool. They are still kept when they ERROR, and all mode still keeps everything. A request-path framework table like _voltro_api_keys is NOT excluded — a slow lookup there is a real user waiting.

• VOLTRO_TRACING_SLOW_MS (default 500) — the "slow" threshold used by interesting.

• VOLTRO_TRACING_PERSIST_DELIVERY (default off) — also persist the subscription delivery spans (the firehose; rarely wanted).

• VOLTRO_TRACING_SAMPLE (0–1, default 1) — per-trace sampling (a kept trace keeps all its eligible spans; errors are never sampled out). =0 with no explicit mode means off.

• VOLTRO_TRACING_TTL_HOURS (default 24) — the GC prunes rows past the TTL; it loops until the whole over-TTL backlog drains each pass (batched set-based delete), so it always catches up. It WARNs if a pass errors or is shifting very high volume — a silently-failing GC is no longer silent.

• VOLTRO_TRACING_BUFFER=off — disables the durable flusher + the in-memory tracer entirely (and drops the table).

_voltro_traces has no foreign keys — TRUNCATE TABLE _voltro_traces reclaims the space instantly if it ever grew under old defaults. To get the in-app Traces history in prod anyway, set VOLTRO_TRACING_PERSIST=interesting (keeps it small) — but prefer OTLP for production-grade tracing.

#Dev error-surfacing toast

In dev, if any hop of a trace you triggered errors — even when the frontend call itself returned OK (a fire-and-forget emit, a detached workflow, a swallowed downstream error) — an auto-toast appears in the browser pointing at the failing hop and at voltro logs --trace <id>.

This catches the class of failure that's normally invisible: the UI shows success, but a downstream hop quietly failed. The toast surfaces it immediately with the exact traceId to chase.

#The debugging loop

When you debug a failure:

1. Get the traceId — from the dev error toast, an error line's fields.traceId, or /_voltro/inspect/traces?onlyErrors=1.
2. Run voltro logs --trace <id> — that's the end-to-end causal chain, every log line of the request across hops, in order.

This is the fastest way to answer "where did this error come from?" — far faster than inferring it from source or from the user's description.

#Shipping traces to a vendor (Sentry / Datadog)

The same trace continuity flows to a vendor APM when you install a deep-observability plugin — no OTEL_* env. Each contributes an OTel span-processor to the framework's tracer via contributeObservability, so the vendor runs as a consumer of the existing tracer (the in-app Traces panel keeps working) and receives the identical spans:

• @voltro/plugin-sentry — every mutation/query/action error becomes a Sentry issue correlated to its trace_id + span_id, with the request's log lines as breadcrumbs; server performance traces are on by default (traces: false disables them). The browser integration keeps rpc markers inside the sampled page/navigation transaction and links each marker to its Effect/server trace.
• @voltro/plugin-datadog — framework spans → the Datadog Agent's OTLP receiver (traces: true), plus log forwarding with dd.trace_id correlation.

See Observability › Routing traces to a vendor for the contribution surface + the sampler caveat.

#@effect/opentelemetry — an optional peer every voltro dev needs

The OpenTelemetry SDK is an optional peer of @voltro/runtime, not pulled in transitively. It is loaded whenever a tracer layer is active — and a trace sink activates the layer just like an exporter does. So:

• voltro dev needs it by default: the in-process trace ring above is a sink. Only VOLTRO_TRACING_BUFFER=off removes the need.
• voltro serve needs it when the inspect trace ring is on (inspect: { traces: … } or VOLTRO_INSPECT_TRACES), when an exporter is on (FRAMEWORK_TRACING, FRAMEWORK_METRICS, any OTEL_EXPORTER_OTLP_* endpoint), or when an observability plugin contributes a span processor or metric reader. A serve with none of these boots without it.
• VOLTRO_TRACING_PERSIST does not decide it either way — it chooses what the ring persists.

Declare it in the api app, and keep it in every install that boots (a pruned production image, a lockfile rebuilt from scratch):

pnpm add @effect/opentelemetry

A boot that needs it and lacks it is a refusal, printed once: it names each cause and the switch that removes it, and the install line. voltro doctor reports @effect/opentelemetry is not installed when the installed @voltro/runtime cannot see it — before a deploy finds out.

#"Missing peer" warnings on install (benign)

pnpm install may print missing-peer warnings for @opentelemetry/sdk-logs and @opentelemetry/sdk-trace-web — these are optional peers of @effect/opentelemetry, used only if you export logs/browser traces to OTLP. Boot and the in-memory trace ring work without them, so the warnings are safe to ignore. To silence them, either add the two packages to your app, or add a pnpm rule:

// package.json
"pnpm": { "peerDependencyRules": { "ignoreMissing": ["@opentelemetry/sdk-logs", "@opentelemetry/sdk-trace-web"] } }