Overview

Every framework primitive emits OpenTelemetry spans when tracing is enabled. You don't instrument anything by hand — the handler boundaries, the transactional scope, and the streaming deliveries are all traced automatically, and the spans nest into a single waterfall per request.

#Auto-emitted spans

Span	What it marks
client.mutation.<rpcTag> / client.subscription.<rpcTag>	The browser-side span that roots the trace and sets the shared traceId.
mutation.<rpcTag>	The rpc mutation-handler boundary.
action.<rpcTag>	The unary action-handler boundary.
subscription.<rpcTag>	The streaming-rpc handler setup (subscribe → first snapshot). NOT the open-duration.
subscription.<rpcTag>.snapshot / .delta	Each data delivery to the subscriber, with its real produce→push latency.
store.transactional	The postgres BEGIN/COMMIT (or retry) scope. Carries db.system=postgresql + db.operation=transaction.
webhook<path>	An inbound *.webhook.tsx route (an incoming HTTP handler, not a query). Continues the caller's trace when the request carries a traceparent header.

#Span attributes

Handler spans carry:

• rpc.tag — the rpc tag of the call
• subject.type — the authenticated subject type
• tenant.id — the active tenant

@effect/rpc's own RpcServer.<tag> transport span is suppressed from the trace view — the framework spans carry the real latencies. For streaming subscriptions the transport span would just be a misleading open-duration bar, so the per-delivery .snapshot / .delta spans surface the real latency instead.

#Nesting

Spans nest. A mutation handler span wraps the store.transactional span, which wraps any Effect.withSpan you add inside the executor:

client.mutation.createOrder
└─ mutation.createOrder           rpc.tag, subject.type, tenant.id
   └─ store.transactional         db.system=postgresql, db.operation=transaction
      └─ createOrder.charge-card  (your custom span)

#Enabling modes

Three modes, env-driven, no code changes:

# Default — no EXPORTER. `voltro dev` still runs a buffer-only tracer
# (powers the Traces dashboards + traceId-in-logs); nothing is shipped
# off-box. `VOLTRO_TRACING_BUFFER=off` disables even that.
unset FRAMEWORK_TRACING OTEL_EXPORTER_OTLP_ENDPOINT

# Console (local dev — spans printed to stdout)
FRAMEWORK_TRACING=console voltro dev .

# OTLP (production — sends to any OTLP/HTTP collector: Jaeger, Tempo, Honeycomb, etc.)
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 voltro dev .
# OTEL_SERVICE_NAME defaults to 'voltro-api'; override per app:
OTEL_SERVICE_NAME=my-api OTEL_EXPORTER_OTLP_ENDPOINT=... voltro dev .

The framework auto-detects: if any standard OpenTelemetry env-var (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) is set, the mode resolves to otlp automatically. Set FRAMEWORK_TRACING explicitly (console / otlp / off) to force a mode.

Even with no exporter, voltro dev installs a buffer-only tracer — that's what powers the DevTools / cloud Traces panel and the trace id stamped on every log line. Set VOLTRO_TRACING_BUFFER=off to disable it entirely.

#Metrics export

Separate from tracing, the framework records metrics into Effect's global MetricRegistry — one source of truth (voltro_rpc_*, voltro_http_*, voltro_plugin_hook_*, voltro_subscription_* + voltro_subscriptions_active, voltro_db_*, voltro_ppr_*, voltro_queue_*, plus effect_fiber_* and any custom metric). Three ways to get them out:

The web server additionally exports partial prerendering, labelled by page (the declared route pattern, never a resolved URL): voltro_ppr_shell_serves_total{page}, voltro_ppr_hole_passes_total{page}, voltro_ppr_hole_settles_total{page}, voltro_ppr_hole_errors_total{page} and the voltro_ppr_hole_pass_seconds{page} histogram. voltro dev and voltro start emit the same set. Shell hit-rate is the isr cache's own x-voltro-cache HIT/STALE/MISS accounting — a ppr shell is a normal isr entry.

voltro_ppr_hole_errors_total is the one to alert on: a failed hole pass is invisible from outside. The shell is already on the wire with a 200, so the page renders and every <Await> boundary simply stays on its fallback — a page that looks like it is loading and never will. A single hole rejecting is not counted there; that settles the deferred-error envelope and renders the boundary's errorFallback.

# OTLP metrics — the SAME OTEL endpoint that enables trace export also enables
# metrics. Ships to any OTLP/HTTP collector (Prometheus OTLP, Grafana Agent,
# the Datadog Agent's OTLP port, etc.). @effect/opentelemetry auto-bridges the
# Effect MetricRegistry into the OTel MeterProvider — no per-metric wiring.
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 voltro start
# metrics-only endpoint (no traces):
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://collector:4318/v1/metrics voltro start
# console (local dev — periodic metric dump to stdout):
FRAMEWORK_METRICS=console voltro dev .
# export cadence (ms): OTEL_METRIC_EXPORT_INTERVAL (default 60000).

The framework auto-detects: OTEL_EXPORTER_OTLP_ENDPOINT (or the metrics-specific endpoint) → metrics export otlp; force with FRAMEWORK_METRICS (console / otlp / off).

Two pull/push consumers read the SAME registry, so they can't disagree:

• @voltro/plugin-prometheus — GET /metrics scrape endpoint (Prometheus / Grafana).
• @voltro/plugin-datadog — agentless push to Datadog's /api/v2/series (for setups without an Agent / OTLP collector). With an Agent, prefer pointing OTEL_EXPORTER_OTLP_ENDPOINT at it instead.
• The dashboard Metrics panel reads the same snapshot via GET /_voltro/inspect/metrics.

#Database metrics (voltro_db_*)

Every dialect store emits the same five series, so a dashboard built against one database keeps working after a migration to another. The labels are dialect (postgres · mysql · mariadb · mssql · sqlite · turso) and op (select · insert · update · delete · upsert · raw · transaction · ddl).

Series	Type	What it answers
voltro_db_queries_total	counter	Query rate, split by operation kind.
voltro_db_query_duration_seconds	histogram	p50/p95/p99 per operation — histogram_quantile over the buckets.
voltro_db_errors_total	counter	Statement failure rate.
voltro_db_operations_in_flight	gauge	Concurrency the framework is holding right now.
voltro_db_eager_fallback_total	counter	Eager loads that dropped off the single-roundtrip fast path.

No table name is ever a label. Table names grow with your schema, and a label that grows with the schema is how a scrape target falls over. The table appears in the log line instead.

voltro_db_operations_in_flight is not the driver's pool queue. It counts operations the framework currently has in flight, which is an upper bound on the connections it holds — the same number on every dialect. Your driver's own waiting count is not exposed. In practice you alert on this gauge sitting near your pool size together with the duration histogram's tail growing: a starved pool shows up as acquire time inside the query timing.

#voltro_db_eager_fallback_total — the one to alert on

An eager: query normally compiles to one round trip (a JSON aggregate). When it can't, the framework silently uses the portable multi-query walker instead — correct, and one round trip per relation level, on every call. That is a permanent per-query cliff with no error attached to it, which is why it is counted. The reason label separates the two very different cases:

• not-compilable — the query shape can never take the fast path (an unregistered relation, an ambiguous inferred foreign key, or an eager read under physical tenant isolation). Steady state. Worth knowing about, not worth paging on.
• execute-failed — the fast path compiled, ran, and threw, so the query paid for both paths. This is the one to alert on. It usually means a database or driver upgrade changed something under the JSON-aggregate query.

Both also log: a warn the first time a given table and reason are seen, then again at most every 5 minutes while it persists (VOLTRO_DB_EAGER_FALLBACK_WARN_INTERVAL_MS, 0 = once only). The counter is never rate-limited — the log line answers "is this happening now", the counter answers "has this been happening since the deploy three weeks ago".

#Routing traces to a vendor

The env-driven OTLP path above ships traces to any OTLP/HTTP collector. For deep, opt-in vendor integration — install one plugin, get traces + errors + logs all correlated by the same traceId, zero OTEL_* env — a plugin can contribute to the framework's tracer directly via contributeObservability:

interface ObservabilityContribution {
  resourceAttributes?: Record<string, string>
  spanProcessors?: ReadonlyArray<unknown>   // OTel SpanProcessor — receives every framework span
  metricReaders?: ReadonlyArray<unknown>    // OTel MetricReader
  sampler?: unknown                          // OTel Sampler (use with care — see below)
}

The CLI gathers every plugin's contribution at boot and passes the merged span-processors / metric-readers / resource-attributes into the framework's NodeSdk tracer. The vendor SDK runs as an OTel consumer — it gets a SpanProcessor alongside the framework's own buffer sink; it never becomes the global provider. So the in-app Traces dashboard keeps working and the vendor receives the identical spans.

This is how @voltro/plugin-sentry (SentrySpanProcessor) and @voltro/plugin-datadog (OTLPTraceExporter → DD Agent) route traces with no env fiddling.

Don't contribute a sampler that drops spans. The framework's buffer sink (powering the in-app Traces dashboard) is a SpanProcessor — it only sees recorded spans. A vendor sampler set as the tracer's sampler gates recording for the WHOLE tracer, blinding the dashboard. The framework stays always-on; vendors sample at their own export layer (e.g. Sentry's tracesSampleRate is applied by the Sentry client, not a tracer sampler).

To attach an error to the exact active span, the interceptor context carries both ctx.traceId and ctx.spanId — plugins read them explicitly (the active OTel span lives in Effect's fiber context, not the AsyncLocalStorage vendor SDKs read implicitly).

#Adding your own spans

Inside a mutation / action / workflow executor, just use Effect:

import { Effect } from 'effect'

export const myAction = defineAction({
  name: 'reports.rebuild',
  guards: [{ scope: 'reports:write' }],
  /* input, output */
})

export default (input, ctx) =>
  Effect.gen(function* () {
    yield* someExpensiveWork.pipe(
      Effect.withSpan('myAction.expensive-work', {
        attributes: { 'input.id': input.id },
      }),
    )
    return { ok: true }
  })

The span nests under the auto-emitted action.myAction span. In synchronous (non-Effect) executors, ad-hoc spans require explicitly adopting the Effect runtime — usually not worth it; rely on the auto spans the framework emits at each handler boundary.

Continue to Distributed tracing for how one traceId flows frontend → api → api, and Traces & logs from the shell for the voltro traces / voltro logs workflow.