Prometheus

@voltro/plugin-prometheus exposes the framework's metrics to Prometheus / Grafana at a scrape endpoint. It only READS — the framework emits the metrics automatically; the plugin formats them.

#The Metrics-API (single source of truth)

The framework records metrics into Effect's global MetricRegistry. One snapshot sees everything:

• Core RPC — voltro_rpc_requests_total{tag,status}, voltro_rpc_errors_total{tag}, voltro_rpc_duration_seconds{tag} (histogram). Emitted at the mutation / action handler boundary.
• HTTP routes — voltro_http_requests_total{route,status}, voltro_http_duration_seconds{route}.
• Plugin interceptors — voltro_plugin_hook_duration_seconds{hook}, voltro_plugin_hook_errors_total{hook}.
• Subscriptions — voltro_subscriptions_active{tag} (gauge of currently-open subscriptions), voltro_subscription_deliveries_total{tag,kind} + voltro_subscription_delivery_seconds{tag,kind} (per-delivery produce→push latency; kind = snapshot | delta). Backpressure & resume: voltro_subscription_buffered_bytes{tag} (gauge of pending bytes per blocked subscription), voltro_subscription_coalesced_total{tag} (updates collapsed onto the newest state while a consumer was blocked), voltro_subscription_overrun_total{tag} (streams closed with SubscriptionOverrun), voltro_subscription_oversized_total{tag} (events over reactive.socket.oversizedEventBytes — telemetry, not a cap), voltro_subscription_stalled_total{tag} (deliveries the transport did not accept within reactive.socket.stallWarnAfterMs — a consumer that is not acknowledging chunks).
• Schedules (crons) — voltro_schedule_runs_total{schedule,status} (firings by name + outcome — status = succeeded | failed), voltro_schedule_duration_seconds{schedule} (histogram), and voltro_schedule_last_success_timestamp_seconds{schedule} (a gauge holding the UNIX time of the last SUCCESS). Emitted by the framework scheduler, so every cron gets them with no per-handler wiring. A cron fires unattended — the failure mode is silent — so this is the series to alert on: time() - voltro_schedule_last_success_timestamp_seconds{schedule="…"} > <interval × N> fires when a job stops succeeding (a failure counter alone can't catch a job that stopped firing at all, but the last-success gauge going stale does). A failure moves the counter but deliberately NOT the gauge. The onOverlap: 'queue' backlog has its own three: voltro_schedule_queued{schedule} (a gauge of firings waiting for their turn, reported PER REPLICA so a fleet's depth is the sum at query time), voltro_schedule_queue_wait_seconds{schedule} (how long a firing waited before it started — invisible in the duration histogram, which starts at firedAt), and voltro_schedule_queue_recoveries_total{schedule,outcome} (a firing whose replica stopped reporting: adopted means the firing behind it ran the work, abandoned means it was silent past the adoption bound and the work will NOT run — that one is the alert). Externally driven schedules add two more: voltro_schedule_external_triggers_total{schedule,outcome} (every arrival, by what the app decided — a low duplicate rate is healthy and means a platform redelivery was absorbed, but a rate near 1 means your cluster is firing each occurrence twice and only the claim gate is keeping the work correct; noOccurrence at ANY rate means the cadence deployed to your scheduler is not the one the code declares) and voltro_schedule_external_missed_total{schedule} (occurrences nobody triggered — in external mode the app arms no timer, so a scheduler that stops firing is otherwise SILENT, and this is the series that tells that apart from a schedule with nothing to do).
• Workflows (durable execution) — voltro_workflow_runs_total{workflow,status} (terminal outcomes — status = succeeded | failed), voltro_workflow_duration_seconds{workflow} (histogram), and voltro_workflow_last_success_timestamp_seconds{workflow} (last-success gauge). Emitted by the workflow run-recording seam. Because the framework applies no retry of its own, a failed run is terminal — it is the dead-letter state — so voltro_workflow_runs_total{status="failed"} is the dead-letter rate, and the last-success gauge going stale is the "this workflow stopped completing" alert (same shape as the schedule alert). A failure moves the counter but not the gauge.
• Partial prerendering (web) — voltro_ppr_shell_serves_total{page}, voltro_ppr_hole_passes_total{page}, voltro_ppr_hole_settles_total{page}, voltro_ppr_hole_errors_total{page} and voltro_ppr_hole_pass_seconds{page} (histogram). page is the DECLARED route pattern (/blog/[slug]), never a resolved URL — a label that grows with visitors is how a scrape target falls over. voltro dev and voltro start emit the same set. voltro_ppr_hole_errors_total is the alert: the shell already went out with a 200, so a failed hole pass leaves every <Await> boundary on its fallback and nothing else says so.
• Queues — voltro_queue_consumed_total{topic,outcome} (outcome = ok | dead-lettered; the two together are every message the runner finished with, so the dead-letter RATE is a division with no join), voltro_queue_retries_total{topic}, voltro_queue_produced_total{topic}, and voltro_queue_lag_messages{topic,partition} — a gauge of the backlog behind the message just picked up, read out of the fetch response rather than an admin round trip. A message abandoned by a rebalance is deliberately in no outcome: its new owner redelivers and counts it there.
• @voltro/cache counters, Effect's own effect_fiber_* runtime metrics, and any custom metric you or another plugin defines.

Two consumers read the SAME snapshot, so they never disagree:

• GET /metrics (this plugin) → Prometheus text exposition format, consumed by Prometheus / Grafana.
• GET /_voltro/inspect/metrics → JSON MetricSample[], consumed by the dashboard's Metrics panel (quantiles derived from the histogram buckets, histogram_quantile-style).

#Custom metrics

Define your own counter / gauge / histogram and it shows up in both consumers automatically:

import { counter, gauge, histogramMetric } from '@voltro/runtime'
import { Effect, Metric } from 'effect'

const signups = counter('app_signups_total', 'User sign-ups.')

export default (input, _ctx) =>
  Effect.gen(function* () {
    // … create the user …
    yield* Metric.increment(signups)
    return { ok: true }
  })

#Wiring

// app.config.ts
import { prometheusPlugin } from '@voltro/plugin-prometheus'

export default {
  type: 'api' as const,
  name: 'api',
  plugins: [prometheusPlugin()],
}

Scrape config — one target per replica, never a single load-balanced hostname:

scrape_configs:
  - job_name: voltro
    metrics_path: /metrics
    static_configs:
      # one target per replica — NOT one LB-fronted hostname
      - targets: ['api-0.internal:4000', 'api-1.internal:4000']

#Multi-replica scraping

The registry is per-process: each replica serves its own counters, and Prometheus keeps the series apart via the per-target instance label — exactly the pull model. But that only works when every replica is scraped directly. Behind a load-balancer, successive scrapes hit different processes and the per-process counters interleave into one incoherent series. On Kubernetes, use pod-level discovery so every pod becomes its own target:

scrape_configs:
  - job_name: voltro
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'

The plugin's name option disambiguates multiple plugin instances within one process (it suffixes the plugin name) — it does not identify replicas; replica identity is the scrape target's instance label.

#Options

prometheusPlugin({
  path: '/metrics',          // scrape path (default /metrics)
  token: process.env.PROMETHEUS_TOKEN,  // require `Authorization: Bearer <token>`
  processMetrics: true,      // include node process_* metrics (default true)
  version: '1.4.2',          // sets voltro_build_info{version} = 1
})

• token — defaults to the PROMETHEUS_TOKEN env. Unset → the endpoint is open; gate it at the network layer (Prometheus uses bearer_token to send it). A path guard there can name /metrics as written — the route answers its exact spelling only, /Metrics and /metrics/ are 404.

Every activation reports the effective gate: metrics: locked at info level, or metrics: open — configure PROMETHEUS_TOKEN or restrict network access at warning level. Both include path and tokenGated, never the token. An empty explicit token also leaves the endpoint open. This warning does not infer that a network firewall exists: an ingress forwarding / may expose the scrape endpoint publicly. Set PROMETHEUS_TOKEN (and the matching scraper bearer credential), or verify the network restriction yourself. Check an unauthenticated request to the actual configured scrape path: a bearer-locked endpoint returns 401, not metrics.

• processMetrics — emits process_resident_memory_bytes, process_heap_bytes, process_cpu_seconds_total.
• The response is text/plain; version=0.0.4; charset=utf-8 with cache-control: no-store.

#Output shape

Counters / gauges render as name{labels} value. Histograms render as cumulative name_bucket{labels,le="…"} series plus name_sum + name_count + the +Inf bucket — exactly what histogram_quantile() expects in Grafana.

# HELP voltro_rpc_requests_total Total RPC handler invocations.
# TYPE voltro_rpc_requests_total counter
voltro_rpc_requests_total{status="ok",tag="mutation.todos.create"} 42
# TYPE voltro_rpc_duration_seconds histogram
voltro_rpc_duration_seconds_bucket{tag="mutation.todos.create",le="0.001"} 3
voltro_rpc_duration_seconds_bucket{tag="mutation.todos.create",le="+Inf"} 42
voltro_rpc_duration_seconds_sum{tag="mutation.todos.create"} 0.21
voltro_rpc_duration_seconds_count{tag="mutation.todos.create"} 42

#Scope

Mutation / action / HTTP / plugin metrics, subscription metrics (active gauge + per-delivery latency histogram, split snapshot vs delta), schedule/cron metrics AND workflow metrics (run counter + duration + last-success gauge each) are all covered. OTLP metrics export reads the same registry: the OTEL_EXPORTER_OTLP_ENDPOINT that enables trace export also exports metrics (@effect/opentelemetry bridges the registry into the OTel MeterProvider) — that is the path for the Datadog Agent / any OTLP collector; agentless Datadog is @voltro/plugin-datadog. This plugin is specifically the Prometheus pull (scrape) surface.