What an inspect answer is about

Every 2xx /_voltro/inspect/* answer is an Observation: the payload plus what a reader needs in order to act on it.

{
  "data": { "…": "the payload" },
  "scope": { "kind": "process" },
  "origin": {
    "replicaId": "api-7d9f-x2k",
    "instanceId": "api-7d9f-x2k@1787893389882.k3f9aa",
    "startedAt": 1787893389882,
    "version": "0.56.0",
    "bootPath": "serve"
  },
  "completeness": { "complete": false, "fleetSize": 3, "reason": "process-scoped: this is 1 of 3 replicas" },
  "capturedAt": 1787893390411
}

#Why the payload alone was not enough

/subscriptions answers with the subscriptions of the one process that received the request. /schedules answers with the whole fleet's, read from the shared database. Both used to be plain JSON, so on a multi-replica deployment the first is an unlabelled sample and reads exactly like the second.

On one replica the difference is invisible — and one replica is every development environment, every e2e run and every template. So the environment in which the two look identical is the one everybody builds and tests in, and the difference only appears in production, where nobody can go and read the source to settle it.

#The four scopes

scope.kind	What it means	Examples
process	True of origin and of nothing else. Another replica answers differently.	/subscriptions, /metrics, /logs, /cache, /events, /traces, /cluster
shared-store	Read from storage every replica shares — any of them would answer the same.	/schedules, /workflows/*, /migrations, /database, /data/*
fleet	Assembled from more than one process. completeness says who answered.	/members, /subscriptions?scope=fleet
declaration	From the source tree. Identical on every replica of one version — and different across a rolling deploy.	/app, /routes, /manifest, /env, /rpc

declaration is its own kind rather than "fleet" on purpose: during a rolling deploy your fleet genuinely runs two versions, and origin.version is how you see which one answered.

#fleetSize — the label that needs no aggregation

A process-scoped answer carries how many replicas exist:

"completeness": { "complete": false, "fleetSize": 3, "reason": "process-scoped: this is 1 of 3 replicas" }

So a plain curl states that it is a fraction, and of what. Query a few times and union by origin.instanceId — which you can now do, because the answer says which process produced it.

When the field is missing, it means "could not tell", not "one". If the app runs no membership registry, fleetSize is absent and complete is false with a reason. Reporting 1 there would tell you that you are seeing the whole fleet.

#?scope=fleet

curl -s -H "authorization: Bearer $VOLTRO_INSPECT_TOKEN" \
  'localhost:4000/_voltro/inspect/subscriptions?scope=fleet' | jq .

Each replica publishes its counters into _voltro_replica_observations on a timer, so any replica can answer by reading rather than by asking the others. The answer is fleet-scoped and says what it is missing:

{
  "data": { "resume": [
    { "replicaId": "api-a", "version": "0.56.0", "ageMs": 4021, "payload": { "…": "counters" } },
    { "replicaId": "api-b", "version": "0.56.0", "ageMs": 9114, "payload": { "…": "counters" } }
  ] },
  "scope": { "kind": "fleet", "assembledBy": "api-a", "assembledAt": 1787893390411 },
  "completeness": {
    "complete": false, "responded": 2, "expected": 3, "missing": ["api-c"],
    "reason": "1 replica(s) have written nothing readable"
  }
}

Four properties are deliberate:

• A replica that has written nothing is missing, not absent. Dropping it would make a partial answer look complete — the same unlabelled sample, one level up and more expensive, because now you believe you asked everybody.
• A stale row is reported with its ageMs, not filtered out. Removing it hides that the answer is partial; keeping it unmarked presents fiction as current.
• A row from a replica that is GONE is departed, not stale. These are different facts with different next steps: a stale row belongs to a replica membership still lists, so it is running and has stopped refreshing — a fault, and a reason for complete: false. A departed row belongs to a process that exited; that is what a scale-down looks like, and it is not a reason for anything. Conflating them made complete unreachable after any replica had ever exited. They are still listed, with their age, because a reader who cannot see them cannot tell a fleet that scaled down from a table being written by something nobody is tracking. A departed row stays listed for 24 hours after its last write, then the retention sweep deletes it (VOLTRO_REPLICA_OBSERVATIONS_TTL_HOURS, at least 1). Live replicas rewrite their row every 30 seconds, so the sweep never reaches them. Without it, every process that ended without a graceful shutdown would stay listed forever — under voltro dev each boot is a new replica ID.
• Mixed versions are named in completeness.versions when the responders disagree. A rolling deploy spans two shapes, and averaging them silently is wrong in a way nothing downstream can detect. Departed replicas are excluded from the tally — a process that has been dead for hours reporting a "version split" is a line somebody acts on.

A single-process memory app answers with its own census. Otherwise, without shared discovery the request is refused with 501 and a reason — never answered with this replica's own numbers. Handing back a sample to someone who asked for the fleet in writing is exactly the failure the envelope exists to prevent.

#?replica=<id> — asking one named replica

# this replica's own answer (the default)
curl -s … /_voltro/inspect/subscriptions

# every replica's counters, assembled from the shared store
curl -s … '/_voltro/inspect/subscriptions?scope=fleet'

# ONE named replica, asked through the one you can reach
curl -s … '/_voltro/inspect/subscriptions?replica=api-7d9f-x2k'

The answer comes back with that replica's origin, process-scoped: the proxy does not launder whose answer it is.

The address is looked up in _voltro_replica_observations — the caller names an ID, never a URL, and an id we do not know produces a 404 with no request leaving the process. That is what keeps this from being an SSRF primitive. "unknown replica" and "that replica published no reachable address" give the same message on purpose: telling them apart would tell a caller which ids exist.

A replica publishes an address only when it is genuinely reachable by a peer. An unset POD_IP falls back to 127.0.0.1, which is a shrug rather than a statement, so it is recorded as not reachable and no peer will try it. Set VOLTRO_INSPECT_ADVERTISE_HOST to declare one — including 127.0.0.1, when the peers really are on this machine.

A proxied request carries a hop header and is always answered locally, so ?replica= cannot cycle. A peer that does not answer inside a short deadline becomes a 504 naming it, because a diagnostic that hangs is worse than one that says no.

#Select several replicas

Use repeated replica parameters for an explicit subset, or replicas=all to read the discovered fleet. Do not combine the two forms. Authentication runs before discovery; only published peer addresses are used. Reads have bounded concurrency and timeouts, and a cancelled request stops its outstanding reads.

curl -s 'http://localhost:4000/_voltro/inspect/logs?replica=pod-a&replica=pod-b'
curl -s 'http://localhost:4000/_voltro/inspect/metrics?replicas=all'

For process observations, the response contains data.results: each entry names the requested replica and contains either its observation or an unavailable status. completeness counts requested, answered, missing and undiscovered replicas. Shared-store endpoints return one observation, never duplicated rows. The dashboards project these results according to the domain; see Select replicas.

A process SSE connection accepts one replica ID. Open one connection per selected process. A shared SQL CDC feed stays unselected.

The stream answers a selection before it starts streaming, with the same refusals as a read: an unknown ID, or one without a reachable address, is a 404 with no request leaving the process; a peer that does not accept the connection inside the deadline is a 504; a peer that answers with another process's feed is a 502. A refused stream is a JSON response, never an event stream that closes at once, and it leaves no error in the receiving process's log. A peer that disconnects later ends the stream; the next connection reports its new state.

#Writes are never fleet-addressable

?scope=fleet and ?replica= exist only for reads. A mutating endpoint (/invoke, /seeds/run, /data/import, /migrations/rollback, /agent/call) rejects them: fanning a destructive operation out across a fleet is not something an accidental query parameter should be able to ask for.

#Reading it from your own tooling

# the payload
curl -s … /_voltro/inspect/subscriptions | jq .data.resume

# is this the whole picture?
curl -s … /_voltro/inspect/subscriptions | jq '.completeness | {complete, fleetSize}'

# which pod answered?
curl -s … /_voltro/inspect/subscriptions | jq -r .origin.replicaId

capturedAt is the origin's clock. Do not compare it with another replica's — replica clocks disagree, which is why the fleet view reports an ageMs computed against one reader's instant rather than a timestamp you are invited to subtract.

#The live stream

/_voltro/inspect/stream (SSE) carries origin on every event, not on a handshake. A consumer that connects late — a reconnect, a second tab, a curl piped into jq — never sees a handshake, and this is a live stream from whichever replica the connection landed on. Without the per-event stamp, a tail on a three-replica fleet shows a third of it continuously, with nothing on the wire to say so.

ts is that replica's clock. Do not order two origins by it.

It is mounted on both boot paths. voltro serve used to answer 404 here — the stream was wired for voltro dev and nowhere else — so every live view in the dashboard worked in development and was dead in production. Both paths now mount it through one builder.

Authenticating it needs no special step from you, and one from the dashboard. EventSource accepts no headers, so a browser cannot attach a bearer to an SSE request the way it does to every other inspect call. The dashboard passes the app's token through a same-origin cookie scoped to its own proxy path, cleared the moment the stream opens; the proxy moves it into an Authorization header. Nothing about the token ever appears in a URL, and the app receives a header like any other caller.

That dashboard proxy is also why a production app page must not open the inspect stream merely because localStorage.framework_debug is set. The app page has no bearer-capable EventSource transport and no authority to mint the dashboard cookie. Voltro therefore keeps its in-page console and SSE relays development-only; the persisted production flag affects local framework console output, not inspect traffic.

#In the dashboards

Both dashboards unwrap .data in their fetch layer and keep the envelope. A page whose data is process-local renders a notice saying which replica it is showing and how many exist; a page whose answer is complete renders nothing, because a banner over a complete answer teaches people to ignore banners.

#The Fleet panel

Both dashboards ship a Fleet page — the self-hosted DevTools under /apps/<id>/fleet, the hosted console under the app's Fleet tab. One page, one shared component, two transports.

It renders three populations, and the two after the first are the point:

• the replicas that answered, each with its version, how long ago it published, and whether a peer can reach it;
• the replicas that are silent — membership knows them and they have published nothing readable. Shown as their own section rather than omitted, because "not shown" and "not there" look identical and mean opposite things;
• the replicas whose answer is stale, with the age. Filtering them would make a partial answer look complete; leaving them unmarked would present old numbers as current.

A mixed-version note appears when the responders disagree — a rolling deploy is in flight and the numbers span two shapes.

The completeness banner is phrased as a ratio ("3 of 5 replicas answered") rather than a count, because a count invites the reader to believe that is the fleet.

#Data explorer refresh state

Cloud and local dashboards refetch the selected page when its table emits a CDC event and after every successful stream connection, including reconnects. A disconnected stream displays a reconnecting notice; snapshots continue to refresh every 15 seconds. This is a connection state, not evidence that every replica's changes reached the observed process. The upstream Observation still states which data scope and origin produced the snapshot.

Failed reads replace the prior rows with an error. Changing the app, its inspect credentials, filters, sort, page or count mode discards responses from the prior selection. Empty pages retain the source's total and count mode; none remains unknown rather than becoming zero. Cloud stores page metadata and rows in one transaction before publishing reactive changes.

Local live streams use a same-origin proxy, and every stream a dashboard page holds shares one browser connection: the page posts its targets, each with its own app bearer, to /api/dashboard/proxy/streams and receives one event stream that labels each event with its target. Over HTTP/1.1 a browser opens at most six connections per origin, across all tabs; one connection per app used them all up at six apps and left every other dashboard request waiting. Each app reconnects on its own backoff behind that connection, and every successful handshake is reported, so consumers resync after it. Bearers travel in the request body, never in stream cookies or URLs.

Shared dashboard providers may use DataSource.observation when the payload is an array and cannot retain envelope fields during serialization. The metrics page reads this field; object-shaped inspect results retain _observation. Cloud's RPC output schemas preserve scope metadata, including for the standing expectation, budget, experiment, check and agent-tool inventories. Their collection timestamp alone is not a statement about fleet completeness.

Local inspect pages serialize refreshes and bind results to the selected app, URL and read credential. A failed refresh clears the previous snapshot. Cloud waits for app resolution before opening inspect subscriptions. Fleet reads require assembly and completeness metadata; missing evidence is an error, not a zero-replica fleet. Departed replicas remain visible separately.

Web /metrics and /metrics/registry preserve nonfinite values as +Inf, -Inf or NaN strings. Histogram quantiles use finite bucket boundaries. Unscoped console records are valid log entries; their missing logger scope does not invalidate the snapshot.

The namespace directory uses a deterministic ID for each physical namespace. Its database declaration uses the ID type, so restarting an app does not require a primary-key type conversion.

Replica collection limits concurrent reads to four and bounds the complete collection to ten seconds. Answers already received remain visible; replicas that could not answer within that budget are listed as unavailable, including queued reads. Shared-store failover uses the same total budget. Plugin inspect data is treated as process-local because plugins may hold local buffers or configuration; selected replicas are compared separately.

Web and SSR apps use the same replica selector in Cloud and local Devtools. voltro dev discovers processes of the same real project directory on this machine. Each process publishes its own heartbeat; stale processes remain labelled and are not used as routing targets.

For discovery across hosts, configure the same VOLTRO_WEB_REPLICA_GROUP for replicas of one app and environment, a PostgreSQL connection (DB_URL or the discrete database fields), and each process's POD_IP or VOLTRO_INSPECT_ADVERTISE_HOST. For example:

VOLTRO_WEB_REPLICA_GROUP=storefront-production
DB_URL=postgres://app:password@database:5432/app
VOLTRO_INSPECT_ADVERTISE_HOST=web-1.internal
VOLTRO_REPLICA_ID=web-1
VOLTRO_INSPECT_TOKEN=shared-inspect-read-secret

Use a different group for every app/environment, and a different advertised host and replica ID for every process. The CLI creates public._voltro_web_replicas; this registry needs PostgreSQL table creation and read/write privileges. Both voltro dev and voltro start publish every ten seconds, consider heartbeats stale after thirty seconds, and remove only their own process generation on shutdown. All replicas must be able to reach each other's advertised inspect port and use the same read token. PostgreSQL measures heartbeat age, so host clock differences do not extend freshness.

Production without a configured discovery group reports discovery as unavailable rather than presenting one load-balancer response as the whole fleet. Explicit reads of a known local replica still work. Process-local logs, routes, cache and metrics respect the selection; selected live streams verify the responder's replica ID.

#Addressing one replica for an action

/_voltro/inspect/target/<inspect-suffix>?targetReplica=<replica-id> addresses exactly one process for an Inspect GET, POST, PATCH or DELETE, including binary downloads. It is separate from the replica / replicas read-selection parameters and cannot be combined with them. The server resolves the ID through its registered peer addresses, checks authorization for the actual HTTP method, and verifies the target again on the receiving process. Unknown targets and redirects fail; there is no fallback or automatic mutation retry.

The caller's credentials are forwarded without elevation. Binary bodies remain streams, with upstream cancellation and a transfer deadline. If delivery fails, a write may already have been applied: inspect its outcome before retrying. This server primitive does not by itself bind a separate upload-ticket exchange or dashboard action to a process; those adapters must carry the target throughout their operation.

For example, address a storage deletion through POST /_voltro/inspect/target/plugins/storage/delete?targetReplica=<replica-id>. The target dispatcher removes the target/ segment only on the addressed process. Older servers without this dispatcher return 404 instead of ignoring an unfamiliar query parameter and executing the original mutation. A targetReplica query on the ordinary Inspect path is rejected.

Targeted writes forward both the read bearer and the separate x-voltro-inspect-write credential; the receiving process checks both again. Targeted downloads forward only the read credential.

The shared HTTP listener supports exact binary Inspect paths separately from its UTF-8 JSON handlers. Development and production mount the storage upload relay before text decoding. The relay accepts only a signed, replica-bound upload ticket, enforces its byte limit, and forwards once to the registered peer. Inspect shutdown and the development host guard apply to this entrance too. Both dashboard hosts send upload admission to the selected replica and transfer the bytes through this relay. Oversized transfers receive HTTP 413 before the connection closes.