What an inspect answer is about

Every 2xx /_voltro/inspect/* answer is an Observation: the payload plus what a reader needs in order to act on it.

{
  "data": { "…": "the payload" },
  "scope": { "kind": "process" },
  "origin": {
    "replicaId": "api-7d9f-x2k",
    "instanceId": "api-7d9f-x2k@1787893389882.k3f9aa",
    "startedAt": 1787893389882,
    "version": "0.56.0",
    "bootPath": "serve"
  },
  "completeness": { "complete": false, "fleetSize": 3, "reason": "process-scoped: this is 1 of 3 replicas" },
  "capturedAt": 1787893390411
}

#Why the payload alone was not enough

/subscriptions answers with the subscriptions of the one process that received the request. /schedules answers with the whole fleet's, read from the shared database. Both used to be plain JSON, so on a multi-replica deployment the first is an unlabelled sample and reads exactly like the second.

On one replica the difference is invisible — and one replica is every development environment, every e2e run and every template. So the environment in which the two look identical is the one everybody builds and tests in, and the difference only appears in production, where nobody can go and read the source to settle it.

#The four scopes

scope.kind	What it means	Examples
process	True of origin and of nothing else. Another replica answers differently.	/subscriptions, /metrics, /logs, /cache, /events, /traces, /cluster
shared-store	Read from storage every replica shares — any of them would answer the same.	/schedules, /workflows/*, /migrations, /database, /data/*
fleet	Assembled from more than one process. completeness says who answered.	/members, /subscriptions?scope=fleet
declaration	From the source tree. Identical on every replica of one version — and different across a rolling deploy.	/app, /routes, /manifest, /env, /rpc

declaration is its own kind rather than "fleet" on purpose: during a rolling deploy your fleet genuinely runs two versions, and origin.version is how you see which one answered.

#fleetSize — the label that needs no aggregation

A process-scoped answer carries how many replicas exist:

"completeness": { "complete": false, "fleetSize": 3, "reason": "process-scoped: this is 1 of 3 replicas" }

So a plain curl states that it is a fraction, and of what. Query a few times and union by origin.instanceId — which you can now do, because the answer says which process produced it.

When the field is missing, it means "could not tell", not "one". If the app runs no membership registry, fleetSize is absent and complete is false with a reason. Reporting 1 there would tell you that you are seeing the whole fleet.

#?scope=fleet

curl -s -H "authorization: Bearer $VOLTRO_INSPECT_TOKEN" \
  'localhost:4000/_voltro/inspect/subscriptions?scope=fleet' | jq .

Each replica publishes its counters into _voltro_replica_observations on a timer, so any replica can answer by reading rather than by asking the others. The answer is fleet-scoped and says what it is missing:

{
  "data": { "resume": [
    { "replicaId": "api-a", "version": "0.56.0", "ageMs": 4021, "payload": { "…": "counters" } },
    { "replicaId": "api-b", "version": "0.56.0", "ageMs": 9114, "payload": { "…": "counters" } }
  ] },
  "scope": { "kind": "fleet", "assembledBy": "api-a", "assembledAt": 1787893390411 },
  "completeness": {
    "complete": false, "responded": 2, "expected": 3, "missing": ["api-c"],
    "reason": "1 replica(s) have written nothing readable"
  }
}

Four properties are deliberate:

• A replica that has written nothing is missing, not absent. Dropping it would make a partial answer look complete — the same unlabelled sample, one level up and more expensive, because now you believe you asked everybody.
• A stale row is reported with its ageMs, not filtered out. Removing it hides that the answer is partial; keeping it unmarked presents fiction as current.
• A row from a replica that is GONE is departed, not stale. These are different facts with different next steps: a stale row belongs to a replica membership still lists, so it is running and has stopped refreshing — a fault, and a reason for complete: false. A departed row belongs to a process that exited; that is what a scale-down looks like, and it is not a reason for anything. Conflating them made complete unreachable after any replica had ever exited. They are still listed, with their age, because a reader who cannot see them cannot tell a fleet that scaled down from a table being written by something nobody is tracking.
• Mixed versions are named in completeness.versions when the responders disagree. A rolling deploy spans two shapes, and averaging them silently is wrong in a way nothing downstream can detect. Departed replicas are excluded from the tally — a process that has been dead for hours reporting a "version split" is a line somebody acts on.

If there is no shared store, the request is refused with 501 and a reason — never answered with this replica's own numbers. Handing back a sample to someone who asked for the fleet in writing is exactly the failure the envelope exists to prevent.

#?replica=<id> — asking one named replica

# this replica's own answer (the default)
curl -s … /_voltro/inspect/subscriptions

# every replica's counters, assembled from the shared store
curl -s … '/_voltro/inspect/subscriptions?scope=fleet'

# ONE named replica, asked through the one you can reach
curl -s … '/_voltro/inspect/subscriptions?replica=api-7d9f-x2k'

The answer comes back with that replica's origin, process-scoped: the proxy does not launder whose answer it is.

The address is looked up in _voltro_replica_observations — the caller names an ID, never a URL, and an id we do not know produces a 404 with no request leaving the process. That is what keeps this from being an SSRF primitive. "unknown replica" and "that replica published no reachable address" give the same message on purpose: telling them apart would tell a caller which ids exist.

A replica publishes an address only when it is genuinely reachable by a peer. An unset POD_IP falls back to 127.0.0.1, which is a shrug rather than a statement, so it is recorded as not reachable and no peer will try it. Set VOLTRO_INSPECT_ADVERTISE_HOST to declare one — including 127.0.0.1, when the peers really are on this machine.

A proxied request carries a hop header and is always answered locally, so ?replica= cannot cycle. A peer that does not answer inside a short deadline becomes a 504 naming it, because a diagnostic that hangs is worse than one that says no.

#Writes are never fleet-addressable

?scope=fleet and ?replica= exist only for reads. A mutating endpoint (/invoke, /seeds/run, /data/import, /migrations/rollback, /agent/call) rejects them: fanning a destructive operation out across a fleet is not something an accidental query parameter should be able to ask for.

#Reading it from your own tooling

# the payload
curl -s … /_voltro/inspect/subscriptions | jq .data.resume

# is this the whole picture?
curl -s … /_voltro/inspect/subscriptions | jq '.completeness | {complete, fleetSize}'

# which pod answered?
curl -s … /_voltro/inspect/subscriptions | jq -r .origin.replicaId

capturedAt is the origin's clock. Do not compare it with another replica's — replica clocks disagree, which is why the fleet view reports an ageMs computed against one reader's instant rather than a timestamp you are invited to subtract.

#The live stream

/_voltro/inspect/stream (SSE) carries origin on every event, not on a handshake. A consumer that connects late — a reconnect, a second tab, a curl piped into jq — never sees a handshake, and this is a live stream from whichever replica the connection landed on. Without the per-event stamp, a tail on a three-replica fleet shows a third of it continuously, with nothing on the wire to say so.

ts is that replica's clock. Do not order two origins by it.

It is mounted on both boot paths. voltro serve used to answer 404 here — the stream was wired for voltro dev and nowhere else — so every live view in the dashboard worked in development and was dead in production. Both paths now mount it through one builder.

Authenticating it needs no special step from you, and one from the dashboard. EventSource accepts no headers, so a browser cannot attach a bearer to an SSE request the way it does to every other inspect call. The dashboard passes the app's token through a same-origin cookie scoped to its own proxy path, cleared the moment the stream opens; the proxy moves it into an Authorization header. Nothing about the token ever appears in a URL, and the app receives a header like any other caller.

#In the dashboards

Both dashboards unwrap .data in their fetch layer and keep the envelope. A page whose data is process-local renders a notice saying which replica it is showing and how many exist; a page whose answer is complete renders nothing, because a banner over a complete answer teaches people to ignore banners.

#The Fleet panel

Both dashboards ship a Fleet page — the self-hosted DevTools under /apps/<id>/fleet, the hosted console under the app's Fleet tab. One page, one shared component, two transports.

It renders three populations, and the two after the first are the point:

• the replicas that answered, each with its version, how long ago it published, and whether a peer can reach it;
• the replicas that are silent — membership knows them and they have published nothing readable. Shown as their own section rather than omitted, because "not shown" and "not there" look identical and mean opposite things;
• the replicas whose answer is stale, with the age. Filtering them would make a partial answer look complete; leaving them unmarked would present old numbers as current.

A mixed-version note appears when the responders disagree — a rolling deploy is in flight and the numbers span two shapes.

The completeness banner is phrased as a ratio ("3 of 5 replicas answered") rather than a count, because a count invites the reader to believe that is the fleet.