Overlap & backfill

Two timing edge cases every recurring job hits eventually: a run that's still going when the next firing is due, and firings that were missed while the process was down. Voltro makes both explicit policies on the schedule.

#Overlap — onOverlap

When a firing arrives and the previous run of the same schedule is still in flight:

Policy	Behaviour
skip (default)	Don't start a second run. Record a skipped row and move on.
queue	Serialize: wait for the in-flight run to finish, then run this one. Never concurrent, across every replica.
parallel	Start the new run immediately, alongside the old one.

defineSchedule({
  name: 'reindex',
  cron: '*/5 * * * *',
  timezone: 'UTC',
  onOverlap: 'skip',          // a slow reindex shouldn't pile up
  handler: async ({ app }) => { /* … */ },
})

Choosing:

• skip — idempotent or "latest state wins" jobs (reindex, cache warm). The default, and almost always right.
• queue — every firing's work matters and must happen in order (sequential batch processing). Runs serialize behind one another.
• parallel — runs are independent and you genuinely want concurrency (fan-out to per-tenant work).

queue caveat — unbounded growth. If a queue job consistently takes longer than its interval, the queue grows without bound and the schedule falls further behind. queue assumes runs are usually faster than the cadence, with occasional overruns. If runs are reliably slower than the interval, your cadence is wrong, not your overlap policy.

A manual Run now from the dashboard always runs, regardless of onOverlap — operators expect the button to fire.

#The watchdog — maxRuntimeMs

Every run races a watchdog (default 30 minutes). A run that exceeds it stops being awaited and is recorded failed with errorTag: 'timeout', so a run row never sits running forever.

defineSchedule({
  name: 'nightlyExport',
  cron: '0 2 * * *',
  timezone: 'UTC',
  maxRuntimeMs: 2 * 60 * 60_000,   // 2 hours — a big export
  handler: async ({ app }) => { /* … */ },
})

The watchdog stops waiting and records the timeout; it cannot truly abort a Promise's in-flight side effects (JavaScript has no thread-kill). Make long handlers cooperative — check a deadline, or do the heavy lifting in a workflow with its own step-level durability.

#Across replicas — the heartbeat

onOverlap is a cluster-wide rule, not a per-process one — for skip and for queue alike. A firing stands down, or takes its place in line, based on what every replica can see: a firing is suppressed (skip) or parked (queue) when an occurrence of the same schedule is running on any replica. That check reads _voltro_schedule_runs, so it needs a way to tell a run that is still working from one whose process died mid-run and left its row on running forever.

That is heartbeatAt: a run in flight bumps it every 30 seconds, and a row silent for three beats is read as dead rather than as a live occurrence. Without it the only evidence a row carried was firedAt, and a window generous enough never to cut off a long run is also long enough to suppress a half-hourly cron for hours after one restart.

// app.config.ts
export default defineAppConfig({
  scheduling: {
    scheduleHeartbeatMs: 30_000,   // default; VOLTRO_SCHEDULE_HEARTBEAT_MS overrides it
  },
})

Raise it in a deployment with many long-running schedules where the writes matter more than the detection latency; a run shorter than one interval writes no beat at all and costs nothing either way. It is not a lease — a stale beat only stops a corpse from holding the schedule shut; nothing takes ownership of the run.

The skip is logged with what it measured, so the two cases are legible from the log alone:

schedule: overlap skip  name=reindex scope=cluster heldBy=pod-7f4c lastSeenMs=1840

#queue — the line is durable

The coordination gate elects one replica per occurrence, and consecutive occurrences are independent races — so occurrence N lands on one pod and N+1 on another as a matter of course. A queue that lived in one process could not see the other's, which is why the line is a row, not a promise.

A firing that cannot start yet records itself as queued in _voltro_schedule_runs and waits until both hold:

1. no run of this schedule is live on any replica, and
2. no queued occurrence with an earlier scheduledAt is still waiting.

Then it takes the turn with a conditional update, so two replicas can never both conclude that they are next. Ordering is by scheduledAt — the cron-derived instant, identical on every replica — not by which pod noticed first.

A waiting firing beats its own row exactly like a running one does, so a pod that dies while queued is a corpse by the same rule — and the firing behind it adopts the occurrence rather than writing it off, running it in order, before its own, with that occurrence's own trigger and coordinationOutcome left intact. A rolling deploy is routine, so the alternative would mean losing the backlog on every deployment. The adopter stamps itself as the row's replicaId, and the same conditional write makes it safe against a waiter that was only slow: that one's own turn-taking write then finds nothing and it stands down.

Adoption is bounded, and the bound comes from the schedule's own cadence — twice its period, measured from the abandoned waiter's last beat. That is a deliberate refusal to pick a number, because no single one is right twice:

• The occurrence's age cannot be the measure. A deep queue is supposed to hold occurrences many periods old — that is what the policy means — so bounding by age would refuse to recover exactly the backlog queue exists to preserve.
• A fixed silence window cannot either. Nothing looks at an abandoned row until the next firing, so its silence is at least one period by construction: an hour is generous for a * * * * * schedule (sixty missed occurrences still run) and a coin flip for an hourly one.

A pod that dies holding an occurrence is silent for about one period when the next firing arrives — comfortably inside. An outage lasting several periods is outside, and the occurrence is recorded missed rather than run late. That is the answer a reader of backfill: 'skip' expects for work an outage stranded; backfill governs slots nobody ever claimed, this governs one a replica claimed and then died holding.

Set scheduling.queueAdoptionWindowMs (or VOLTRO_QUEUE_ADOPTION_WINDOW_MS) only to state a policy the cadence cannot express — "never run anything stranded for more than five minutes, whatever its period". It replaces the derived bound for every schedule.

A queued row whose schedule was renamed or deleted is written off at the next boot: its name never fires again, so nothing could ever adopt it, and the retention sweep skips queued rows. Only rows that are also silent are touched — a beating waiter under an unfamiliar name belongs to another deployment sharing the database.

The backlog is visible while it exists, and measured while it grows:

schedule: queued behind a run in flight   name=batchImport heldBy=pod-7f4c
schedule: queued behind an earlier firing name=batchImport behind=2026-09-13T10:05:00.000Z
schedule: adopted a queued firing from a replica that stopped reporting  name=batchImport runId=schrun_…

Metric	What it answers
voltro_schedule_queued	How many firings are parked right now, per schedule. Reported per replica — sum it across the fleet at query time.
voltro_schedule_queue_wait_seconds	How long a firing waited before it started. The duration histogram starts at firedAt, so the lateness a queue accumulates is invisible there.
voltro_schedule_queue_recoveries_total	Occurrences whose replica stopped reporting, by outcome. adopted is a rolling deploy working as intended — seeing it constantly means replicas are dying mid-wait. abandoned is the one to alert on: silent past the bound, recorded missed, and its work will not run.

A queued row is exempt from the run-ledger retention sweep, because it is a position in the line rather than a record of a firing: deleting one drops the work it represents.

Single-process deployments (dev, memory, sqlite — anything whose coordinationOutcome is single) keep the in-process chain and never write a queued row: there is no second replica to agree with.

A waiting firing re-checks when the run ledger changes, not only on its own cadence. Whether that reaches it is a property of the deployment: with changeStrategy: 'cdc' on postgres, or a broadcast transport, the blocking run's terminal write arrives as an event and the next occurrence starts on it — measured across two connections at under 400 ms, against just over a second when the same run has only the poll to wait for. Without either, the poll is the whole answer and the latency is that second; the interval is unchanged for exactly that reason.

#Backfill — backfill

When the process was down across one or more firing instants, what should happen on boot? Computed from the last _voltro_schedule_runs row for the schedule.

Policy	Behaviour
skip (default)	Ignore missed firings. Resume from the next future occurrence.
latest	Fire once to catch up to the most recent missed slot; record the older missed slots as missed (not silently dropped).
all	Fire every missed slot in order.

defineSchedule({
  name: 'dailyDigest',
  cron: '0 9 * * *',
  timezone: 'Europe/Berlin',
  backfill: 'latest',         // missed Tuesday's 9am after a deploy? send one catch-up, log the rest as missed
  handler: async ({ app }) => { /* … */ },
})

Choosing:

• skip — the firing was time-sensitive and a late run is worse than no run ("send the 9am alert" — 9am has passed, don't send it at noon).
• latest — you want the side effect to have happened recently, but replaying every missed slot would spam ("the digest should be reasonably current").
• all — every slot represents real work that must not be lost (per-period billing rollups). Dangerous for side-effecting jobs — a week of downtime means a week of catch-up firings. Opt in deliberately.

Backfill runs before the live timer is armed, so a caught-up firing never races the first scheduled one. The catch-up walk is capped (1000 slots) so a schedule that hasn't run in months doesn't enumerate forever.

Missed slots recorded under latest show up in the dashboard with the missed status — visible evidence of the gap, not a silent hole.

#Backfilling an explicit range — voltro schedule backfill

Boot backfill only walks forward from the last recorded run, and cluster-cron coordination caps its own catch-up at one day. A longer outage — or a schedule added after the fact that should have "always existed" — needs an explicit operator instruction naming the range:

voltro schedule backfill hourly-sync --from 2026-08-10T00:00:00Z --to 2026-08-12T00:00:00Z

Every cron occurrence in (from, to] fires sequentially, in order, each against its own cron-derived scheduledAt — a handler (or a workflow: payload function) reading ctx.scheduledAt computes against its slot, not "now". Firings record as trigger: 'manual' in _voltro_schedule_runs, so the catch-up is a legible ledger; a failed slot records its failure and the next slot still fires.

The verb is bounded and confirmable, because a range verb that can enqueue 100k runs is an outage generator:

• Above 25 occurrences it refuses and prints the count — re-run with --yes after reading the number.
• Above the per-request cap (default 1,000, raisable with --limit up to a hard ceiling of 10,000) it refuses outright, firing nothing — never a silent prefix that reports completeness. Run bigger catch-ups in slices.

The same verb is POST /_voltro/inspect/schedules/:name/backfill with { "from", "to", "confirm", "limit" } (a refusal answers 409 with the count and reason), on voltro dev and voltro serve alike — it needs the inspect surface open (VOLTRO_INSPECT_TOKEN).