Background tasks
What runs on a timer, and how to run it when there is more than one instance.
Ten tasks run on a timer. Two of them write money.
| Task | Every | Does |
|---|---|---|
webhook-deliveries | 10s | Attempts every delivery that is due |
prune-auth-codes | 15m | Drops spent and expired authorization codes |
prune-oauth-flows | 15m | Drops social sign-ins that were never finished |
prune-oidc | 15m | Abandoned sign-ins, dead tokens, retired keys |
renew-subscriptions | 1h | The billing run. Invoices and rolls periods |
collect-invoices | 1h | Charges what the billing run raised |
expire-quotes | 1h | Tidies the status column on lapsed quotes |
end-expired-trials | 1h | Converts — or downgrades — a finished trial |
refresh-sso-discovery | 6h | Re-reads each customer IdP's discovery document |
prune-audit-logs | 24h | Ages out audit entries after a year |
prune-sns-messages | 24h | Drops processed SES feedback claims after 30 days |
One task, one process
Every tick takes a lease before it runs. A task already running somewhere else is skipped, silently — that is the ordinary case on every replica but one.
This is not tidiness. renewSubscription reads a subscription, checks its
period is over, and writes an invoice, with nothing holding the row in between.
Two sweeps at the same moment both pass the check and both bill, and nothing
downstream catches it: the only unique constraint on an invoice is its number,
and a number is minted per invoice.
Without the lease, scaling the API past one replica double-bills every subscription due that hour. The lease is what makes replicas safe — it is not an optimisation.
The lease is a row in scheduler_lock, so an operator can see what holds what:
SELECT name, holder, "expiresAt" FROM scheduler_lock ORDER BY name;A holder is host:pid:random. An empty table means nothing is running right
now, which is normal between ticks.
Where it runs
By default the scheduler starts inside the API, which is right for a single instance and for development.
pnpm --filter api start # API, with the schedulerFor a deployment, run it as its own workload and take it out of the API:
BASE316_SCHEDULER=off pnpm --filter api start # API, serving requests only
pnpm --filter api scheduler # the tasks, on their ownTwo reasons to split rather than leave it on everywhere. A billing sweep sharing a pod with request traffic is a latency spike nobody can explain from the outside. And the API is then free to scale on load, because nothing about its replica count touches billing any more.
The value must be exactly off. Anything else — unset, false, 0 — leaves
the scheduler running, because the safe reading of an ambiguous value is the one
where the invoices still go out.
If a task stops running
A lease outlives its holder by design: a process killed mid-sweep would otherwise block the task for ever. The lease is five minutes for most tasks and thirty for the two billing ones, renewed continuously while the task runs, so the worst case is that a task waits one lease before another instance picks it up.
If a task appears stuck for longer than that, the row says who has it:
SELECT * FROM scheduler_lock WHERE name = 'renew-subscriptions';An expiresAt in the past means nothing holds it and the next tick will take
it. An expiresAt that keeps moving forward means the task genuinely is still
running. Deleting the row is safe but rarely the answer — it does not stop the
process that holds it, so it invites exactly the concurrent run the lease
prevents.
Notes
- A missed run catches up on the next tick rather than being skipped. The billing run is hourly rather than daily for the same reason: a period ending at 09:00 is invoiced within the hour, not at midnight.
expire-quotesis cosmetic.effectiveStatusalready refuses a lapsed quote, so if this task never runs, nothing is mis-accepted — the reporting just reads oddly.- Tasks are timed from the end of the previous run, so a slow drain cannot pile up concurrent copies inside one process either.