- Issues
- #144
alarm() handlers can overlap: a due alarm is re-dispatched while its handler is still running
ewhauser opened ·
## Summary
A due alarm can be dispatched a second time while the first `alarm()` invocation is still executing, producing two concurrent handler invocations for one scheduled alarm. Any handler that awaits I/O (an outbound `fetch`, in our case) hits this reliably. Cloudflare's Durable Objects never overlap alarm handlers, so DO code written to that contract silently double-fires side effects on celld.
## Environment
- celld v0.2.0, commit `3f22aedd1ea4d413b93e84afb1ce385f04be84f1`, built from source - macOS 26.5.2 arm64, 3-node fleet, MinIO (RELEASE.2025-09-07) as the bucket - Bundle is plain Cloudflare DO code (`new_sqlite_classes`), deployed with `celld deploy`
## Observed
Our DO's `alarm()` claims a batch from storage, POSTs a webhook, then updates storage. Every alarm produced **two deliveries with the same logical run id from two distinct invocation ids**, the second entering ~8–11 ms after the first — consistently right as the first invocation's outbound fetch was awaited. Tagging each invocation with a random id and logging enter/exit shows two `alarm()` executions with overlapping windows for a single `setAlarm`.
## Root cause (from reading the source at this commit)
- `due_alarm_entry` (crates/celld/storage.rs) selects `_cf_ALARM WHERE at_ms <= now` — the row remains present for the whole handler execution. - `begin_alarm_handler` records the in-flight handler in the `active_alarms` map, but the due-selection path never consults it, so the cell's self-poll ("each cell thread self-polls its own db between requests") hands the same due row out again while the first handler is still running. - The row is only deleted in `finish_alarm_handler`, on success, when the handler *ends*.
So the overlap window is exactly the handler's duration; any await inside `alarm()` widens it past the poll interval. Cloudflare's semantics are that the alarm is consumed at dispatch and handlers are serialized, with retry only after a handler failure.
## Minimal repro (distilled from our case)
```js export class Overlap { constructor(state) { this.state = state; } async fetch(req) { if (new URL(req.url).pathname === "/arm") { await this.state.storage.put("log", []); await this.state.storage.setAlarm(Date.now() + 1000); return new Response("armed\n"); } return Response.json(await this.state.storage.get("log")); } async alarm() { const id = crypto.randomUUID(); let log = (await this.state.storage.get("log")) ?? []; await this.state.storage.put("log", [...log, { id, enter: Date.now() }]); await fetch("https://example.com/"); // any awaited I/O log = (await this.state.storage.get("log")) ?? []; await this.state.storage.put("log", [...log, { id, exit: Date.now() }]); } } ```
Arm once, wait, read the log: two distinct `id`s with interleaved enter/exit timestamps. On Cloudflare: exactly one.
## Expected
One `alarm()` invocation per due alarm; a second dispatch only after the first completes with a failure (the retry path), never concurrently.
## Workaround
Wrapping the handler body in `state.blockConcurrencyWhile()` restores serialization (the second dispatch queues behind the input gate and finds the alarm cleared). One subtlety for anyone else doing this: carry failures *out* of the block and rethrow after it, because a rejected `blockConcurrencyWhile` resets the actor and the rethrow is what lets the alarm retry ladder see the failure.
A possible fix shape, from the outside: have the due-selection skip scopes present in `active_alarms` (the bookkeeping already exists), or mark/consume the row at dispatch as Cloudflare does. Happy to send a `git format-patch` per the contribution policy if that's useful — and thanks; aside from this, the alarm implementation (durable arm gating, the wake-entry reconciliation rules) has held up impressively under our fault-injection testing.
ewhauserreferenced this in feat(eve-ambient): optional celld-backed correlation mailbox #453 months ago on GitHub
- RY
Thanks for the notice. This has been fixed and will be in the next release.
ewhauserreferenced this in Design: full-payload, end-to-end idempotent event handoffs #33 months ago on GitHub
- BA
This was fixed in v0.2.1
bartlomiejureferenced 9c0ca09ef5d53 months ago on GitHub
bartlomiejureferenced bc0e29cddf303 months ago on GitHub
bartlomiejureferenced 89e4ffc53a143 months ago on GitHub