denoland/celldPublic

self-hosted, distributed Durable Objects

v0.5.0 regression: a pending timer in alarm() delays re-arms by ~15 s

FU

The following is AI-written (Claude Opus 5.5). It caught this in our e2e suite after we upgraded from 0.4.0, reduced it to the repro below, and traced the cause.

## Summary

Since v0.5.0, an `alarm()` handler that returns while a timer it started is still pending keeps the alarm in its firing state until that timer ends or `CELLD_OPERATION_DEADLINE_MS` (15 s by default) expires it. A `setAlarm()` made in the meantime, including the handler re-arming itself, fires only after that.

The everyday trigger is `await fetch(url, { signal: AbortSignal.timeout(30_000) })` inside `alarm()`: the fetch finishes in milliseconds, but the signal's timer is still pending. A cron- or watchdog-style alarm that re-arms itself 100 ms out then fires once every ~15.5 s. Nothing reaches the application: no error, no `isRetry`. v0.4.1 does not do this.

## Environment

- celld v0.5.0 (`12d5b63`) and v0.5.1 (`42269c1`): attested release binaries, and a v0.5.1 source build. v0.4.0 (`a52f990`) and v0.4.1 (`10cb130`) release binaries for comparison. - macOS 26.6.2 arm64, `celld dev` (one node, local store). `celld dev` runs the ordinary node (`celld --no-control-plane …`), so the code path below is the one a fleet node runs.

## Repro

`wrangler.jsonc`:

```jsonc { "name": "alarm-rearm-repro", "main": "index.js", "compatibility_date": "2026-01-01", "durable_objects": { "bindings": [{ "name": "PROBE", "class_name": "Probe" }] }, "migrations": [{ "tag": "v1", "new_sqlite_classes": ["Probe"] }] } ```

`index.js`:

```js import { DurableObject } from "cloudflare:workers";

export class Probe extends DurableObject { constructor(ctx, env) { super(ctx, env); ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS fired (at INTEGER NOT NULL)"); }

async fetch(request) { const url = new URL(request.url); if (url.pathname === "/start") { this.ctx.storage.kv.put("leave_timer", url.searchParams.get("timer") === "1"); await this.ctx.storage.setAlarm(Date.now()); return new Response("started\n"); } // Milliseconds from the first firing to each firing. const fired = this.ctx.storage.sql.exec("SELECT at FROM fired").toArray().map((r) => r.at); return Response.json(fired.map((at) => at - fired[0])); }

// Fires three times, re-arming itself 100 ms out, like a cron or a watchdog. async alarm() { this.ctx.storage.sql.exec("INSERT INTO fired (at) VALUES (?)", Date.now()); // What `await fetch(url, { signal: AbortSignal.timeout(30_000) })` leaves // behind after the fetch has finished: a pending timer. if (this.ctx.storage.kv.get("leave_timer")) AbortSignal.timeout(60_000); const n = this.ctx.storage.sql.exec("SELECT COUNT(*) AS n FROM fired").one().n; if (n < 3) await this.ctx.storage.setAlarm(Date.now() + 100); } }

export default { fetch(request, env) { const name = new URL(request.url).searchParams.get("name") ?? "a"; return env.PROBE.getByName(name).fetch(request); }, }; ```

```sh celld dev --port 8795 & curl -s 'localhost:8795/start?name=a' # alarm() leaves nothing pending curl -s 'localhost:8795/start?name=b&timer=1' # alarm() leaves an AbortSignal.timeout pending sleep 35 curl -s 'localhost:8795/?name=a' curl -s 'localhost:8795/?name=b' ```

Milliseconds from the first firing to each firing:

| | `a`: nothing pending | `b`: `AbortSignal.timeout(60_000)` pending | |---|---|---| | v0.4.1 | `[0,104,206]` | `[0,103,205]` | | v0.5.1 | `[0,104,206]` | `[0,15503,31008]` | | v0.5.1 + the fix below | `[0,103,206]` | `[0,104,208]` |

A fuller harness, in which a request re-arms the alarm to "now" 500 ms after a firing, shows the same thing. The figure is how long the re-armed alarm took to run:

| `alarm()` leaves pending | v0.4.0 | v0.5.1 | v0.5.1 + fix | |---|---|---|---| | nothing | 3 ms | 2 ms | 3 ms | | `setTimeout(() => {}, 60_000)` | 2 ms | 14,528 ms | 2 ms | | `AbortSignal.timeout(60_000)` | 3 ms | 14,490 ms | 3 ms | | a finished `fetch(…, { signal: AbortSignal.timeout(60_000) })` | 6 ms | 14,500 ms | 2 ms | | `setTimeout(() => {}, 3_000)` | 3 ms | 2,491 ms | 6 ms | | nothing, but a request left a 60 s timer | 2–3 ms | 4 ms | 2 ms |

With the 60 s `setTimeout`, v0.4.1 took 5 ms and v0.5.0 took 14,499 ms, so the change arrived in v0.5.0. The ~14.5 s is the 15 s deadline minus the 0.5 s between the firing and the re-arm, and a 3 s timer holds the alarm for the rest of its 3 s. No firing was recorded as a retry (`retryCount: 0`, `isRetry: false`), and none was duplicated.

## Root cause (v0.5.1, `42269c1`)

1. v0.5.0 made a successfully completed cell event keep its pending I/O. `InFlight::keeps_native_ops()` (`crates/celld/js.rs`) gained `|| self.completed_cell_event`, which is set when the event answers (`self.completed_cell_event = self.scope.is_some()`). That is right for a fetch. In v0.4.x the term was absent, so once the reply was sent the drive aborted the leftover ops (`abort_ops`) and ended. 2. `InFlight::finished()` (`self.retired() && !self.has_refed_ops()`) therefore stays false while a referenced timer is pending, and the `while !entry.finished()` loop in `drive_cell_inner` (`crates/celld/runtime.rs`) keeps the alarm's drive running, bounded by `handler_budget()`. 3. `RuntimeManager::fire_alarm` (`crates/celld/runtime.rs`) awaits that drive (`drive.await`) after the handler's reply and before returning. The actor turns its return into `Event::AlarmFinished` only then. 4. Until `AlarmFinished` arrives, the core keeps `AlarmState::Firing` and stages newer alarm observations for the completion path (`crates/logic/lib.rs`: "Runtime observations can arrive before `AlarmFinished` … Stage a real move for the completion path"), so the re-arm cannot fire. 5. The only other way out is `Timer::OperationDeadline` on the `FireAlarm` effect, at `CELLD_OPERATION_DEADLINE_MS`. `expire_operation` then settles the firing as `Err(Failure::Ambiguous)` ("re-arms it exactly as a failed handler would"), so the core books a handler that succeeded as a stuck one.

The wait in step 3 is needed only when the handler did not complete. On success the handler already settled its claim in the completion turn (`Answer::Alarm` → `settle_alarm(true, false)` in `js.rs`) and replied with its final `AlarmSnapshot` and write position. The `Ok` branch of `fire_alarm` uses nothing else; `final_write` is read only on the `Err` path, where a failed or cancelled alarm settles in the drive's final turn.

## Fix

Detach the drive on `Ok`, and keep awaiting it on `Err`. It's in our fork as one commit on top of v0.5.1: https://github.com/futurepaul/celld/commit/b5f57ea (branch [`fix/alarm-completion`](https://github.com/futurepaul/celld/tree/fix/alarm-completion)).

```diff --- a/crates/celld/runtime.rs +++ b/crates/celld/runtime.rs @@ -2467,14 +2467,24 @@ impl RuntimeManager { js::abort_request_for_shutdown(request_id); } let result = receive.await.context("cell isolate dropped alarm result")?; - let final_write = drive.await.expect("cell alarm drive task panicked")?; - // A cancelled or failed alarm settles its claim in the drive's final - // isolate turn. The event reply is itself held behind every wake-entry - // arm, so waiting for the drive makes that final cache authoritative - // before the core sees completion. match result { - Ok((alarm, wrote)) => Ok((alarm, self.alarm_covered(&cell, alarm), wrote)), + // A handler that returned settled its claim in its completion + // turn (`Answer::Alarm`), so the snapshot and the delta it replied + // with are final. A completed cell event keeps its pending I/O + // (`completed_cell_event`), and that work runs on in the drive + // under the handler budget, as it does after a fetch. Waiting for + // it here held the core's `Firing` state, and with it every + // re-arm, until the operation deadline. + Ok((alarm, wrote)) => { + drive.detach(); + Ok((alarm, self.alarm_covered(&cell, alarm), wrote)) + } Err(error) => { + // A cancelled or failed alarm settles its claim in the drive's + // final isolate turn. The event reply is itself held behind + // every wake-entry arm, so waiting for the drive makes that + // final cache authoritative before the core sees completion. + let final_write = drive.await.expect("cell alarm drive task panicked")?; // The drive completed its final isolate turn before this // branch. Its alarm cache is therefore authoritative even // when the handler failed: `Some` is the automatic retry or ```

The leftover work keeps running in the detached drive under `handler_budget()`, like a fetch's after-response work. `AlarmRequestRegistry` is keyed by `(cell, op)`, so the detached drive cannot collide with the next firing's entry.

Two things you may want to do differently:

- The detached drive's result is now dropped: an error from the handler's leftover I/O, or a panic, goes unreported. Awaiting the handle in a detached task and emitting a `tracing::warn!` would keep it visible. - The fix should come with a regression test that re-arms from an `alarm()` holding an `AbortSignal.timeout`, i.e. the repro above. We couldn't add or run one: on the public tree `cargo test -p celld` runs 0 tests, and `--cfg celld_internal_tests` doesn't compile (missing `*_for_world` / `*_for_test` helpers). Your internal suite hasn't been run against this change.

## Verification

- The repro above is red on the v0.5.1 release binary and on an unpatched v0.5.1 source build (`[0,15514,31018]`), and green with the fix (tables above). - Leftover work still completes after the detach: an `alarm()` that left a 1 s timer writing a SQL row ran at 5 ms, and the timer's row landed at 1,010 ms. - Our app, a Durable Object runtime that multiplexes cron, retries and inbox runs onto one alarm per object and wraps its fetches in `AbortSignal.timeout(30_000)`, failed one e2e check on 10 of 12 runs of v0.5.1: its re-armed alarm fired ~15.3 s late. With the fix that check passes 10/10, and the full suite passes 312/312, as it did on v0.4.0.

## Workaround

Don't leave timers pending in `alarm()`. Use an `AbortController` and clear its timer in a `finally` instead of `AbortSignal.timeout()`:

```js async function fetchWithTimeout(url, init, ms) { const controller = new AbortController(); const timer = setTimeout(() => controller.abort(new DOMException("timed out", "TimeoutError")), ms); try { return await fetch(url, { ...init, signal: controller.signal }); } finally { clearTimeout(timer); } } ```

On v0.5.1 this re-armed in 3 ms, against 14,453 ms for `AbortSignal.timeout` on the same fetch. Lowering `CELLD_OPERATION_DEADLINE_MS` also shortens the stall, but it tightens every other operation deadline too.

  1. bartlomiejuclosedlast month on GitHub

  2. BA

    Thanks for the report. This is now fixed and will be released in the next version.

  3. bartlomiejureferenced 81d1c52fb5belast month on GitHub

  4. bartlomiejureferenced 0aa22f091625last month on GitHub

  5. bartlomiejureferenced 46692d514ac3last month on GitHub

  6. ryreferenced def3ad02192dlast month on GitHub

  7. bartlomiejureferenced bab562b22376last month on GitHub

  8. bartlomiejureferenced af69bb4337c5last month on GitHub

  9. bartlomiejureferenced bad4649d01f0last month on GitHub