denoland/celld

Public

Alarm-driven Durable Object writes aren't durability-gated / owner-node loss drops un-flushed alarm history

PH

Hi — I've been learning celld by building a small toy project: (several LLM agents, one cell each, chatting with each other on DO alarms, with a dashboard that watches the bucket (leases, per-cell LTX activity)). It's a really neat system and I've been having a blast with it.

While testing failover scenarios, I hit something that I think might be a bug: the agents' conversation history — written from `alarm()` handlers— didn't survive owner-node failure. When I watched the bucket, I saw that those cells never shipped any `ltx/` segments no matter how long they ran. Writes from ordinary HTTP requests worked and were durable just as expected, but the writes originating from alarms never were, unless some HTTP request happened to touch the cell afterward (which then flushed the whole backlog at once)

I spent a little time in the source code to make sure I wasn't just holding it wrong, and I think the durability gate is only reached from the response path, or in other words alarm invocations never mark a cell dirty for the replicator.

## Environment

- celld 0.1.0, `ghcr.io/denoland/celld` (3 nodes, Docker, host networking) - Bucket: MinIO (single instance, same host) - Host: macOS / OrbStack;

## Reproduction

1. Deploy a worker with a SQLite-backed DO class that runs an alarm loop:

```js export class Agent extends DurableObject { async init() { // called once via an HTTP route await this.ctx.storage.setAlarm(Date.now() + 5_000) } async alarm() { this.ctx.storage.sql.exec( `CREATE TABLE IF NOT EXISTS log (ts INTEGER, note TEXT)`) this.ctx.storage.sql.exec( `INSERT INTO log (ts, note) VALUES (?, 'tick')`, Date.now()) await this.ctx.storage.setAlarm(Date.now() + 5_000) // keep looping } async rows() { return this.ctx.storage.sql.exec(`SELECT count(*) n FROM log`).toArray() } } ```

2. Init the cell once over HTTP, then generate traffic **only via alarms** (no further HTTP requests that write). Let it accumulate rows for a few minutes. 3. Observe the bucket: the cell's prefix contains **only `own.json`** — zero `ltx/` segments ever appear, no matter how long the alarm loop runs. (Contrast: a cell written via ordinary request/response HTTP grows `ltx/` segments immediately.) 4. `docker compose stop <owner-node>`. Let a surviving node reclaim the cell. 5. Read the row count through a survivor: **0**. All alarm-driven rows are gone.

Observed timings for the reclaim itself were fine (~8s from container-dead to all cells served by survivors, default lease TTL) — the failover mechanics work; it's the replication that never happened.

  1. BA

    Hey @philrenaud, thanks for the report. We've already fixed this and a new version will be released later this wek.

  2. bartlomiejuclosed3 months ago on GitHub

  3. philrenaudmentioned3 months ago on GitHub

  4. philrenaudsubscribed3 months ago on GitHub

  5. PH
    Author

    Thanks @bartlomieju!

  6. bartlomiejumentioned3 months ago on GitHub

  7. bartlomiejusubscribed3 months ago on GitHub