denoland/celld

Public

Cell event can re-enter a freed isolate after its residency is removed

BP

## Affected version

`celld` v0.2.1 (`ae8fac053d79f971bfcb996054bb43eb2f9b05da`).

## Impact

A cell event that is suspended awaiting host I/O can retain an `Arc<Slot>` after the cell's residency has been removed. Pool cleanup can then treat the isolate as unused and drop its `Worker` and V8 heap. When the I/O completes and the event re-enters the isolate, `Slot::turn()` panics with:

```text thread 'tokio-rt-worker' panicked at crates/celld/pool.rs:86:21: entered isolate 0 after it was freed ```

The release profile uses `panic = "abort"`, so this terminates the whole celld process. In a fleet, active streaming HTTP responses on that replica abort and callers may observe 500 responses while ownership moves.

## Minimal reproduction

The race is this deterministic lifecycle interleaving:

1. Start a cell event and let it suspend on asynchronous I/O after its first JS turn. 2. Remove/stop that cell runtime while the event is suspended. `stop_cell()` drops the `CellHandle` and therefore its `Residency`. 3. Run `reap_empty()`. With no cells and no active turn, the pool retires the slot and `free_retired()` changes `worker` to `None`. 4. Complete the event's I/O so `drive_cell()` attempts its next `slot.turn(...)`. 5. `Slot::turn()` observes `worker == None` and panics with `entered isolate 0 after it was freed`.

### Runnable regression

A public test-only patch that applies directly to v0.2.1 is available here:

https://gist.github.com/bplotnick/df3fd3a4ffee01269c9d04b2c56ba1d9

Run it in a clean checkout:

```sh git checkout --detach ae8fac053d79f971bfcb996054bb43eb2f9b05da curl -L https://gist.githubusercontent.com/bplotnick/df3fd3a4ffee01269c9d04b2c56ba1d9/raw/75d8dbcbf1c41cd5fd0c2d956e48a35bc9faebc8/celld-v0.2.1-suspended-event-repro.patch | git apply cargo test -p celld suspended_cell_event_survives_stop_and_pool_reap -- --nocapture ```

The command exits 101 after `Slot::turn()` panics with `entered isolate 0 after it was freed`. The proposed patch includes the same regression test; it passes with the production fix.

In current code, the relevant ownership transfer is:

```rust // RuntimeManager::cell_isolate let slot = handle.residency.slot().clone();

// drive_cell holds only Arc<Slot> while it awaits I/O. tokio::spawn(drive_cell(slot, job, ...));

// While the event is suspended, stop_cell() removes its CellHandle. // This is the isolate's only resident cell, so dropping the handle releases // the final Residency and reduces the slot's resident-cell count to zero. drop(handle); pool.reap_empty();

// The suspended event resumes against the freed worker. slot.turn(|worker| worker.turn_deliver(...)).await; ```

The regression test exercises that ordering through the production dispatch path. Its Durable Object awaits a response from a controlled local HTTP server. After the server confirms that the event is suspended awaiting host I/O, the test stops the cell, calls `reap_empty()`, and only then releases the HTTP response. On v0.2.1 the resumed event panics in `Slot::turn()`; with the fix it completes normally.

The production occurrence ended with exit code 139 at `2026-08-14T19:31:18.221Z`. Immediately before the panic, the node had published a `ControlPlane` cell and a new `AgentSession` cell in isolate 0. A peer logged `celld connection failed: error from user's Body stream`; the client then received three consecutive `500: Unknown` responses while reconnecting its session event stream.

## Cause

`Residency` protects an isolate only while the cell is registered. Cell dispatch paths clone the slot but do not take an `Affiliation`. `turns` protects only a queued/running JS turn and falls to zero while the event awaits I/O. Therefore neither `cells`, `requests`, nor `turns` prevents `free_retired()` from freeing the worker during the suspended interval.

This affects both `drive_cell()` and the resident `drive_worker_on_cell()` path.

## Proposed fix

Create an `Affiliation` under the cell-registry lock when dispatching every cell event or resident Worker request, pass that affiliation into the driver, and retain it for the driver's entire async lifetime. The existing `requests` counter then prevents `free_retired()` and slot reuse until the last suspended event exits, even if the cell's `Residency` is dropped concurrently.

## Tradeoffs

The fix intentionally allows an isolate's memory to outlive the cell's registration while an already-dispatched event drains. A slow or hung host operation can therefore delay retirement and reuse of that isolate. The `Affiliation` does not keep the cell published, preserve ownership, or admit new events; it only keeps the isolate and its V8 heap available to work that can still re-enter them. It is released automatically when the driver exits, including normal error paths. An operation that never completes can retain it for as long as the driver remains pending, so a hard upper bound would require a separate operation-timeout or drain policy.

Keeping the event's `Residency` instead would continue counting the cell as resident after `stop_cell()` removed it, conflating cell lifecycle with in-flight execution. Waiting in `stop_cell()` for every event would avoid temporary retention but could block eviction or ownership transfer on slow I/O. `Affiliation` keeps those concerns separate: `stop_cell()` does not wait for suspended events, while only the isolate resource needed by existing work is pinned until that work exits.

The focused regression fails on v0.2.1 with:

```text entered isolate 0 after it was freed test runtime::tests::suspended_cell_event_survives_stop_and_pool_reap ... FAILED ```

It passes with this change:

```text test runtime::tests::suspended_cell_event_survives_stop_and_pool_reap ... ok ```

If this approach is acceptable, I can send a patch

  1. ryclosed3 months ago on GitHub

  2. RY

    Fixed — and thank you for an exemplary report: the root cause, the deterministic reproduction, and the proposed fix were all correct, and the fix landed essentially as you proposed. Every cell-event and resident-Worker dispatch now takes its `Affiliation` under the cell-registry lock and holds it for the driver's entire async lifetime, so a suspended event keeps its claim on the isolate across `stop_cell` and pool reaping; retirement still proceeds, and only the free waits for the last suspended event to drain, per the tradeoff you outlined.

    The interleaving you filed is also now pinned permanently in our verification suites — including a model-checked spec of the slot lifecycle whose broken configuration reproduces exactly this counterexample — so this class of bug cannot silently return.

    The fix will ship in the next release (0.3.0, expected next week).

  3. bartlomiejureferenced 9c0ca09ef5d53 months ago on GitHub

  4. bartlomiejureferenced bc0e29cddf303 months ago on GitHub

  5. bartlomiejureferenced 89e4ffc53a143 months ago on GitHub