denoland/celld

Public

Panic "cell storage reached outside a turn": expired/stuck handlers settle their alarm claim from the driving task

FU

The following is AI-written. GLM-5.3 caught this crash loop and diagnosed this as the cause.

## Summary

`celld` panics and dies whenever a turn exceeds its handler budget, because the budget-expiry path settles the turn's alarm claim on the driving task — outside any turn — and the settle touches turn-guarded storage. On a fleet where any cell's alarm handler regularly runs long (e.g. a cron workflow doing outbound HTTP that can hang), the node crash-loops at the budget cadence.

## Environment

- celld 0.3.0 (89e4ffc), release build, Linux x86_64 - two nodes on one host (witness ensemble), ~30 cells - cells with `cron` workflows that make outbound fetches; a handler that hangs (no fetch timeout) outlives the budget

## Crash

``` thread 'celld-core' panicked at crates/celld/storage.rs:142:5: cell storage reached outside a turn ```

Every ~6 minutes under normal traffic (300s default budget + alarm cadence), recovered only by systemd restart. Relevant frames:

``` celld::storage::finish_alarm_handler_with_retry_policy <celld::js::InFlight>::fail js.rs:1905 <celld::js::InFlight>::time_out js.rs:1959 celld::runtime::drive_cell::{closure#0} runtime.rs:2172 tokio current-thread scheduler (celld-core) ```

## Root cause

In `drive_cell` (runtime.rs), every other `entry` mutation runs inside `slot.turn(|worker| ...)` — on the isolate, with the cells installed. The `Wake::Expired` and `Wake::Idle` arms are the exception:

```rust Wake::Expired => { entry.time_out(budget); // <- bare, on the serial core break; } Wake::Idle => { entry.stuck(); // <- ditto break; } ```

`time_out`/`stuck` → `InFlight::fail` → `settle_alarm` → `storage::finish_alarm_handler_with_retry_policy`, which reaches the turn-guarded accessors. On the driving task the thread-local is null, so `cells()` asserts and the process dies.

The invariant is already documented ten lines below the loop — "recording it is storage only the isolate can reach" — and the post-loop `owes_alarm()` path follows it correctly via `turn_finish_alarm`. The two budget arms are the one asymmetric violation. The stateless drive loops have the same shape; their entries carry no alarm claim so `settle_alarm` early-returns there, but they deserve the same discipline.

## Fix

Lock the isolate (installing cells) before failing, the same way `turn_finish_alarm` does:

```diff --- a/crates/celld/js.rs +++ b/crates/celld/js.rs pub fn turn_finish_alarm(&mut self, entry: &mut InFlight) { let Some(inner) = self.inner.as_mut() else { return; }; let (_locker, _cells) = inner.lock(); entry.settle_alarm(false, false); } + + /// Fail a request that exceeded its handler budget, inside the isolate: + /// `InFlight::fail` settles a claimed alarm, and that bookkeeping is + /// turn-guarded storage — calling `time_out` from the driving task hits + /// the null thread-local and panics the process. + pub fn turn_time_out(&mut self, entry: &mut InFlight, budget: Duration) { + let Some(inner) = self.inner.as_mut() else { + return; + }; + let (_locker, _cells) = inner.lock(); + entry.time_out(budget); + } + + /// Same discipline for a handler that is waiting on nothing. + pub fn turn_stuck(&mut self, entry: &mut InFlight) { + let Some(inner) = self.inner.as_mut() else { + return; + }; + let (_locker, _cells) = inner.lock(); + entry.stuck(); + } } ```

```diff --- a/crates/celld/runtime.rs +++ b/crates/celld/runtime.rs Wake::Expired => { - entry.time_out(budget); + slot.turn(|worker| worker.turn_time_out(&mut entry, budget)).await; break; } Wake::Idle => { - entry.stuck(); + slot.turn(|worker| worker.turn_stuck(&mut entry)).await; break; } ```

(applied to all three drive loops; the `break` arms type as `!` so the match is unaffected).

Deployed on our fleet since yesterday: the crash-loop stopped immediately, no panics in 26+ hours of the same traffic that previously killed a node every ~6 minutes.

## Side note: production backtraces are empty

The release profile sets `strip = true`, so even with `RUST_BACKTRACE=1` the panic prints "stack backtrace:" and nothing else — no symbols ship in the binary, and there's no split-debuginfo file to install alongside. We had to rebuild with `CARGO_PROFILE_RELEASE_STRIP=false` on the host to get the frames above. Consider splitting debuginfo at build time (or defaulting `strip = "debuginfo"` instead of full strip) so production panics are symbolizable.

  1. RY

    thanks for the report. we'll release the fix in 0.3.1 soon

  2. ryclosed2 months ago on GitHub

  3. futurepaulreferenced this in Store blip of 1-2s permanently bricks cells: reactivation seeds the replica past the bucket's coverage and the fence halt abandons pending uploads #1732 months ago on GitHub

  4. bartlomiejureferenced 554fa82872e62 months ago on GitHub

  5. bartlomiejureferenced a52f9905425b2 months ago on GitHub