- Issues
- #170
Panic "cell storage reached outside a turn": expired/stuck handlers settle their alarm claim from the driving task
futurepaul opened ·
The following is AI-written. GLM-5.3 caught this crash loop and diagnosed this as the cause.
## Summary
`celld` panics and dies whenever a turn exceeds its handler budget, because the budget-expiry path settles the turn's alarm claim on the driving task — outside any turn — and the settle touches turn-guarded storage. On a fleet where any cell's alarm handler regularly runs long (e.g. a cron workflow doing outbound HTTP that can hang), the node crash-loops at the budget cadence.
## Environment
- celld 0.3.0 (89e4ffc), release build, Linux x86_64 - two nodes on one host (witness ensemble), ~30 cells - cells with `cron` workflows that make outbound fetches; a handler that hangs (no fetch timeout) outlives the budget
## Crash
``` thread 'celld-core' panicked at crates/celld/storage.rs:142:5: cell storage reached outside a turn ```
Every ~6 minutes under normal traffic (300s default budget + alarm cadence), recovered only by systemd restart. Relevant frames:
``` celld::storage::finish_alarm_handler_with_retry_policy <celld::js::InFlight>::fail js.rs:1905 <celld::js::InFlight>::time_out js.rs:1959 celld::runtime::drive_cell::{closure#0} runtime.rs:2172 tokio current-thread scheduler (celld-core) ```
## Root cause
In `drive_cell` (runtime.rs), every other `entry` mutation runs inside `slot.turn(|worker| ...)` — on the isolate, with the cells installed. The `Wake::Expired` and `Wake::Idle` arms are the exception:
```rust Wake::Expired => { entry.time_out(budget); // <- bare, on the serial core break; } Wake::Idle => { entry.stuck(); // <- ditto break; } ```
`time_out`/`stuck` → `InFlight::fail` → `settle_alarm` → `storage::finish_alarm_handler_with_retry_policy`, which reaches the turn-guarded accessors. On the driving task the thread-local is null, so `cells()` asserts and the process dies.
The invariant is already documented ten lines below the loop — "recording it is storage only the isolate can reach" — and the post-loop `owes_alarm()` path follows it correctly via `turn_finish_alarm`. The two budget arms are the one asymmetric violation. The stateless drive loops have the same shape; their entries carry no alarm claim so `settle_alarm` early-returns there, but they deserve the same discipline.
## Fix
Lock the isolate (installing cells) before failing, the same way `turn_finish_alarm` does:
```diff --- a/crates/celld/js.rs +++ b/crates/celld/js.rs pub fn turn_finish_alarm(&mut self, entry: &mut InFlight) { let Some(inner) = self.inner.as_mut() else { return; }; let (_locker, _cells) = inner.lock(); entry.settle_alarm(false, false); } + + /// Fail a request that exceeded its handler budget, inside the isolate: + /// `InFlight::fail` settles a claimed alarm, and that bookkeeping is + /// turn-guarded storage — calling `time_out` from the driving task hits + /// the null thread-local and panics the process. + pub fn turn_time_out(&mut self, entry: &mut InFlight, budget: Duration) { + let Some(inner) = self.inner.as_mut() else { + return; + }; + let (_locker, _cells) = inner.lock(); + entry.time_out(budget); + } + + /// Same discipline for a handler that is waiting on nothing. + pub fn turn_stuck(&mut self, entry: &mut InFlight) { + let Some(inner) = self.inner.as_mut() else { + return; + }; + let (_locker, _cells) = inner.lock(); + entry.stuck(); + } } ```
```diff --- a/crates/celld/runtime.rs +++ b/crates/celld/runtime.rs Wake::Expired => { - entry.time_out(budget); + slot.turn(|worker| worker.turn_time_out(&mut entry, budget)).await; break; } Wake::Idle => { - entry.stuck(); + slot.turn(|worker| worker.turn_stuck(&mut entry)).await; break; } ```
(applied to all three drive loops; the `break` arms type as `!` so the match is unaffected).
Deployed on our fleet since yesterday: the crash-loop stopped immediately, no panics in 26+ hours of the same traffic that previously killed a node every ~6 minutes.
## Side note: production backtraces are empty
The release profile sets `strip = true`, so even with `RUST_BACKTRACE=1` the panic prints "stack backtrace:" and nothing else — no symbols ship in the binary, and there's no split-debuginfo file to install alongside. We had to rebuild with `CARGO_PROFILE_RELEASE_STRIP=false` on the host to get the frames above. Consider splitting debuginfo at build time (or defaulting `strip = "debuginfo"` instead of full strip) so production panics are symbolizable.
- RY
thanks for the report. we'll release the fix in 0.3.1 soon
ryclosed2 months ago on GitHub
futurepaulreferenced this in Store blip of 1-2s permanently bricks cells: reactivation seeds the replica past the bucket's coverage and the fence halt abandons pending uploads #1732 months ago on GitHub
bartlomiejureferenced 554fa82872e62 months ago on GitHub
bartlomiejureferenced a52f9905425b2 months ago on GitHub