- Issues
- #159
Open streaming responses do not keep cells active, and eviction does not terminate their response bodies
bplotnick opened ·
## Description
celld considers a Durable Object request finished once the Worker returns its `HttpResponse`. The request's activity guard does not remain attached to a streaming response body, and subsequently forwarded chunks do not otherwise update the cell's `last_used_mono_ms`.
As a result, an active SSE response that emits periodic heartbeat comments can still become eligible for `CELLD_IDLE_EVICT_S` eviction. When the cell is evicted, celld stops its residency/runtime but does not deterministically complete or fail response bodies that were already handed to the HTTP server. The client can therefore be left with an open body whose producer has stopped, rather than receiving a clean EOF.
For a gzip-compressed response, a later proxy or transport timeout can surface this as a truncated gzip body, for example:
```text java.io.EOFException: source exhausted prematurely at okio.InflaterSource.read(InflaterSource.kt:45) at okio.GzipSource.read(GzipSource.kt:67) ```
## Reproduction
Regression/characterization tests based on upstream `v0.3.0` (`89e4ffc53a14ecb496d2ca5014ff9d19b0061ad9`) are available on this branch:
https://github.com/bplotnick/celld/tree/repro/sse-idle-eviction
The reproduction commit is [`d6b2b9a`](https://github.com/bplotnick/celld/commit/d6b2b9a50876ad5d6e38062e0f915ddcbf3d5ebd).
The patch contains two focused tests:
1. `response_body_chunks_do_not_refresh_the_idle_lease` models a resident cell with a one-second idle lease and SSE heartbeats at 250, 500, and 750 ms. Because there is no response-body activity event to feed into the decision core, the 1,000 ms resource sample selects the cell for eviction. This characterization test passes on the affected revision. 2. `evicting_a_streaming_cell_orphans_the_response_body` runs a real V8 Durable Object returning a `ReadableStream`, observes its SSE heartbeat, evicts the cell, and requires the response body to complete. It fails on the affected revision with `eviction left the response body open`.
Run them with:
```sh cargo test -p celld-logic response_body_chunks_do_not_refresh_the_idle_lease -- --nocapture cargo test -p celld evicting_a_streaming_cell_orphans_the_response_body -- --nocapture ```
Observed results:
```text test response_stream_repro_tests::response_body_chunks_do_not_refresh_the_idle_lease ... ok
thread 'runtime::tests::evicting_a_streaming_cell_orphans_the_response_body' panicked: eviction left the response body open ```
We also reproduced this through a deployed HTTP endpoint with `CELLD_IDLE_EVICT_S=300`:
- The SSE client received `: ping` every 15 seconds through T+315s. - Heartbeats then stopped. - No final SSE frame, clean EOF, or response error arrived. - The HTTP request remained pending until the client test timed out at T+390s. - Durable application state remained intact and reconnecting immediately produced `: connected` followed by new heartbeats.
The timing is consistent with the 300-second idle lease plus the next eviction or isolate-maintenance pass.
## Expected behavior
To match Cloudflare Durable Object semantics, celld should hold request activity for the lifetime of the response body. An open streaming response should therefore prevent ordinary idle eviction and hibernation until the body closes, errors, or is canceled by the client. This should not require application code to call `waitUntil()` or emit periodic chunks merely to retain the runtime.
Forced lifecycle operations still need a bounded escape hatch. If celld must stop the cell for shutdown, handoff, fencing, or another non-idle reason, every response stream owned by that cell should be deterministically terminated and its writer bookkeeping released. For an HTTP response whose headers have already been sent, this should result in a prompt, well-defined client-visible completion/cancellation rather than an orphaned body that waits for a downstream timeout.
Cloudflare documents that Durable Objects remain active while handling response streams and that `waitUntil()` is unnecessary for this behavior:
https://developers.cloudflare.com/workers/runtime-apis/context/#waituntil
Cloudflare's lifecycle documentation also excludes a request/event that is still being processed from hibernation eligibility and describes bounded handling during runtime shutdown:
https://developers.cloudflare.com/durable-objects/concepts/durable-object-lifecycle/
## Suggested direction
- Associate every forwarded response stream with its owning cell and request. - Transfer or extend the request's activity guard into the response body so it is released only when the body closes, errors, or is canceled. - Exclude stream-active cells from ordinary idle eviction, just as celld already excludes an actively executing request. - During forced runtime stop, close or cancel all response streams registered to the cell and remove their response-stream writer entries before discarding the runtime. - Bound forced shutdown with an explicit grace period so a broken or malicious infinite stream cannot prevent node termination indefinitely.
This should apply to all streaming HTTP response bodies, not only SSE.
- BA
This is now done as of v0.4.0. Thanks for the report.