A frozen (paused) owner keeps its cells unavailable until it resumes, even after its lease expires
ivanjeremic opened ·
**Version:** celld 0.4.1 (`ghcr.io/denoland/celld:v0.4.1`, `sha256:ce8bbc3c26a1…`). The setup was two nodes and MinIO `RELEASE.2025-09-07T16-13-09Z` on one Docker network (Docker Desktop, Apple Silicon, linux/arm64), with default `CELLD_TTL_MS` (10 s) and default fleet durability. The application was `examples/counter`.
### What happened
I froze a cell's owner with `docker pause`: every thread stopped, sockets left open. That's the shape of a paused VM, a `SIGSTOP`, or a long stall. While it stayed frozen, requests for that cell sent to the other node never completed. Over three minutes the client sent 56 requests to the surviving node, and each one got no response before it gave up at 3 s. The frozen owner's lease had long since expired.
When I unpaused the owner, it self-fenced immediately (`SELF-FENCE: node lease not renewed within TTL — halting`, `node_lease_watchdog_fence`, exit 3). The survivor then served the cell at once, and the counter continued at the next value. **Safety held:** no acknowledged write was lost or repeated. The gap is availability while an owner is stalled.
For contrast, when I killed the owner with `docker kill` (SIGKILL), the survivor recovered in 8.2 s. It sealed the dead session's node log and served the cell at epoch 2.
### Expected
`docs/guarantees.md`, under Self-fencing, says: "a request is safe even before the fence runs, because celld compares the current time against the published expiry each time it routes a request." I read that as: once the frozen owner's published expiry passes, the survivor stops routing to it and acquires the cell, the way it does for a crashed owner, within about one TTL.
### Reproduction
Run this in bash from a checkout of this repository, with a Linux `esbuild` binary at `$ESBUILD`:
```sh docker network create lab docker run -d --name minio --network lab -e MINIO_ROOT_USER=lab-access \ -e MINIO_ROOT_PASSWORD=lab-secret-key minio/minio:RELEASE.2025-09-07T16-13-09Z server /data docker run --rm --network lab --entrypoint sh minio/mc \ -c 'mc alias set l http://minio:9000 lab-access lab-secret-key && mc mb l/cells' printf 'AWS_ACCESS_KEY_ID=lab-access\nAWS_SECRET_ACCESS_KEY=lab-secret-key\nAWS_REGION=us-east-1\n' > creds.env B="--bucket s3://cells --endpoint http://minio:9000 --region us-east-1" IMG=ghcr.io/denoland/celld:v0.4.1
docker run --rm --network lab --env-file creds.env -v "$PWD/examples/counter:/app" \ -v "$ESBUILD:/usr/local/bin/esbuild:ro" -w /app $IMG deploy . $B docker run -d --name a --network lab --env-file creds.env -p 127.0.0.1:8080:8080 $IMG $B \ --listen 0.0.0.0:8080 --internal-listen 0.0.0.0:8081 --advertise a:8081 docker run -d --name b --network lab --env-file creds.env -p 127.0.0.1:8082:8080 $IMG $B \ --listen 0.0.0.0:8080 --internal-listen 0.0.0.0:8081 --advertise b:8081
curl 'http://127.0.0.1:8080/?name=x' # node a acquires the cell docker pause a for i in $(seq 60); do curl -s -m 3 'http://127.0.0.1:8082/?name=x' && break; done # no answer docker unpause a # a self-fences (exit 3) curl 'http://127.0.0.1:8082/?name=x' # b answers at once, next value ```
### The survivor's log during the freeze
The survivor evicted the frozen node as a gray follower and fell back to bucket acknowledgements. No activation of the frozen owner's cell appears until after the unpause:
``` WARN celld::node_log: gray follower evicted event="log_evict_verdict" member="node_1b90…" rule="backstop" outstanding_ms=1517 … WARN celld::node_log: log ensemble degraded; acks ride the bucket epoch=5 why="gray follower node_1b90… evicted" ```
Its only other warnings were `http_connection_failures … cause="incomplete_message"`, from the requests the client abandoned.
I haven't traced this in the source. The requests look as if they're forwarded to the frozen owner and wait on it, instead of being re-routed once its expiry has passed.
### Why it matters
A VM live-migration pause, a frozen container, or a long garbage-collection or swap stall can stop an owner without killing it. Its cells then stay unavailable for as long as the stall lasts, while a crash recovers in seconds. The workaround, having the supervisor kill a stalled node, is reasonable, but it's worth stating in `docs/guarantees.md` if this behaviour is intended.
I found this while evaluating celld as a runtime option for [Zelavis](https://github.com/zelavis/zelavis).
- MA
Thanks for the detailed reproduction. The fix has been merged, and I’m closing this issue as fixed. The change is awaiting a public release.
The ingress now rechecks ownership after the cached route expires, so fresh requests can complete while the old owner stays paused. Recovery still depends on the existing lease checks and available storage.
bartlomiejureferenced 9c2d299dd6ablast month on GitHub
ryreferenced e519694581dclast month on GitHub
ryreferenced 4250d366aceblast month on GitHub
ryreferenced 3127cf8e307elast month on GitHub
ryreferenced 12d5b6333fe5last month on GitHub