denoland/celldPublic

self-hosted, distributed Durable Objects

Renewal retry after a failed CAS waits a full `TTL/3` escalating on store blips

SA

I've been benchmarking celld v0.4.0 directly and ended up with a couple of lease findings I couldn't fully explain, so I had an agent (as I'm not fluent in rust, yet) validate them against the published image (`0.4.0`, `a52f990`) before opening anything. Everything below is offline, no patched build.

A node gets exactly two renewal attempts per lease: the cadence is `TTL/3` on success ([lib.rs#L1931-L1935](https://github.com/denoland/celld/blob/a52f9905425bc41134d817694bdc2c50bcc5e856/crates/logic/lib.rs#L1931-L1935)) and the post-failure retry is also `TTL/3`, measured from the read-back rather than from the authority left ([lib.rs#L1970-L1974](https://github.com/denoland/celld/blob/a52f9905425bc41134d817694bdc2c50bcc5e856/crates/logic/lib.rs#L1970-L1974)). So the read-back branch that finds the prior record unchanged ([lib.rs#L2037-L2046](https://github.com/denoland/celld/blob/a52f9905425bc41134d817694bdc2c50bcc5e856/crates/logic/lib.rs#L2037-L2046)) parks the node until `read-back + TTL/3` even when that lands past the published expiry.

Losing two renewals is then fatal regardless of how much authority is left, or of the store coming back in the meantime.

### Reproduction

No store needed for the scheduling part. `put_cas` answers `Ok(None)` only for a clean 412/409 ([bucket.rs#L20-L24](https://github.com/denoland/celld/blob/a52f9905425bc41134d817694bdc2c50bcc5e856/crates/celld/bucket.rs#L20-L24) comment, also it seems that on bucket lane there is no `Failure::Definite` at all, causing anything else to become ambiguous) and the CAS lane is `max_retries: 0` ([bucket.rs#L576-L580](https://github.com/denoland/celld/blob/a52f9905425bc41134d817694bdc2c50bcc5e856/crates/celld/bucket.rs#L576-L580)), so a 5xx, a reset, or a write outliving the actor timeout all reach the same branch.

Against `celld-logic` directly: feed `Failure::Ambiguous` on the renewal CAS, then a read-back returning the prior record, and the emitted `Timer::NodeLeaseRenew` lands at `now + TTL/3` with no reference to `expires_ms`. I have this as a small crate taking `crates/logic` as a path dependency, happy to share it.

End to end against the published image, 9s TTL, conditional `PUT`s on the node prefix failing 5xx while reads pass through. The fault is lifted the instant the second renewal fails, ~2.8s before expiry, with an independent writer PUT-ing on the same route 4×/s as a witness:

``` 12:43:34.425 acquire applied; published expiry 12:43:43.414 (TTL 9000ms) 12:43:36.527 fault enabled (PUT -> 500) 12:43:37.418 renewal PUT If-Match "38c4…" -> 500 (ambiguous) 12:43:37.420 read-back GET 200, ETag "38c4…" (unchanged, still ours) 12:43:40.423 renewal PUT If-Match "38c4…" -> 500; prior_authority_headroom_ms=2992 12:43:40.424 read-back GET 200, ETag "38c4…" (unchanged, still ours) 12:43:40.615 fault lifted 12:43:40.640 independent PUT on the same route: 200, and every 281ms after no further lease I/O from celld 12:43:43.417 SELF-FENCE: node lease not renewed within TTL — halting (exit 3) ```

All tested runs fenced with the store writable for the last ~2.8s and zero renewal attempts in that window. The control that lifts the fault one attempt earlier survived 2/2, renewing at the next tick with ~2993ms headroom.

So the whole margin is one lost renewal. And the fence looks avoidable here: the read-back had just proven the record was still ours and unchanged, so another `If-Match` was available and safe.

  1. sandros94referenced this in Every self-fence reports `NodeLeaseExpired` so a replaced/removed lease reads as a TTL expiry #1832 months ago on GitHub

  2. bartlomiejuassigned bartlomieju2 months ago on GitHub

  3. BA

    Thanks for the report. This is now fixed and will be released later this week.

  4. bartlomiejuclosed2 months ago on GitHub

  5. bartlomiejureferenced fd7b96b993f52 months ago on GitHub

  6. bartlomiejureferenced b61356b29deb2 months ago on GitHub

  7. bartlomiejureferenced 7ac68e0bd2a72 months ago on GitHub

  8. bartlomiejureferenced 10cb1303dac72 months ago on GitHub