denoland/celldPublic

self-hosted, distributed Durable Objects

A declared bounded loss reuses the seqs of acknowledged writes; a 3-node fleet keeps a one-follower ensemble; no operator path out of 'refusing to seal'

EF

## Summary

Three behaviours on v0.6.0 that make a declared loss hard to detect or recover from, found while reproducing the restart sequences in #244 (same setup: two or three nodes with stable `CELLD_NODE` ids, fleet durability, MinIO, dedicated VMs, synthetic data, a client that matches every acknowledged write by content).

### 1. After a declared loss, new writes reuse the seqs of written-off acknowledged writes

Owner restarted alone while its follower stayed down past the witness grace: loss declared (1,590 and 1,658 acknowledged writes, 2 of 2 runs). The owner then accepted about 450 new writes. The follower was brought back with its stable id and its original disk.

- Every written-off row was still intact on the follower's disk (`peerlog/<owner>/<dead session>/*.batch`) before and after it rejoined. None was restored, and the rejoin logged nothing about them. - The new writes took the written-off seqs: 428 and 463 seqs now hold different rows than the ones that were acknowledged under them.

A client that stored a seq from an acknowledged write can later read a different row under it. Expected: either no seq reuse after a loss, or a loud failure / epoch marker a client can detect, and a way to recover rows a returning follower still holds.

### 2. A three-node fleet keeps a one-follower ensemble

With three nodes and stable ids, the owner's ensemble held **one** follower in all five runs, including after 2.5 min of load with the third node live (the source reads as if it recruits up to two). Only that follower is a witness:

- that follower up first, third node down past the grace, then the owner: 0 lost; - third node up first, ensemble follower down: 1,610 lost; - owner alone: 1,501 lost.

So a third node does not protect the acknowledged tail. Expected: a second follower recruited, or docs stating that the ensemble size is one.

### 3. No operator path out of "refusing to seal"

When a follower restarts under a new id at its old address, the old member's seal request (`POST /peer/log/seal`, `x-cells-peer-target` = the old id) is answered `409 Conflict` with `peer request targets a different node session` (captured on the wire, 3 of 3). `node_log.rs` (around L5063-5080) counts any HTTP error as an answering, so undecided, member, and the pass retries every 30 s with no limit. Observed for 10.5 to 26.5 min per run; nothing clears it except:

- restarting that node with its **old** id on the same disk: recovered, 0 lost (2 of 2); - stopping the new-id process: declared loss within one retry (1,462 lost); - moving it to another address: declared loss (1,155 lost); - deleting the dead owner's record in the bucket: **silent** loss (1,440 lost, no loss record, no log line).

Expected: treat this 409 as "the old member is gone, its disk may still hold the tail" with a clear log line naming the old id to restart with, and a CLI command to resolve an undecided member.

  1. EfficiencyCorpreferenced this in Docs: without CELLD_NODE, or with one node restarted alone, a restart after a fleet-wide fence loses acknowledged writes or wedges recovery #244last month on GitHub

  2. bartlomiejureferenced 655ff6caf8b6last month on GitHub

  3. bartlomiejureferenced 90b43017241flast month on GitHub