denoland/celldPublic

self-hosted, distributed Durable Objects

disk saturates without bound after a cell is evicted by snapshot

LE

lemeb opened ·

Hi! :) We run the default fleet durability, and we have been encountering some disk-related issues.

The main culprit is peerlogs (`state/peerlog/<leader>/<gen>/`), which grows out of control until Kubelet evicts the pod. This happens all the time, on relatively little traffic.

Our guess on why this happens:

* followers only delete peerlog batches once the leader marks them as covered, and such mark is advanced in order * a batch counts as covered when every cell in it is durable * when a cell is evicted by a snapshot, `evict()` never credits the cell's `durable_txid`, then drops the cell * the batch is uncovered forever, and blocks the batches after it * this is only "fixed" by epoch change (e.g. pod restarting)

Our candidate patch (absolutely untested):

```patch --- a/crates/celld/ltx_repl.rs +++ b/crates/celld/ltx_repl.rs @@ fn evict(...) if let Some(snapshot) = snapshot { let published = asyncrt::timeout_at( deadline, self.publish_handoff_snapshot(cell, epoch, &snapshot), ) .await; if matches!(published, Ok(Ok(()))) && self.epoch_replicated(cell, epoch).await { + handle.durable_txid.fetch_max(snapshot.max_txid.0, Ordering::SeqCst); artifact = Some(EvictionRestoreArtifact::Snapshot); ```

  1. lemebrenamed follower peerlog grows without bound after a cell is evicted by snapshot to disk saturates without bound after a cell is evicted by snapshotlast month on GitHub

  2. bartlomiejuclosedlast month on GitHub

  3. bartlomiejureferenced 655ff6caf8b6last month on GitHub

  4. bartlomiejureferenced 90b43017241flast month on GitHub

  5. BA

    Fixed in v0.6.2: follower disk use stays bounded after a snapshot eviction or a deleted facet.

  6. lemebreferenced this in Follower never deletes its peerlog after session is sealed #259last month on GitHub