disk saturates without bound after a cell is evicted by snapshot
lemeb opened ·
Hi! :) We run the default fleet durability, and we have been encountering some disk-related issues.
The main culprit is peerlogs (`state/peerlog/<leader>/<gen>/`), which grows out of control until Kubelet evicts the pod. This happens all the time, on relatively little traffic.
Our guess on why this happens:
* followers only delete peerlog batches once the leader marks them as covered, and such mark is advanced in order * a batch counts as covered when every cell in it is durable * when a cell is evicted by a snapshot, `evict()` never credits the cell's `durable_txid`, then drops the cell * the batch is uncovered forever, and blocks the batches after it * this is only "fixed" by epoch change (e.g. pod restarting)
Our candidate patch (absolutely untested):
```patch --- a/crates/celld/ltx_repl.rs +++ b/crates/celld/ltx_repl.rs @@ fn evict(...) if let Some(snapshot) = snapshot { let published = asyncrt::timeout_at( deadline, self.publish_handoff_snapshot(cell, epoch, &snapshot), ) .await; if matches!(published, Ok(Ok(()))) && self.epoch_replicated(cell, epoch).await { + handle.durable_txid.fetch_max(snapshot.max_txid.0, Ordering::SeqCst); artifact = Some(EvictionRestoreArtifact::Snapshot); ```
lemebrenamed follower peerlog grows without bound after a cell is evicted by snapshot to disk saturates without bound after a cell is evicted by snapshotlast month on GitHub
bartlomiejureferenced 655ff6caf8b6last month on GitHub
bartlomiejureferenced 90b43017241flast month on GitHub
- BA
Fixed in v0.6.2: follower disk use stays bounded after a snapshot eviction or a deleted facet.
lemebreferenced this in Follower never deletes its peerlog after session is sealed #259last month on GitHub