- Issues
- #143
Cell becomes permanently unactivatable once its LTX chain outgrows the activation deadline (v0.1.0)
junosuarez opened ·
When a cell's replica chain grows large enough, it can't be restored if owner node dies. If the restore takes longer than the request timeout, the cell is not restored, the new epoch lock fails, and the cell never becomes live again.
Measured on v0.1.0, 3 nodes, RustFS single-node over NFS (~2 ms per object):
| writes | LTX objects | chain | activation | |---|---|---|---| | 3 | 1 | 1 MB | 0.17 s | | 7,395 | 2,334 | 146 MB | 4.35 s | | 8,191 | 2,585 | 162 MB | 8.16 s | | 37,233 | 10,675 | 668 MB | **never completes** |
**Reproduction.** Deploy `examples/counter` (from the celld repo) and call `/` for ~40k requests (each hits the same `room-42` cell). Then `kill -9` celld. The new owner then logs this every ~20 s, indefinitely:
``` INFO celld::ltx_repl: restored remote replica cell="Counter:60125d5d…" from=1 to=2 ```
`from`/`to` never advance because activation never completes. Requests return `route failed: RestoreFailed` (503, then 500) and eventually just hang. Still dead after 5+ minutes with all three nodes `Running` and idle.
Two smaller things in the same path: `restore` accumulates every file into a `Vec<Vec<u8>>` before merging, so the entire chain is resident in memory during restore (668 MB here, in a 1 GiB container — a larger cell is OOM-killed rather than timing out); and a cell that cannot possibly restore inside the deadline retries forever instead of reporting why.
Compaction is presumably the intended fix, and the code already anticipates it (`replica.rs:18`, "L0-only … so adding compaction later 'just works'"). Until then, either carrying restore progress across attempts or running activation off the request path would break the loop on its own.
Since the threshold is `chain_length × per-object latency > deadline`, a faster object store buys proportionally more headroom, a WAN-latency one much less.
- JU
Docker compose repro env: https://github.com/junosuarez/celld-restore-cliff
- RY
This should be solved in 0.2.0, which will be released today. It has LTX compaction. Let me know if it works for you.
ryclosed3 months ago on GitHub
junosuarezreferenced 87fd1ec4c3363 months ago on GitHub
- JU
Confirmed fixed in v0.2.0 from my reproduction and I see compaction working. The key change that lets me call this fixed isn't merely compaction, but that a restore that takes longer than a request timeout is no longer discarded.