Additive LTX compaction never succeeds once a level-0 boundary snapshot overlaps level 1 ("the compaction source does not continue the destination level")
coccyx opened ·
celld v0.4.1, two-node fleet on EC2 (t4g.large, arm64, AL2023), S3 in us-west-2 through the instance role, `CELLD_DURABILITY=fleet`, compaction left at defaults.
Every threshold compaction on every cell fails with the same error, and has since at least 2026-09-11:
``` WARN celld::ltx_repl: additive LTX compaction failed event="ltx_compaction" trigger="threshold" cell=CoordinatorDO:020b7be0… epoch=23 source_level=0 destination_level=1 queue_ms=0 elapsed_ms=115 result="error" error=the compaction source does not continue the destination level ```
Over six hours on one node: 3,234 failures across four cells (the coordinator, the registry, two session cells), nine successes in the last 48 hours. Chains grow without bound as a result. One session cell is at 666 objects / 3.58 GB, with 542 objects in its current epoch and single segments of 111 MB. Restores replay all of it, which shows up as 30-second activations, and once the chain passed 256 MiB the restore switched to paged reads and hit a separate credential problem I've filed separately.
The chain shape is the same on every affected cell. Level 1 covers txids 1..N in normal-sized objects, and level 0 holds one wide object that starts at 1 (or at the epoch's cut) and ends past N:
``` CoordinatorDO:020b7b… e23 level 0: 844 objects, first 0000000000000001-0000000000000c7a.ltx, last 00000000000027fb-00000000000027fb.ltx level 1: 12 objects, first 0000000000000001-0000000000000100.ltx, last 0000000000000b01-0000000000000c00.ltx
InstallationRegistryDO:735be0… e30 level 0: 370 objects, first 0000000000000001-0000000000000776.ltx level 1: 7 objects, last 0000000000000601-0000000000000700.ltx
InvestigationDO:ee68a0… e17 level 0: 537 objects, includes 0000000000000005-0000000000000358.ltx (111 MB) level 1: 5 objects, last 0000000000000303-0000000000000355.ltx ```
Reading `crates/ltx/src/replica_compactor.rs`: `seek` becomes `previous_max + 1`, the bounded level-0 listing returns the wide object because its `max_txid >= seek`, and the `min_txid != seek` check rejects it. Nothing else covers those txids, so the same call fails on every attempt from then on. I think the wide object is the boundary snapshot from a truncate checkpoint; the compactor either needs to accept a source whose range straddles `seek`, or the checkpoint should not write a level-0 object that spans an already-compacted range.
Happy to pull more listings or logs from this fleet if that helps.
- RY
fixed now - will be released in 0.5.0 soon
ryclosedlast month on GitHub
bartlomiejureferenced 9c2d299dd6ablast month on GitHub
ryreferenced e519694581dclast month on GitHub
ryreferenced 4250d366aceblast month on GitHub
ryreferenced 3127cf8e307elast month on GitHub
ryreferenced 12d5b6333fe5last month on GitHub