The following is AI-written by GLM-5.3

## Summary

On celld 0.3.0, a store outage of just 1–2 seconds under write load can leave cells **permanently unrestorable** (`route failed: RestoreFailed` on every request, across restarts, forever). Two defects compound:

1. `Effect::Halt` (the lease self-fence) exits the process instantly, dropping replication handles — and with them the in-process retries that would re-upload staged L0 rows once the store returns. 2. When the process comes back and reactivates the local image, `LtxRepl::activate` seeds the replica's position from `db.pos()` — the **local** position, which is ahead of what the bucket actually received. The skipped rows are never uploaded; the next writes upload *past* them, and the bucket's L0 chain goes non-contiguous. Every later restore refuses the cell:

``` restore <cell> e1: non-contiguous ltx files: have up to 0000000000000095 but next file starts at 00000000000000b3 ```

(Sibling report: #170, from the same fleet.)

## Environment

- celld 0.3.0 (89e4ffc), Linux x86_64, single node and two-node witness ensemble both affected - any cell with write traffic during the outage

## Mechanism

- `sync_cell` (ltx_repl.rs ~1213–1233) stages WAL rows locally (`db.sync()`), then uploads `from..=dpos` sequentially; a failed PUT aborts the pass. `from = replica.pos().txid + 1`, and `pos` only advances via `seed_pos` after a successful pass — so while the handle lives, the gap is retried. Fine. - The lease watchdog fences the node during the outage (`node_lease_watchdog_fence`). `Effect::Halt` (actor.rs:2495) sends the exit immediately. `release()` (ltx_repl.rs:945) has already dropped the handle by then; the staged rows now exist only in the local DB file. - The evict path already guards a sibling of this (ltx_repl.rs:963 comment, "fidelity audit, DRIFTED #3": eviction refuses when its final durability pass fails, precisely because a preserved snapshot could "re-seed `durable_txid` past the truth") — but `release()` (the fence/crash footprint) has no such guard, and the seed itself is still unclamped. - On restart, boot's local-cell inventory reactivates the image (`activate` with `fresh: true` over an existing `dst`). The comment at ltx_repl.rs:765 states the assumption — "The just-opened db's position is the replica's seed … **equal to the remote under epoch fencing**" — which a fenced process violates: `db.pos()` (ltx_repl.rs:788) is ahead of the bucket, `seed_pos` (ltx_repl.rs:795) skips the gap, and the next pass starts at `db.pos + 1`. Post-restart writes then upload past the hole; restores brick.

## Deterministic repro (red/green verified)

A regression test that runs the whole sequence without leases, fences, or real I/O timing — a `FlakyStore` wrapper fails PUTs on demand:

1. activate a cell (epoch 1); write two commits; `sync_wait` → Durable; bucket coverage ≥ txid 1. 2. arm the store failure; write two more commits; `sync_wait` → Failed (rows staged locally, bucket refuses them). 3. `release(cell, 1)` — exactly what a fenced or crashed process leaves. 4. disarm the failure; `activate` again (the local image is reused); write one more commit — post-restart traffic; `sync_wait` → Durable. 5. assert the bucket's L0 listing is contiguous from txid 1.

On 0.3.0 this fails with the production symptom:

``` assertion `left == right` failed: L0 chain must be contiguous at txid 2 left: 3 right: 2 ```

With the fix below it passes in 0.04s. Full test module (drops into the internal test build — it uses only in-repo APIs, gated `#[cfg(all(test, celld_internal_tests))]`):

```rust #[cfg(all(test, celld_internal_tests))] mod reactivation_gap_tests { use super::*; use object_store::memory::InMemory; use object_store::path::Path as StorePath; use object_store::{ObjectMeta, ObjectStore, PutMultipartOpts, PutOptions, PutPayload}; use std::sync::atomic::{AtomicBool, Ordering};

/// A store whose writes fail on demand: the deterministic stand-in for /// the sub-second store blip that interrupts an upload mid-pass (and, in /// production, fences the node out from under it). struct FlakyStore { inner: Arc<dyn ObjectStore>, fail_puts: AtomicBool, }

impl std::fmt::Display for FlakyStore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!(f, "FlakyStore({})", self.inner) } }

impl std::fmt::Debug for FlakyStore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("FlakyStore").finish() } }

#[async_trait::async_trait] impl ObjectStore for FlakyStore { async fn put_opts( &self, location: &StorePath, payload: PutPayload, opts: PutOptions, ) -> object_store::Result<object_store::PutResult> { if self.fail_puts.load(Ordering::SeqCst) { return Err(object_store::Error::Generic { store: "flaky", source: "injected store blip".into(), }); } self.inner.put_opts(location, payload, opts).await } async fn get_opts( &self, location: &StorePath, options: object_store::GetOptions, ) -> object_store::Result<object_store::GetResult> { self.inner.get_opts(location, options).await } async fn put_multipart_opts( &self, location: &StorePath, options: PutMultipartOpts, ) -> object_store::Result<Box<dyn object_store::MultipartUpload>> { self.inner.put_multipart_opts(location, options).await } fn list( &self, prefix: Option<&StorePath>, ) -> futures_util::stream::BoxStream<'_, object_store::Result<ObjectMeta>> { self.inner.list(prefix) } async fn list_with_delimiter( &self, prefix: Option<&StorePath>, ) -> object_store::Result<object_store::ListResult> { self.inner.list_with_delimiter(prefix).await } async fn copy(&self, from: &StorePath, to: &StorePath) -> object_store::Result<()> { self.inner.copy(from, to).await } async fn copy_if_not_exists( &self, from: &StorePath, to: &StorePath, ) -> object_store::Result<()> { self.inner.copy_if_not_exists(from, to).await } }

fn opts<'a>(cell: &'a str) -> ActivationOptions<'a> { ActivationOptions { cell, epoch: 1, fresh: true, took_over: false, resume_local: false, prior: None, } }

/// Reactivating a local image that staged rows the bucket never received /// must not seed the replica past the bucket's coverage. A fenced or /// crashed process leaves exactly that state (writes staged, uploads /// interrupted), and a seed past them skips their upload forever: the /// bucket's L0 chain goes non-contiguous and every later restore of the /// cell fails (found live: 1-2s store outages left cells permanently /// `RestoreFailed` — "non-contiguous ltx files: have up to 0x95 but next /// file starts at 0xb3"). #[test] fn reactivation_does_not_seed_past_bucket_coverage() { let rt = tokio::runtime::Builder::new_current_thread() .enable_all() .build() .unwrap(); rt.block_on(async { let dir = std::env::temp_dir().join(format!( "celld-reactivation-gap-{}", std::process::id() )); std::fs::create_dir_all(&dir).unwrap(); let store = Arc::new(FlakyStore { inner: Arc::new(InMemory::new()), fail_puts: AtomicBool::new(false), }); let repl = LtxRepl::start_with_store_for_test(&dir, store.clone()); let cell = "FragmentCell:reactivation-gap";

// Ship a first batch so the bucket has proven coverage. let act = repl.activate(opts(cell)).await.unwrap(); let writer = rusqlite::Connection::open(&act.path).unwrap(); writer .execute_batch("CREATE TABLE t(x); INSERT INTO t VALUES (1);") .unwrap(); writer.execute_batch("INSERT INTO t VALUES (2);").unwrap(); assert!(matches!( repl.sync_wait(cell, 1, Duration::from_secs(10)).await, SyncWait::Durable )); let shipped = repl.covered_txid(cell, 1).await; assert!(shipped >= 1, "baseline batch must reach the bucket");

// The blip: further writes stage locally while the bucket refuses // their uploads. store.fail_puts.store(true, Ordering::SeqCst); writer.execute_batch("INSERT INTO t VALUES (3);").unwrap(); writer.execute_batch("INSERT INTO t VALUES (4);").unwrap(); assert!(matches!( repl.sync_wait(cell, 1, Duration::from_secs(10)).await, SyncWait::Failed ));

// The fence's footprint: the handle vanishes mid-upload and the // files stay put — what release() leaves behind. repl.release(cell, 1);

// The store recovers; the restart reactivates the local image and // takes new writes — whose uploads now land PAST any skipped gap, // the exact shape that makes later restores fail forever. store.fail_puts.store(false, Ordering::SeqCst); let act = repl.activate(opts(cell)).await.unwrap(); drop(act); writer.execute_batch("INSERT INTO t VALUES (5);").unwrap(); assert!(matches!( repl.sync_wait(cell, 1, Duration::from_secs(10)).await, SyncWait::Durable ));

// Everything staged must now be durable, and the L0 chain must be // contiguous from the start — the shape restore requires. let covered = repl.covered_txid(cell, 1).await; let files = repl.client_for(cell, 1).ltx_files(0, TXID(0)).await.unwrap(); let mut next = 1; for file in &files { assert_eq!( file.min_txid.0, next, "L0 chain must be contiguous at txid {next}" ); next = file.max_txid.0 + 1; } assert!( covered + 1 >= next, "bucket coverage ({covered}) must reach the last staged row ({})", next - 1 ); }); } } ```

We validated it locally by temporarily widening the `celld_internal_tests` gates on `start_with_store_for_test` and friends to plain `cfg(test)` (no other changes) — red on 0.3.0, green with the fix.

## Fix (deployed on our fleet since yesterday)

**1. Clamp the seed to what the bucket proves** (ltx_repl.rs, replacing the `if let Some(pos) = seed` block at ~793):

```rust let mut replica = Replica::new(db, self.client_for(cell, epoch)); // A reused local image can carry staged L0 rows the bucket never // received — a fenced or crashed process leaves exactly that — and // seeding past them skips their upload forever. Clamp to what the // bucket proves; re-uploading overlapping rows is idempotent by TXID. let seed = match seed { Some(pos) => { let covered = self.covered_txid(cell, epoch).await; if pos.txid.0 > covered { Some(Pos::new(TXID(covered), 0)) } else { Some(pos) } } None => seed, }; if let Some(pos) = seed { replica.seed_pos(pos); } ```

This also seeds `shipped_txid`/`durable_txid` truthfully (they derive from `seed`), so `all_shipped_tiered()` stops being vacuously true over a gap.

**2. Linger on halt to flush** (actor.rs, `Effect::Halt` arm): the core has already fenced — nothing is served or acknowledged — but the process stays alive as a flusher until `all_shipped_tiered()` (the existing ensemble-change barrier), bounded (we use 300s) so a dead store still fences the node:

```rust let ltx = match &self.host { Some(CellHost::V8(rm)) => rm.replication().map(|r| r.ltx()), _ => None, }; let fence = self.fence.clone(); crate::asyncrt::spawn(async move { if let Some(ltx) = ltx { let deadline = std::time::Instant::now() + std::time::Duration::from_secs(300); while !ltx.all_shipped_tiered() && std::time::Instant::now() < deadline { crate::asyncrt::sleep(std::time::Duration::from_millis(250)).await; } } let _ = fence.send(code); }) .detach(); ```

## End-to-end verification (live)

Killing the store for ~4s under a stream of writes, twice, on separate fresh worlds:

- **0.3.0**: node self-fenced; after restart the affected cells failed forever with `route failed: RestoreFailed` and the bucket listing showed the `0x95 → 0xb3` hole. - **patched**: fence still fired (correct), uploads flushed during the linger, and after restart the cells restored cleanly — write sequences continued on the same cells with unbroken history.

## Side note

`release()` is the unguarded sibling of the guarded `evict()` (the "DRIFTED #3" fidelity fix): any crash between staging and upload leaves the same state. The seed clamp covers that whole class, since it makes reactivation itself distrust the local image's position.