denoland/celld

Public

database corruption after eviction

BP

Note: I faced db corruption a few times and used AI to diagnose this. It wrote the following text. I am not sure the accuracy - it seems plausible, but it is above my current level of expertise to be sure. I typically I would not be a meat proxy like this, but the issue definitely is real and the corruption appears to not be happening after the suggested fix, so I figured I would post this.

## Summary

`celld` can reuse a preserved local SQLite database under the same ownership epoch after an on-demand or sticky eviction. The eviction path deletes the database's local LTX metadata. When the cell wakes, `Db::open()` therefore sees an empty LTX history and starts again at transaction ID 1, while replication continues writing into the existing remote epoch prefix.

The new transaction objects overwrite the beginning of the old epoch, but any old objects above the new transaction high-water mark remain. A later remote restore treats the combined objects as one transaction stream. Depending on the pages changed in each generation, this causes acknowledged data to roll back or produces a structurally malformed SQLite database.

This violates the fencing invariant documented in `docs/fencing.md`: one epoch must describe one append-only writer generation.

## Affected version

Observed on `celld` v0.2.1 (`ae8fac0`) with the in-process LTX replicator.

## Relevant code

- `crates/logic/lib.rs:3442-3446`: on-demand eviction calls `begin_eviction(id, false)`, retaining ownership and the current epoch. - `crates/celld/ltx_repl.rs:601-614`: eviction preserves `db.sqlite` as `db.evicted`, but unconditionally removes `db.sqlite-litestream`. - `crates/celld/ltx_repl.rs:403-436`: activation accepts and reuses a same-epoch `db.evicted` snapshot. - `crates/ltx/src/db.rs:450-457`: with no local LTX files, `Db::pos()` returns transaction position zero. - The next sync consequently writes `0000000000000001-0000000000000001.ltx` into a prefix that may already contain transactions 1 through N.

## Minimal regression test

The following unit test uses the existing in-memory object-store constructor. It demonstrates deterministic acknowledged-data rollback without requiring a large enough workload to damage a b-tree. It belongs in `crates/celld/ltx_repl.rs` under a `celld_internal_tests` test module.

```rust #[tokio::test] async fn same_epoch_eviction_must_not_restart_ltx_history() -> anyhow::Result<()> { use object_store::memory::InMemory; use rusqlite::Connection; use std::sync::Arc;

let local = tempfile::tempdir()?; let store = Arc::new(InMemory::new()); let repl = LtxRepl::start_with_store_for_test(local.path(), store); let cell = "Counter:repro"; let epoch = 7;

let active = repl .activate(ActivationOptions { cell, epoch, fresh: true, took_over: false, resume_local: false, }) .await?;

let db = Connection::open(&active.path)?; db.execute_batch( "CREATE TABLE counter (value INTEGER NOT NULL);\ INSERT INTO counter VALUES (0);", )?; repl.await_durable(cell, epoch, 1).await?;

// Leave a remote tail above transaction 1. for value in 1..=4 { db.execute("UPDATE counter SET value = ?1", [value])?; repl.await_durable(cell, epoch, value as u64 + 1).await?; } drop(db);

// This is the problematic combination: retain the epoch and SQLite file, // but discard its LTX metadata. repl.evict(cell, epoch, true).await;

let active = repl .activate(ActivationOptions { cell, epoch, // same epoch fresh: false, took_over: false, resume_local: false, }) .await?; let db = Connection::open(&active.path)?; db.execute("UPDATE counter SET value = 100", [])?; repl.await_durable(cell, epoch, 100).await?; drop(db);

// Remove the healthy local copy so the next activation must consume the // now-mixed remote stream. repl.evict(cell, epoch, false).await;

let restored = repl .activate(ActivationOptions { cell, epoch: epoch + 1, fresh: false, took_over: true, resume_local: false, }) .await?; let db = Connection::open(&restored.path)?; let value: i64 = db.query_row("SELECT value FROM counter", [], |row| row.get(0))?;

// Expected: 100. Before the fix, the stale tail from the first generation // is applied after the replacement transaction 1 and restores 4. assert_eq!(value, 100); Ok(()) } ```

To reproduce structural corruption instead of value rollback, replace the counter updates with transactions that repeatedly insert enough rows to split b-tree pages, delete them, and create/drop indexed tables. After the final remote restore, `PRAGMA integrity_check` reports orphaned pages, missing index entries, or duplicate page references.

## Production evidence

The affected ControlPlane cell had a clean local epoch 69 snapshot. Its remote epoch 69 prefix contained transaction objects from three different wake generations:

- transaction 1: last modified 2026-08-18; - transactions 2 through 11: last modified 2026-08-17; - transactions 12 and above: last modified 2026-08-15.

The local epoch 69 database passed `PRAGMA integrity_check`. Epoch 70, created by a remote restore from the mixed epoch 69 prefix, was the first local snapshot to fail integrity checking. Subsequent epochs inherited the damage. The failure included duplicate page references, orphaned pages, and rows missing from all indexes.

This timestamp stratification cannot be produced by a single append-only LTX generation. It exactly matches transaction numbering restarting at 1 on each same-epoch wake and overwriting only the prefix reached by that generation.

## Expected behavior

Every new LTX writer generation must use a fresh epoch, or a same-epoch local resume must retain and validate the complete local LTX position. A remote epoch must never contain objects from separate transaction-ID-zero lineages.

## Actual behavior

A preserved same-epoch SQLite snapshot is reopened without its LTX metadata. The next write begins at transaction 1 and unconditionally overwrites objects in the existing epoch prefix, leaving an incompatible tail.

## Suggested fix

The simplest safe rule is to advance the ownership epoch for every activation, including on-demand and sticky eviction wakes. The preserved previous-epoch SQLite snapshot can still be reused locally when ownership rules prove it safe.

If same-epoch wake must remain supported, eviction must preserve the associated LTX metadata atomically with the SQLite snapshot. If either part is missing, activation must acquire a new epoch rather than writing into the old prefix. Remote restoration into an already-used epoch also needs an explicit baseline position; it must not restart at transaction 1.

Defense in depth:

1. Before uploading, reject a local position lower than the highest remote L0 transaction for the same epoch. 2. Validate LTX lineage/checksum continuity during restore. 3. Run `PRAGMA quick_check` on a private restored image before publishing the cell runtime.

These checks should fail activation rather than publish a malformed database, but they do not replace the epoch/position fix.

## Recovery note

Fixing the writer prevents recurrence but does not repair an already mixed remote prefix. Recovery must start from the last clean local snapshot or export readable tables into a fresh SQLite image, then publish that image under a new epoch. The poisoned prefix must not be selected as a restore source again.

  1. RY

    Thank you for the bug report. We were able to repeat the error and are working on a fix.

  2. ryclosed3 months ago on GitHub

  3. bartlomiejureferenced 9c0ca09ef5d53 months ago on GitHub

  4. bartlomiejureferenced bc0e29cddf303 months ago on GitHub

  5. bartlomiejureferenced 89e4ffc53a143 months ago on GitHub

  6. nick-inkeepreferenced this in Durable Object storage.deleteAll() drops Litestream control tables and breaks WAL capture #1683 months ago on GitHub