denoland/celld

Public

Durable Object storage.deleteAll() drops Litestream control tables and breaks WAL capture

NI

`storage.deleteAll()` on a Durable Object drops `_litestream_seq` and `_litestream_lock` from the cell's SQLite database. WAL capture for the cell's current epoch then fails continuously, and a later write fails the output gate with `DurabilityUnproven` (HTTP 500).

The `deleteAll()` call itself returns 200. The failure surfaces up to 60s later, in a different request.

| | | |---|---| | Version | v0.3.0 (`89e4ffc5`), also v0.0.2–v0.2.1 | | Platform | macOS arm64, release binary | | Store | MinIO, S3 endpoint |

## Root cause

`delete_all_with_alarm` picks tables by schema scan (`crates/celld/storage.rs:3418-3423`):

```rust let mut statement = c.prepare( "SELECT name FROM sqlite_schema WHERE type='table' AND name NOT LIKE 'sqlite_%'", )?; ```

and drops everything except a two-entry allowlist (`:3428-3436`):

```rust for table in tables { if table == "_cf_METADATA" || (!delete_alarm && table == "_cf_ALARM") { continue; } let quoted = table.replace('"', "\"\""); c.execute_batch(&format!("DROP TABLE IF EXISTS \"{quoted}\";"))?; } ```

The vendored Litestream port creates its control tables in that same database (`crates/ltx/src/db.rs:246-251`):

```rust conn.execute_batch( "CREATE TABLE IF NOT EXISTS _litestream_seq (id INTEGER PRIMARY KEY, seq INTEGER);\ CREATE TABLE IF NOT EXISTS _litestream_lock (id INTEGER);", )?; ```

They match neither `sqlite_%` nor the allowlist, so both are dropped. `without_sql_authorizer` (`:3425`) disables the authorizer around the loop, so nothing else rejects it.

The doc comment on `deleteAll()` says "Delete all user storage" (`:3413`), but the enumeration doesn't encode that scope.

## Why the delay

The dropped tables are only touched on the checkpoint path — `INSERT INTO _litestream_lock` (`crates/ltx/src/db.rs:1174`) and `SELECT COUNT(1) FROM _litestream_seq` (`:398`). `DEFAULT_CHECKPOINT_INTERVAL` is 60s (`:182`), so writes keep acking normally until the first checkpoint after `deleteAll()`. Testing inside that window shows nothing wrong.

## Reproduction

A DO that exposes `put` and `deleteAll`:

```js export class Wedge { constructor(state) { this.state = state; } async fetch(request) { const p = new URL(request.url).pathname; if (p === "/put") { await this.state.storage.put("k", "v"); return new Response("ok\n"); } if (p === "/deleteall") { await this.state.storage.deleteAll(); return new Response("ok\n"); } return new Response("?\n", { status: 404 }); } } export default { async fetch(request, env) { return env.WEDGE.get(env.WEDGE.idFromName("cell-1")).fetch(request); } }; ```

```bash celld deploy ./worker --bucket s3://wedge --endpoint http://127.0.0.1:19500 CELLD_TEST_DATA_DIR=./data celld --bucket s3://wedge \ --endpoint http://127.0.0.1:19500 --listen 127.0.0.1:19510 &

curl 127.0.0.1:19510/put # 200 sqlite3 ./data/*/ltx/e1/db.sqlite "SELECT name FROM sqlite_schema WHERE type='table' ORDER BY name;" # _cf_ALARM, _cf_KV, _cf_METADATA, _litestream_lock, _litestream_seq

curl 127.0.0.1:19510/deleteall # 200 sqlite3 ./data/*/ltx/e1/db.sqlite "SELECT name FROM sqlite_schema WHERE type='table' ORDER BY name;" # _cf_ALARM, _cf_KV, _cf_METADATA <- both _litestream_* gone

curl 127.0.0.1:19510/put # within ~60s: hangs ~10s, then HTTP 500 # Worker failed: rejected: Error: route failed: DurabilityUnproven ```

Deterministic across two clean runs: 2/2 control tables present before, 0/2 after.

Node log after `deleteAll()`, retried roughly every 52ms:

``` WARN celld::ltx_repl: ltx wal capture failed error=no such table: _litestream_lock WARN celld::ltx_repl: ltx wal capture failed error=no such table: _litestream_seq ```

In an unbounded run the storm reached 3025 lines over 156s and was still firing when the node was stopped.

## Scope

The cell recovers on its own by failing over to a new epoch (e1 → e2), after which writes ack and replicate normally — so this is a transient data-plane failure, not a permanent wedge. No silent durability loss was observed: every write returning 200 after recovery produced a corresponding LTX segment.

Related: #158 was the same invariant on a different path (the eviction path deleting local LTX metadata). This reproduces at `89e4ffc5`, the commit that closed it.

  1. bartlomiejuassigned bartlomieju3 months ago on GitHub

  2. bartlomiejuadded label bug3 months ago on GitHub

  3. bartlomiejuadded label wontfix3 months ago on GitHub

  4. BA

    Thanks for the report. This was already fixed after v0.3.0 shipped and will be released this week.

  5. bartlomiejuclosed3 months ago on GitHub

  6. bartlomiejuremoved label bug3 months ago on GitHub

  7. bartlomiejuremoved label wontfix3 months ago on GitHub

  8. bartlomiejureferenced 554fa82872e62 months ago on GitHub

  9. bartlomiejureferenced a52f9905425b2 months ago on GitHub

  10. werkamsusreferenced this in Support permanent deletion of a cell and its replicated state #1752 months ago on GitHub