- Issues
- #150
Data loss when multiple large transactions are batched into one replication segment (→ permanent RestoreFailed)
dpsutton opened ·
## The bug
When **multiple large (multi-page) transactions are captured into a single LTX segment** — which happens when they commit in rapid succession, so more than one is pending capture the next time the replicator runs (e.g. under concurrent request load) — that segment omits some pages. Every write is acknowledged, every segment is checksum-valid, the TXID chain is contiguous — but pages of the final database are present in no segment. The cell then fails its **first restore** (handoff / restart / takeover) permanently:
``` restore <cell> e1@340: missing page 2762 in restore plan (incomplete backup) ```
Silent RPO=0 violation — discovered only at restore, long after the data was "durably" acknowledged.
## Reproduce (~2 min)
```sh docker run -d -p 4650:5000 motoserver/moto:latest curl -X PUT http://127.0.0.1:4650/repro
export AWS_ACCESS_KEY_ID=local AWS_SECRET_ACCESS_KEY=local AWS_REGION=us-east-1 npm i -D esbuild celld deploy . --bucket s3://repro --endpoint http://127.0.0.1:4650 --region us-east-1
# one node celld --bucket s3://repro --endpoint http://127.0.0.1:4650 --region us-east-1 \ --listen 127.0.0.1:63000 --internal-listen 127.0.0.1:63001 --advertise 127.0.0.1:63001 &
# 500 transactions × 250KB, 8 concurrent lanes node drive.mjs # N=500 LANES=8 by default
# graceful restart → forces a restore kill -TERM %1 ; sleep 30 celld --bucket s3://repro --endpoint http://127.0.0.1:4650 --region us-east-1 \ --listen 127.0.0.1:63000 --internal-listen 127.0.0.1:63001 --advertise 127.0.0.1:63001 & curl http://127.0.0.1:63000/count # => 500 "RestoreFailed … missing page N" ```
## repro
drive.mjs ```javascript // Drive N large transactions from LANES concurrent callers. // Concurrency causes the replicator to coalesce multiple large // transactions into one LTX segment — the path that drops pages. const N = Number(process.env.N ?? 500); const LANES = Number(process.env.LANES ?? 8); const PORT = Number(process.env.PORT ?? 63000); let next = 0, done = 0, errors = 0; async function lane() { while (true) { const i = ++next; if (i > N) return; const r = await fetch(`http://127.0.0.1:${PORT}/work`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ i }), }); if (!r.ok) errors++; else await r.json(); done++; } } await Promise.all(Array.from({ length: LANES }, lane)); console.log(`wrote ${done} txns, ${errors} request errors`); ```
index.js:
```javascript import { DurableObject } from "cloudflare:workers"; export class Min extends DurableObject { constructor(ctx, env) { super(ctx, env); this.ctx = ctx; ctx.storage.sql.exec("CREATE TABLE IF NOT EXISTS t (id INTEGER PRIMARY KEY, v BLOB)"); } async work(i) { // ~250KB blob → ~60 pages at 4KB; a big multi-page transaction. this.ctx.storage.sql.exec("INSERT INTO t (id, v) VALUES (?, ?)", i, new Uint8Array(250000).fill(i % 251).buffer); return { i }; } async count() { return this.ctx.storage.sql.exec("SELECT COUNT(*) n, MAX(id) m FROM t").toArray()[0]; } } export default { async fetch(req, env) { const u = new URL(req.url); const c = env.CELL.getByName("c"); if (u.pathname === "/work") return Response.json(await c.work((await req.json()).i)); if (u.pathname === "/count") return Response.json(await c.count()); return new Response("min"); } }; ```
## Two conditions are both necessary (controlled results)
| transactions | concurrent? | segments / txns | restore | |---|---|---|---| | small (one INT) | yes (8 lanes) | 511 / 2000 (coalesced) | ✅ clean | | **large (250KB)** | **yes (8 lanes)** | **340 / 500 (coalesced)** | ❌ **missing page** | | large (250KB) | no (1 caller) | 212 / 200 (1:1) | ✅ clean |
- Small transactions never fail, even when coalesced → size matters. - Large transactions restore fine when driven **sequentially** (1 segment per txn, no coalescing) → concurrency matters only because it triggers coalescing. - So: the defect is in **capturing multiple large transactions into one segment**, not concurrency control. The DO single-writer contract is intact — transactions still commit one at a time and the live DB is correct; concurrency matters only because a full request queue makes transactions commit with no gap between them, so several are pending capture at once and get bundled into one segment.
## Where to look
`DEFAULT_MIN_CHECKPOINT_PAGE_N = 1000` (crates/ltx/src/db.rs): the WAL is checkpointed once it grows past ~1000 pages. Large coalesced transactions cross that boundary within a single segment's capture window; pages resident in the WAL at checkpoint time appear to be lost from the segment. Suspect: WAL-frame collection / segment assembly when a checkpoint occurs mid-capture of a multi-transaction batch.
## My real world repro The above is LLM generated minimal reproduction. But this happened to me when i was uploading embeddings for ~4000 photos. (embedding, roll, frame, description) — one row per frame (~4,000 rows). Restarting my celld instance would show this cell as corrupted.
- BA
Thanks for the detailed report. This is now fixed and will ship in the next release.
bartlomiejureferenced 9c0ca09ef5d53 months ago on GitHub
bartlomiejureferenced bc0e29cddf303 months ago on GitHub
bartlomiejureferenced 89e4ffc53a143 months ago on GitHub