Leaked `kv.list()` objects can cause confusing database deadlocks
thoughtpolice opened ·
I have a system I'm building on celld that uses Workflows extensively, which ran into this (a flaw in the KV layer). The below MWE is from Fable 5.1/Opus 5.5, the diagnosis is rewritten in my own prose.
This has been tested against `celld 0.6.0` but it also exists in earlier versions, too.
---
`ctx.storage.kv.list()` gives you a key/value iterator, backed by a SQLite prepared statement object. At some point in time, the LTX replicator will checkpoint the database backing the iterator, after which the statement object becomes invalid. However, this then breaks _everything_ and causes the DO itself to become unusable; all `kv.put()` or storage calls or SQL statement calls after this point will fail. The only way this can be undone is by opening a new `kv.list()` iterator in order to finalize/collect the old one so it can go away.
Workflows can hit this quite reliably. `step.waitForEvent` returns from inside a `kv.list()` loop when it consumes an event, so a Workflow that waits for events and writes about 4 MB afterwards starts failing `status()` and `sendEvent()` with `storage.transaction: database is locked`.
---
MWE: `wrangler.jsonc` & `index.js`
```json { "name": "kv-list-cursor-lock", "main": "index.js", "no_bundle": true, "compatibility_date": "2026-08-20", "durable_objects": { "bindings": [{ "name": "REPRO", "class_name": "Repro" }] }, "migrations": [{ "tag": "v1", "new_sqlite_classes": ["Repro"] }] } ```
```javascript import { DurableObject } from "cloudflare:workers";
export class Repro extends DurableObject { leak() { const kv = this.ctx.storage.kv; kv.put("a", 1); kv.put("b", 2);
// leak the cursor, then commit about ~3 MiB, going past the 1000-page threshold & forcing a checkpoint for (const _ of kv.list()) break; for (let i = 0; i < 200; i++) kv.put("blob" + i, "x".repeat(16384)); } put() { try { this.ctx.storage.kv.put("c", 3); return "ok"; } catch (e) { return e.message; } } drain() { for (const _ of this.ctx.storage.kv.list()); } }
export default { async fetch(request, env) { const repro = env.REPRO.getByName(new URL(request.url).pathname); await repro.leak(); const before = await repro.put(); // this will fail with "database is locked" await repro.drain(); // a new kv.list finalizes the old cursor and removes it const after = await repro.put(); // this will succeed return Response.json({ before, after }); }, }; ```
## Actual output
``` $ celld dev . --host 127.0.0.1 --port 8787 --no-watch & $ curl http://127.0.0.1:8787/run1 {"before":"storage.put: database is locked","after":"ok"} ```
## Expected output
I expect both `before` and `after` to return `ok` and not violate any other consistency guarantees, etc.
- TH
I suppose the more general issue here is: any kv cursor is not safe across background LTX checkpointing. You could probably just as easy write 1 million blobs inside a normal `for` loop without even "leaking" the iterator.
- BA
Thanks for the report, this is now fixed and will be released in the next version.
bartlomiejureferenced 9ff48569e312last month on GitHub
bartlomiejureferenced f2bf648663a6last month on GitHub
ewhauserreferenced this in Merge upstream celld v0.6.1 #90last month on GitHub