- Issues
- #152
Rewrapping a 64 MiB request body before Durable Object fetch exits a 1 GiB node
werkamsus opened ·
## Affected version
`celld` v0.2.1 (`ae8fac053d79f971bfcb996054bb43eb2f9b05da`), using `ghcr.io/denoland/celld:v0.2.1` in a 1 GiB Docker container.
## How we found this
We are evaluating [`littledivy/dgit`](https://github.com/littledivy/dgit) as the Git engine for a self-hosted Durable Object-backed Git backend.
Workerd is Cloudflare's open-source Workers runtime. Wrangler uses it for local development. Dgit targets the Workers and Durable Objects interfaces, and celld runs the same Worker code as a self-hosted runtime. Dgit's 69 MiB / 44,227-object push fixture completed through local Wrangler and Workerd. A 1 GiB celld v0.2.1 node reached its memory limit and failed the same dgit fixture.
This is not an equal memory benchmark: the local Workerd run did not have the same 1 GiB container limit. It shows only that the dgit engine could process the fixture on the other runtime path.
We then removed Git and dgit from the reproduction. The minimal Worker-to-Durable-Object example below reproduces the celld memory failure with a 64 MiB request body. This is why the underlying failure is reported here rather than treated as a dgit-only defect.
## Impact
Rewrapping an incoming 64 MiB request body as a new `Request` and forwarding it to a Durable Object exhausts a 1 GiB node. The node reaches the absolute RSS cap and exits 137. The equivalent path that forwards the original bytes-backed request completes and preserves the payload exactly.
Because the complete process exits, other resident cells would also lose their owner. This minimal test used one cell and did not include a neighboring-cell availability control.
## Controlled result
| Forwarded body | 7-byte non-UTF-8 payload | 64 MiB random payload | |---|---:|---:| | Original incoming `request` | SHA-256 matched | SHA-256 matched | | `new Request(request.url, { method, headers, body: request.body })` | SHA-256 matched | Node exited 137 |
The test does not show binary corruption. It shows a memory failure specific to the rewrapped-stream path.
## Minimal Worker
```js export class Probe { async fetch(request) { const bytes = new Uint8Array(await request.arrayBuffer()); const digest = await crypto.subtle.digest("SHA-256", bytes); const hash = Array.from( new Uint8Array(digest), (byte) => byte.toString(16).padStart(2, "0"), ).join(""); return Response.json({ bytes: bytes.byteLength, hash }); } }
export default { async fetch(request, env) { const url = new URL(request.url); const forwarded = url.searchParams.get("mode") === "stream" ? new Request(request.url, { method: request.method, headers: request.headers, body: request.body, }) : request; return env.PROBE.get(env.PROBE.idFromName("probe")).fetch(forwarded); }, }; ```
`wrangler.jsonc`:
```jsonc { "name": "binary-probe", "main": "index.js", "compatibility_date": "2026-01-01", "durable_objects": { "bindings": [{ "name": "PROBE", "class_name": "Probe" }] }, "migrations": [{ "tag": "v1", "new_sqlite_classes": ["Probe"] }] } ```
## Reproduce
Deploy the Worker to a v0.2.1 node with a 1 GiB container memory limit, then create a 64 MiB binary payload:
```sh dd if=/dev/urandom of=payload.bin bs=1048576 count=64 status=none sha256sum payload.bin ```
Control, forwarding the original request:
```sh curl --data-binary @payload.bin \ -H 'content-type: application/octet-stream' \ http://127.0.0.1:8080/probe ```
The response reports 67,108,864 bytes and the same SHA-256.
Failing path:
```sh curl --data-binary @payload.bin \ -H 'content-type: application/octet-stream' \ 'http://127.0.0.1:8080/probe?mode=stream' ```
The client receives an empty reply. The node logs:
```text shedding on the absolute resident-set cap: rss_bytes=1061883904 in_use_bytes=976824384 ```
The supervised Docker process exits 137.
## Relevant source path
- Public ingress collects and copies the complete request before Worker dispatch: `crates/celld/main.rs:3081-3102`. - `Request` stream draining concatenates all chunks and eagerly computes decoded text: `crates/celld/js/harness.js:139-157`. - Durable Object forwarding drains a stream-backed request before dispatch: `crates/celld/js/harness.js:3014-3038`.
The controlled test indicates that several complete representations remain live during the rewrapped-stream path.
## Expected behavior
A Worker should be able to rewrap and forward a binary request body to a Durable Object without exhausting the node. If the request exceeds an accepted bound, celld should reject the active request before allocation. The process and neighboring cells should remain available.
## Acceptance
1. The 64 MiB rewrapped-stream case completes under a 1 GiB node limit and preserves SHA-256, or returns a bounded request-level rejection. 2. The process does not exit. 3. A regression test covers both the original bytes-backed path and the rewrapped-stream path.
werkamsussubscribed3 months ago on GitHub
bartlomiejureferenced this in TextDecoder.decode is superlinear, so text() on a large body is slow #1543 months ago on GitHub
- BA
Thanks for the report, this is now fixed and will ship in the next release.
bartlomiejureferenced 9c0ca09ef5d53 months ago on GitHub
bartlomiejureferenced bc0e29cddf303 months ago on GitHub
bartlomiejureferenced 89e4ffc53a143 months ago on GitHub