denoland/celldPublic

self-hosted, distributed Durable Objects

v0.4.0

Important upgrade note:

- A v0.3.0 fleet must stop completely before v0.4.0 is deployed. The two
  versions cannot share a fleet because the peer tunnel protocol and large
  Workers KV value references are incompatible.

Highlights:

- Add initial support for Workers KV, Queues, Workflows, and R2 bindings.
  celld can deploy these bindings from the corresponding Wrangler
  configuration.
- Add `celld dev` for local development. It rebuilds the application
  automatically and uses a persistent local object store, so it does not
  require Docker or a cloud bucket.
- Nodes can adopt a new deployment without restarting. Existing requests
  finish on the old deployment, while new requests use the new deployment.
  Durable Objects move at safe points and retain their storage and
  hibernatable WebSockets.
- Strengthen the Durable Object output gate. It now applies one ordering
  rule to responses, outbound calls, Queue deliveries, and WebSocket sends.
  Read-only output waits for earlier request or alarm writes to become
  durable.
- Route proxied fetch, RPC, and WebSocket calls through one versioned peer
  tunnel. Request bodies stream to the owner. celld does not retry a call
  after transmission starts because it does not retain a copy of the body.
- Improve graceful shutdown with batched handoffs, drain tokens, successor
  acknowledgements, and full snapshots that reduce successor replay.
  Configure the orchestrator stop grace to cover the drain-token wait,
  handoff, and no-progress interval.
- Add commands to list cells and inspect or manage Workers KV namespaces
  and Queues:
  - `celld cell list`
  - `celld kv get`, `celld kv put`, `celld kv delete`, `celld kv list`,
    and `celld kv info`
  - `celld kv bulk get`, `celld kv bulk put`, and
    `celld kv bulk delete`
  - `celld queue info`, `celld queue peek`, `celld queue purge`,
    `celld queue pause`, `celld queue resume`, and
    `celld queue redrive`
  The cell and KV listings support pagination, full traversal, and
  newline-delimited JSON.
- Move the public health endpoint from `/__celld/health` to
  `/.well-known/celld/health`. Update load-balancer and readiness checks
  when you upgrade.
- Add a configurable request-body limit, with a 1 GiB default, and a
  per-cell concurrency limit. On Linux, memory-pressure handling now also
  accounts for active cgroup memory.
- Add support for Amazon EKS Pod Identity credentials from the injected
  environment variables and authorization-token file.

- Issues fixed:
  - https://github.com/denoland/celld/issues/148
  - https://github.com/denoland/celld/issues/161
  - https://github.com/denoland/celld/issues/163
  - https://github.com/denoland/celld/issues/165
  - https://github.com/denoland/celld/issues/166
  - https://github.com/denoland/celld/issues/167
  - https://github.com/denoland/celld/issues/168
  - https://github.com/denoland/celld/issues/170

Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Divy Srivastava <me@littledivy.com>
Co-authored-by: Kyle Kelley <rgbkrk@gmail.com>
Co-authored-by: Nathan Whitaker <17734409+nathanwhit@users.noreply.github.com>
Co-authored-by: Phil Renaud <phil@riotindustries.com>
Co-authored-by: Yusuke Tanaka <yusuktan@maguro.dev>
YTPRNWKKDSBIRDRyan Dahl and 6 others authored
a52f990Patch
Parent89e4ffc
164 files changed+36543-1510711 binary
StackedSplit
.github/workflows/release.yml+3 −4
.gitignore+1 −0
Cargo.lock+190 −132
Cargo.toml+12 −1
LICENSE.tokio+21 −0
README.md+186 −115
clippy.toml+23 −18
crates/celld/Cargo.toml+10 −2
crates/celld/actor.rs+1441 −187
crates/celld/actor/production.rs+5 −7
crates/celld/assets.rs+204 −6
crates/celld/asyncrt.rs+28 −3
crates/celld/bucket.rs+738 −52
crates/celld/cell_cli.rs+250 −0
crates/celld/cli_options.rs+142 −0
crates/celld/cli_output.rs+494 −0
crates/celld/control_plane.rs+121 −116
crates/celld/d1_cli.rs+161 −261
crates/celld/dead_node_gc.rs+4 −6
crates/celld/deploy.rs+754 −43
crates/celld/dev.rs+621 −0
crates/celld/drain_token.rs+223 −0
crates/celld/env_vars.rs+20 −0
crates/celld/fleet.rs+381 −80
crates/celld/generation.rs+420 −0
crates/celld/host_services.rs+8 −56
crates/celld/js.rs+3304 −514
crates/celld/js/bootstrap.rs+571 −63
crates/celld/js/crypto.js+37 −53
crates/celld/js/crypto.rs+267 −141
crates/celld/js/harness.js+5122 −512
crates/celld/js/headers.js+211 −37
crates/celld/js/modules.rs+72 −42
crates/celld/js/node_buffer.js+31 −7
crates/celld/js/node_events.js+1 −2
crates/celld/js/node_util.js+1 −1
crates/celld/js/r2_ops.rs+1286 −0
crates/celld/js/storage_ops.rs+149 −47
crates/celld/js/v8_strings.rs+65 −0
crates/celld/js/websocket.rs+239 −82
crates/celld/kv_cli.rs+775 −0
crates/celld/lib.rs+345 −65
crates/celld/local_store.rs+557 −0
crates/celld/ltx_repl.rs+1880 −305
crates/celld/machine.rs+5 −19
crates/celld/main.rs+2782 −1079
crates/celld/main/cli.rs+76 −9
crates/celld/main/peer_tunnel.rs+691 −0
crates/celld/main/websocket.rs+240 −386
crates/celld/memory.rs+108 −9
crates/celld/node_log.rs+2517 −530
crates/celld/operator_cell.rs+251 −0
crates/celld/otlp.rs+0 −137
crates/celld/ownership_store.rs+96 −9
crates/celld/peer_auth.rs+22 −5
crates/celld/peer_probe.rs+2 −2
crates/celld/pool.rs+239 −9
crates/celld/protocol.rs+223 −1
crates/celld/queue_cli.rs+221 −0
crates/celld/replication.rs+20 −6
crates/celld/runtime.rs+1599 −373
crates/celld/startup.rs+0 −6
crates/celld/storage.rs+289 −245
crates/celld/telemetry.rs+33 −284
crates/celld/wake.rs+6 −44
crates/celld/ws_client.rs+1 −1
crates/logic/Cargo.toml+5 −0
crates/logic/drain.rs+154 −0
crates/logic/http.rs+227 −0
crates/logic/isolate.rs+6 −4
crates/logic/kv.rs+280 −0
crates/logic/lib.rs+1137 −330
crates/logic/log_evict.rs+159 −19
crates/logic/log_tier.rs+42 −1
crates/logic/output_gate.rs+286 −0
crates/logic/pressure.rs+75 −26
crates/logic/queue.rs+365 −0
crates/logic/schedule.rs+2 −2
crates/logic/sqlite.rs+1 −1
crates/logic/sweep.rs+12 −0
crates/logic/types.rs+201 −42
crates/logic/wake.rs+1 −1
crates/ltx/Cargo.toml+7 −7
crates/ltx/README.md+11 −28
crates/ltx/reference/litestream-go/testdata/wal-reader/frame-checksum-mismatch/wal+0 −0

Binary file not shown.

crates/ltx/reference/litestream-go/testdata/wal-reader/frame-salts/wal+0 −0

Binary file not shown.

crates/ltx/reference/litestream-go/testdata/wal-reader/ok/wal+0 −0

Binary file not shown.

crates/ltx/reference/litestream-go/testdata/wal-reader/salt-mismatch/wal+0 −0

Binary file not shown.

crates/ltx/scripts/capture-golden.sh+0 −74
crates/ltx/scripts/db_equal.sh+0 −72
crates/ltx/src/bundle.rs+3 −61
crates/ltx/src/client/file.rs+0 −36
crates/ltx/src/client/mod.rs+6 −114
crates/ltx/src/client/object_store.rs+85 −281
crates/ltx/src/compaction_level.rs+0 −57
crates/ltx/src/compactor.rs+0 −240
crates/ltx/src/db.rs+784 −941
crates/ltx/src/host.rs+28 −24
crates/ltx/src/lib.rs+24 −14
crates/ltx/src/ltx.rs+5 −242
crates/ltx/src/lz4_block.rs+8 −59
crates/ltx/src/replica.rs+170 −504
crates/ltx/src/replica_compactor.rs+0 −149
crates/ltx/src/replica_url.rs+0 −278
crates/ltx/src/wal.rs+71 −367
crates/ltx/tests/differential_xtool.rs+0 −585
crates/ltx/tests/faults_inject.rs+0 −441
crates/ltx/tests/fixtures/golden/.gitkeep+0 −1
crates/ltx/tests/fixtures/golden/MANIFEST.md+0 −38
crates/ltx/tests/fixtures/golden/replica/ltx/0/0000000000000001-0000000000000001.ltx+0 −0

Binary file not shown.

crates/ltx/tests/fixtures/golden/replica/ltx/0/0000000000000002-0000000000000002.ltx+0 −0

Binary file not shown.

crates/ltx/tests/fixtures/golden/replica/ltx/0/0000000000000003-0000000000000003.ltx+0 −0

Binary file not shown.

crates/ltx/tests/fixtures/golden/replica/ltx/0/0000000000000004-0000000000000004.ltx+0 −0

Binary file not shown.

crates/ltx/tests/fixtures/golden/replica/ltx/0/0000000000000005-0000000000000005.ltx+0 −0

Binary file not shown.

crates/ltx/tests/fixtures/golden/replica/ltx/0/0000000000000006-0000000000000006.ltx+0 −0

Binary file not shown.

crates/ltx/tests/fixtures/golden/sample.wal+0 −0

Binary file not shown.

crates/ltx/tests/fuzz_bundle.rs+0 −218
crates/ltx/tests/fuzz_parsers.rs+0 −539
crates/ltx/tests/integration_file.rs+0 −315
crates/ltx/tests/integration_resilience.rs+0 −449
crates/ltx/tests/integration_s3.rs+0 −397
crates/ltx/tests/litestream_helpers.rs+0 −525
crates/ltx/tests/property_roundtrip.rs+0 −243
docs/README.md+500 −186
docs/cloudflare-compat.md+319 −358
docs/fencing.md+0 −231
docs/guarantees.md+228 −0
docs/library-api.md+60 −0
docs/limitations.md+46 −53
docs/security.md+118 −94
docs/telemetry.md+4 −4
docs/testing.md+33 −23
examples/README.md+21 −6
examples/alarm/index.js+12 −4
examples/alarm/wrangler.jsonc+3 −1
examples/async/index.js+12 −24
examples/async/wrangler.jsonc+4 −2
examples/body/index.js+6 −6
examples/counter/index.js+7 −4
examples/counter/wrangler.jsonc+3 −1
examples/cron/index.js+2 −40
examples/cron/wrangler.jsonc+1 −6
examples/d1/index.js+4 −23
examples/d1/wrangler.jsonc+0 −4
examples/hello/index.js+4 −2
examples/kv/index.js+21 −0
examples/kv/wrangler.jsonc+11 −0
examples/r2/index.js+33 −0
examples/r2/wrangler.jsonc+8 −0
examples/router/index.js+5 −7
examples/router/wrangler.jsonc+3 −1
examples/rpc/index.js+12 −125
examples/rpc/wrangler.jsonc+4 −2
examples/vectordb/index.js+18 −5
examples/vectordb/wrangler.jsonc+4 −2
examples/wasm/Cargo.toml+1 −1
examples/wasm/README.md+9 −10
examples/wasm/src/lib.rs+0 −2
examples/wasm/wrangler.jsonc+3 −2
examples/webapi/index.js+8 −6
examples/workflow/index.js+36 −0
examples/workflow/wrangler.jsonc+12 −0
examples/wsclient/index.js+14 −26
examples/wsecho/index.js+4 −7