denoland/celldPublic

self-hosted, distributed Durable Objects

v0.2.0

celld v0.2.0 changes three foundations: how cells share memory, how
their state replicates, and how a fleet stays available when its
object store fails or slows.

Isolates and memory:

- Resident cells share isolates and run turn by turn on a shared
  pool. v0.1.0 held one OS thread and one isolate for each cell.
- A resident cell costs ~471 KB against ~3.4 MB in v0.1.0, measured
  linear to 2,500 resident cells (~1.2 GB) on one node. Memory no
  longer limits the resident population; admission and RSS shedding
  set the limit.
- A handler that awaits I/O releases its isolate between turns, so
  co-resident cells cannot stall each other.
- celld reclaims empty cell heaps and reuses retired isolate slots,
  so a long-lived node cannot leak isolates.
- jemalloc is the global allocator.

Two listeners:

- celld now separates the data plane from the control plane. The
  public listener (--listen) serves only your Worker's routes and
  /__celld/health. A new internal listener (--internal-listen)
  carries everything else: the operator API (/state, /shutdown) and
  all node-to-node peer traffic. Keep the internal port on a private
  network.
- A public listener on a non-loopback address requires an explicit
  internal listener. --advertise names the internal address, and
  celld diagnose probes peers at that address.

Replication and compaction:

- celld compacts many small level-0 replication objects into additive
  level-1 blocks (LTX v0.5.2 block format). Compaction is on by
  default and never deletes a source object.
- A takeover of a write-heavy cell reads tens of objects instead of
  thousands. In one measured case, a cell with 4,096 durable writes
  restored from 42 objects and 0.6 MB instead of 4,116 objects and
  19 MB, 3.8 times faster.
- A restore downloads the objects in its plan concurrently, under one
  ceiling for the whole node.
- A takeover pins the exact point in the replicated history that it
  restored from. celld never restores data that a fenced former owner
  uploads after that point, so a cell's history cannot fork and a
  failed write cannot reappear after a failover.
- Compaction runs off the request path with a bounded duty cycle. A
  node that restarts with a large uncompacted backlog drains it
  without blocking durable writes.
- celld and Litestream v0.5.16 can read both block and frame files.
  Litestream v0.5.11 can read only frame files.

Availability under storage faults:

- The time that remains on the last renewed lease bounds each lease
  renewal attempt, so one hung conditional write can no longer use
  the whole authority window. In a test that silently dropped all
  established storage connections for 20 seconds, a v0.1.0-class node
  fenced; a v0.2.0 node kept serving.
- The core event loop owns the lease timer on a dedicated thread.
  Restores, compaction, and diagnostics cannot delay it.
- celld logs each lease attempt at start and at completion, so a
  storage hang is distinguishable from a timer failure.

Shutdown and restart:

- SIGTERM starts a graceful drain. The drain hands each cell to a
  peer directly, so takeover does not wait for the lease TTL.
- POST /shutdown?handoff=preserve on the internal listener prepares a
  same-node restart. The replacement process resumes its local cell
  databases and performs no owner or replica reads.
- The drain waits for its ownership release writes and bounds each
  connection, so one idle connection cannot use the full deadline.
- The process exits without unwinding. This fixes a crash when a warm
  node stops.

Storage backends:

- A gs:// bucket selects Cloud Storage's generation-based
  compare-and-swap and authenticates with Application Default
  Credentials.
- A bucket can take a key prefix, so several independent fleets can
  share one bucket without collisions.

Workers surface:

- Wasm modules deploy and load as compiled WebAssembly.Module values.
  A workers-rs example shows the path, and older nodes refuse the new
  wasm-v1 deployment feature at deploy time.
- A new RPC example exercises entrypoints, callbacks, RpcTarget
  returns, and promise pipelining end to end. Cross-isolate stubs
  remain limited to single method calls on named entrypoints.
- Web Crypto and node:crypto expand: asymmetric import and export
  (SPKI, PKCS#8, JWK, PEM), generateKey, P-384 and P-521, Ed25519 and
  X25519, ECDH derivation, AES-CBC and AES-CTR, DigestStream,
  timingSafeEqual, and one-shot sign and verify.
- setWebSocketAutoResponse answers ping-style messages without waking
  the cell.

OpenTelemetry support (off by default):

- CELLD_OTEL=1 turns on OpenTelemetry traces and logs with no
  collector required: celld writes them directly to the bucket as
  Parquet under a telemetry/ prefix, and DuckDB can query them.
- A standard OTLP/HTTP collector is the alternative sink
  (CELLD_OTEL_SINK=otlp), and the usual OTEL_* variables apply.
- W3C trace context propagates end to end: celld adopts an inbound
  traceparent, and fetch spans, cell dispatch, and console logs join
  the caller's trace.

Overload:

- A stateless request that finds every slot held now waits, bounded
  by CELLD_ADMISSION_WAIT_MS, instead of an immediate refusal. At
  overload, the node no longer spends its capacity on refusals.
- CPU-based shedding is removed. RSS shedding (on by default since
  v0.1.0) is the only pressure trigger.

Configuration:

- celld validates every variable at startup, and an invalid value
  stops startup. Boolean variables accept only 0 or 1.
- Removed: CELLD_WORKERS, CELLD_MAX_COHOSTED, CELLD_MAX_CPU_PERCENT,
  CELLD_RESIDENT_LOW_WATER (ignored since v0.1.0), CELLD_VALIDATE,
  and the /js route.
- Renamed: CELLD_HIBERNATIONS is now CELLD_EVICTIONS.
- Changed: CELLD_IDLE_EVICT_S no longer defaults to 300; idle
  eviction is off unless set. CELLD_UNSAFE_PUBLIC_ADVERTISE takes 1,
  not on.
- Added, among others: CELLD_INTERNAL_ADDR, CELLD_MAX_STATELESS_ISOLATES,
  CELLD_ADMISSION_WAIT_MS, CELLD_LTX_COMPACTION, CELLD_SHUTDOWN_DRAIN_MS,
  and the CELLD_OTEL family.

Upgrading from v0.1.0:

- A v0.1.0 command line with a non-loopback --listen refuses to start
  until an explicit --internal-listen is added.
- The upgrade must not be a rolling update: stop every v0.1.0 node,
  then start the v0.2.0 nodes. v0.2.0 advertises the internal
  listener, which v0.1.0 peers cannot follow, and writes compacted
  block objects, which a v0.1.0 reader cannot restore. A fleet must
  not mix the two versions.

Co-authored-by: Nathan Whitaker <nathan@deno.com>
Co-authored-by: Connor Hindley <conn.hindley@gmail.com>
Co-authored-by: Yusuke Tanaka <yusuktan@maguro.dev>
Co-authored-by: Heyang Zhou <hello@su3.io>
Co-authored-by: Daniel Lee <danielylee@google.com>
DLHZYTCHNWRDRyan Dahl and 5 others authored
3f22aedPatch
Parent553ae73
99 files changed+24219-140652 binary
StackedSplit
.github/workflows/release.yml+2 โˆ’16
Cargo.lock+441 โˆ’46
Cargo.toml+39 โˆ’3
Dockerfile+9 โˆ’6
README.md+58 โˆ’56
crates/celld/Cargo.toml+16 โˆ’3
crates/celld/assets.rs+20 โˆ’21
crates/celld/asyncrt.rs+13 โˆ’46
crates/celld/bucket.rs+387 โˆ’106
crates/celld/control_plane.rs+26 โˆ’25
crates/celld/dead_node_gc.rs+1 โˆ’1
crates/celld/deploy.rs+99 โˆ’16
crates/celld/env_vars.rs+167 โˆ’0
crates/celld/fleet.rs+77 โˆ’48
crates/celld/js.rs+4179 โˆ’6297
crates/celld/js/bootstrap.rs+553 โˆ’0
crates/celld/js/crypto.js+333 โˆ’13
crates/celld/js/crypto.rs+1528 โˆ’0
crates/celld/js/harness.js+527 โˆ’193
crates/celld/js/modules.rs+805 โˆ’0
crates/celld/js/node_crypto.js+245 โˆ’16
crates/celld/js/storage_ops.rs+709 โˆ’0
crates/celld/js/text_encoding.js+6 โˆ’1
crates/celld/js/url.js+8 โˆ’0
crates/celld/js/websocket.rs+847 โˆ’0
crates/celld/js/zlib.rs+256 โˆ’0
crates/celld/lib.rs+9 โˆ’16
crates/celld/ltx_repl.rs+623 โˆ’52
crates/celld/main.rs+1246 โˆ’1647
crates/celld/main/cli.rs+291 โˆ’0
crates/celld/main/machine.rs+245 โˆ’0
crates/celld/main/websocket.rs+831 โˆ’0
crates/celld/otlp.rs+350 โˆ’0
crates/celld/ownership_store.rs+81 โˆ’79
crates/celld/peer_probe.rs+2 โˆ’2
crates/celld/pool.rs+472 โˆ’0
crates/celld/protocol.rs+34 โˆ’0
crates/celld/replication.rs+11 โˆ’3
crates/celld/runtime.rs+1117 โˆ’529
crates/celld/startup.rs+42 โˆ’13
crates/celld/storage.rs+706 โˆ’375
crates/celld/telemetry.rs+1254 โˆ’0
crates/celld/wake.rs+25 โˆ’15
crates/logic/alarm.rs+5 โˆ’17
crates/logic/cache.rs+2 โˆ’2
crates/logic/gate.rs+145 โˆ’0
crates/logic/isolate.rs+303 โˆ’0
crates/logic/lib.rs+1128 โˆ’990
crates/logic/pressure.rs+18 โˆ’75
crates/logic/restore.rs+3 โˆ’4
crates/logic/routing.rs+18 โˆ’50
crates/logic/schedule.rs+0 โˆ’22
crates/logic/sqlite.rs+11 โˆ’36
crates/logic/types.rs+754 โˆ’0
crates/logic/wake.rs+4 โˆ’4
crates/ltx/Cargo.toml+2 โˆ’6
crates/ltx/LICENSE.pierrec-lz4+27 โˆ’0
crates/ltx/README.md+41 โˆ’11
crates/ltx/reference/litestream-go/testdata/db/enforce-retention/database+0 โˆ’0

Binary file not shown.

crates/ltx/reference/litestream-go/testdata/db/write-snapshot-to/database+0 โˆ’0

Binary file not shown.

crates/ltx/reference/ltx-format.md+25 โˆ’16
crates/ltx/scripts/guards.sh+0 โˆ’42
crates/ltx/src/client/file.rs+5 โˆ’51
crates/ltx/src/client/mod.rs+41 โˆ’65
crates/ltx/src/client/object_store.rs+163 โˆ’453
crates/ltx/src/codec.rs+362 โˆ’0
crates/ltx/src/compaction_level.rs+199 โˆ’0
crates/ltx/src/compactor.rs+435 โˆ’0
crates/ltx/src/db.rs+13 โˆ’13
crates/ltx/src/leaser.rs+0 โˆ’1665
crates/ltx/src/lib.rs+8 โˆ’17
crates/ltx/src/ltx.rs+185 โˆ’294
crates/ltx/src/lz4_block.rs+271 โˆ’0
crates/ltx/src/replica.rs+110 โˆ’78
crates/ltx/src/replica_compactor.rs+352 โˆ’0
crates/ltx/src/store.rs+0 โˆ’263
crates/ltx/src/wal.rs+9 โˆ’37
crates/ltx/tests/differential_xtool.rs+8 โˆ’8
crates/ltx/tests/faults_inject.rs+1 โˆ’11
crates/ltx/tests/integration_file.rs+2 โˆ’2
crates/ltx/tests/integration_minio.rs+4 โˆ’68
crates/ltx/tests/integration_resilience.rs+1 โˆ’1
crates/ltx/tests/litestream_helpers.rs+0 โˆ’34
docs/README.md+162 โˆ’27
docs/cloudflare-compat.md+39 โˆ’10
docs/fencing.md+107 โˆ’0
docs/limitations.md+11 โˆ’5
docs/security.md+99 โˆ’40
docs/telemetry.md+134 โˆ’0
docs/testing.md+4 โˆ’3
docs/wasm.md+78 โˆ’0
examples/README.md+6 โˆ’1
examples/rpc/index.js+136 โˆ’0
examples/rpc/wrangler.jsonc+7 โˆ’0
examples/wasm/.gitignore+3 โˆ’0
examples/wasm/Cargo.toml+14 โˆ’0
examples/wasm/README.md+28 โˆ’0
examples/wasm/src/lib.rs+38 โˆ’0
examples/wasm/wrangler.jsonc+8 โˆ’0