denoland/celld

Public

DigitalOcean Spaces: If-Match header silently ignored on PUT, causing node self-fencing cascade

SP

### Summary

DigitalOcean Spaces has an S3 compatibility bug where the `If-Match` header on `PutObject` is silently ignored (or stripped), causing celld's distributed lease mechanism to fail continuously. This triggers a self-fencing cascade where nodes terminate themselves within seconds of startup.

We have **empirical evidence**, a **workaround patch**, and **30+ minutes of successful runtime logs** on a real DO Spaces deployment. We also built and ran a full WebSocket chat application on the patched binary to confirm end-to-end functionality.

### Reference

* **Fork with patch + debug logging:** [https://github.com/SpencerCooley/celld/tree/do-spaces-compat-debug](https://github.com/SpencerCooley/celld/tree/do-spaces-compat-debug)

* **Commit:** `59646d7` — _Workaround: DigitalOcean Spaces If-Match CAS bug + debug logging_

* * *

### Environment

* **celld version:** 0.2.2

* **Storage backend:** DigitalOcean Spaces (S3-compatible)

* **Region:** sfo3

* **Node deployment:** Single Docker container (`--network host`)

* **Credential method:** Explicit `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`

* **Test app:** WebSocket chat room with Durable Objects (hibernatable WebSockets + SQLite persistence)

* * *

### Symptoms

1. Node starts successfully, writes `fleet/peer-auth.json` to the bucket

2. HTTP listener comes up on `0.0.0.0:8080`

3. Within 10 seconds, node terminates with exit code 137 (self-fencing)

4. Logs show repeating cycle:

``` lease WARN lease lost; will retry... error=ServiceError(ErrorResponse...) ```

* * *

### Root Cause Analysis

#### celld Lease Mechanism

celld uses an S3-based distributed lease for cluster coordination. Each node periodically performs a compare-and-swap (CAS) on `leases/{node_id}.json` using `If-Match` with the current ETag.

The `put_cas` function in `crates/celld/bucket.rs` uses the object\_store crate's `PutMode::Update` with `UpdateVersion.e_tag` set to the expected ETag. This should:

* **Succeed** if the object's ETag matches (lease renewed)

* **Return** `Ok(None)` if the ETag doesn't match (another node claimed it)

* **Return** `Err` for ambiguous failures

#### DigitalOcean Spaces Bug

DO Spaces **does not respect** the `If-Match` header on `PUT` requests. Instead of returning HTTP 412 when the ETag doesn't match, it either:

* Silently ignores the header and overwrites the object

* Returns a generic error that breaks the CAS flow

The result: celld's `put_cas` always fails, the node thinks it lost the lease, and self-fences.

#### The Self-Fencing Chain

1. Node A starts, acquires lease via initial `PUT` (no `If-Match` needed)

2. Node A tries to renew lease via `put_cas(ETag=abc123)` — sends `If-Match: abc123`

3. **DO Spaces ignores** `If-Match`, overwrites the object, returns new ETag=`def456`

4. celld's `object_store` sees the response as a failure (or the new ETag doesn't match)

5. celld interprets this as **"another node won the lease"**

6. Node A triggers graceful shutdown (self-fencing) → exit 137

7. **No other node exists**, so the fleet has zero active nodes

8. Process repeats on restart → infinite loop

* * *

### Workaround Patch

We patched `crates/celld/bucket.rs` to detect the Spaces-specific failure pattern and fall back to a non-atomic read-then-write when the ETag still matches the expected value.

**Logic:** When `put_cas` gets a `Precondition` or `AlreadyExists` error:

1. Read the current object's ETag via `head(key)`

2. If the current ETag **matches** the expected ETag, the store is rejecting the conditional header itself (not the content)

3. Safely overwrite with `put(key, body)` — we know no other node changed it

4. Read the new ETag and return it

This is safe because:

* If the ETag **doesn't** match, another node _did_ change it → return `Ok(None)` (proper CAS rejection)

* If the ETag **does** match, DO Spaces is just broken → emulate CAS with simple PUT

#### Files Changed

| File | Change | |--- | ---| | crates/celld/bucket.rs | CAS fallback: when DO Spaces rejects conditional PUT but etag still matches, fall back to read-then-write | | crates/celld/main.rs | Added schedule_timer debug tracing for lease timer diagnostics | | crates/logic/lib.rs | Added lease CAS/fence debug logging throughout lifecycle |

#### Key Change in `bucket.rs`

```rust Err(Error::Precondition { .. } | Error::AlreadyExists { .. }) => { // Some S3-compatible stores (e.g. DigitalOcean Spaces) do not // support If-Match on PutObject and always return 412. // Verify whether the object is actually unchanged by reading // the current etag. If it matches, the store is rejecting // the conditional header itself, and we can safely overwrite. if let Some(expected_etag) = etag { if let Ok(Some((_, actual_etag))) = self.head(key).await { if actual_etag == expected_etag { eprintln!( "[DEBUG] put_cas: store rejected If-Match but etag matches — emulating CAS for {}" , key ); self.put(key, body.clone()).await?; if let Ok(Some((_, new_etag))) = self.head(key).await { return Ok(Some(new_etag)); } } } } Ok(None) } ```

* * *

### Verification

#### Pre-patch: Node self-fences immediately

``` $ docker run --rm --network host ghcr.io/denoland/celld \ --bucket s3://my-bucket --endpoint https://sfo3.digitaloceanspaces.com --region sfo3 \ --listen 0.0.0.0:8080 --advertise 127.0.0.1:8080

peer_auth INFO loaded 2 peers s3 INFO loaded 4 blobs watch INFO new bundle epoch=1 lease WARN lease lost; will retry... error=ServiceError(...)

(Exit 137 after ~10 seconds) ```

#### Post-patch: Node sustains 30+ minutes, 600+ lease renewals

``` [DEBUG] put_cas: store rejected If-Match but etag matches — emulating CAS for nodes/node_xxx.json [DEBUG] node lease CAS applied: etag="ea82f1f4ad36a695100fea6a6d07490b" [DEBUG] hold_node_lease timers armed: generation=1 renew_after=3333 fence_after=10001 now_mono_ms=576 celld listening on 0.0.0.0:8080 (ownership=s3) ... [DEBUG] put_cas: store rejected If-Match but etag matches — emulating CAS for nodes/node_xxx.json [DEBUG] node lease CAS applied: etag="2aa2a18d713ff957ebc7c2e177b4578a" [DEBUG] hold_node_lease timers armed: generation=602 renew_after=3333 fence_after=10001 now_mono_ms=2186331 ```

#### End-to-end test: Chat app running on patched celld

We deployed a WebSocket chat application (Worker + ChatRoom Durable Object) to the patched node:

* Multiple browser tabs connected via WebSocket to `ws://127.0.0.1:8080/ws?room=lobby`

* Messages persisted in DO SQLite storage

* Real-time broadcast between tabs worked correctly

* Node stayed alive serving HTTP + WebSocket traffic continuously

* * *

### Reproduction Steps

1. Create a DigitalOcean Spaces bucket in any region (tested on sfo3)

2. Export credentials:

```bash export AWS_ACCESS_KEY_ID="your-do-key" export AWS_SECRET_ACCESS_KEY="your-do-secret" export AWS_REGION="sfo3" ```

3. Start celld:

```bash docker run --rm --network host ghcr.io/denoland/celld \ --bucket s3://your-bucket \ --endpoint https://sfo3.digitaloceanspaces.com \ --region sfo3 \ --listen 0.0.0.0:8080 \ --advertise 127.0.0.1:8080 ```

4. Observe: node starts, then self-fences within 10 seconds

5. Apply patch from fork, rebuild, restart

6. Observe: node sustains indefinitely

* * *

### Recommendations

#### For celld Developers

1. **Add a** `--s3-compat=digitalocean` **flag** (or auto-detect via endpoint hostname) to apply known workarounds

2. **Document S3 backend compatibility matrix** — which providers support `If-Match` CAS properly

3. **Consider an opt-in non-atomic lease mode** for development/single-node use cases

4. **Add** `head()`**\-based CAS validation** as a fallback path for all backends (defensive)

#### For DigitalOcean

1. **Fix S3 API compatibility:** `If-Match` on PUT must be respected per AWS S3 specification

2. **Return HTTP 412 Precondition Failed** when `If-Match` does not match, instead of silently overwriting

3. **Document S3 compatibility gaps** so users know which features work

* * *

### Note on Multi-Node Safety

This workaround is **safe for single-node deployments** and acceptable for experimental multi-node setups where DO Spaces is the only backend. The fallback only triggers when:

* The store rejects `If-Match`

* **AND** the object's current ETag matches the expected ETag (proving no other node changed it)

For production multi-node clusters, a true S3-compatible backend (or DO Spaces fixing the bug) is required.

* * *

_Issue prepared with empirical testing on DigitalOcean Spaces sfo3 region. Fork branch contains full patch + debug logging for reproducibility._

  1. AB

    Thorough report — the fork, the debug logging, and 30+ minutes of runtime on a real deployment make this easy to reason about. One thing I'd flag before a workaround lands, because I think it changes what the right fix is.

    **The self-fencing cascade is the safe failure mode, not the bug.**

    `leaser.rs` builds mutual exclusion entirely out of conditional writes — `PutMode::Create` (`If-None-Match: *`) to acquire, `PutMode::Update(UpdateVersion { e_tag })` (`If-Match: <etag>`) to renew, with a losing CAS surfaced as `LeaseExistsError` so the node stands by read-only. There is no other primitive underneath it; the 412 *is* the lock.

    If DO Spaces silently ignores `If-Match` on `PutObject`, then `write_lease` can no longer fail when it should. What you're observing — nodes terminating within seconds — is celld correctly noticing the lock object no longer matches what it expects and standing down. That's a liveness failure. The failure it's protecting you from is **two nodes both holding a live lease and both writing LTX data**, which is a silent correctness failure you would not see in logs at all; you'd see it later as corrupted replica state.

    So a patch that relaxes the CAS check to keep nodes alive on Spaces trades a loud liveness failure for a quiet safety one. I'd argue that's strictly worse for a durable-object store, where the whole value proposition is that the data is right.

    **Suggested shape: detect the capability, refuse to guess.**

    Rather than a DO-specific branch, probe the backend's conditional-write semantics once at startup and fail fast when they're absent:

    1. `PutMode::Create` a probe object under the replica prefix, keep the returned ETag. 2. Attempt `PutMode::Update` against that key with a deliberately **wrong** `e_tag`. 3. A conformant backend returns `Error::Precondition`. If the write *succeeds*, the backend does not honour `If-Match` and cannot support the lease protocol. 4. On detection, exit with a specific error naming the backend and the unsupported precondition — not a generic lease error — so an operator gets the 30 minutes back that you spent finding this. 5. Delete the probe object.

    This is deterministic, costs two round-trips at boot, and generalises: Spaces is not the only S3-compatible store with partial conditional-write support, and the same probe catches the next one without a vendor list to maintain.

    If there's demand to run on non-conformant backends anyway, I'd make it an explicit opt-in flag whose documentation states plainly that mutual exclusion is not enforced and concurrent writers are possible — so it's a recorded operator decision rather than a silent degradation. The one thing I'd avoid is inferring "safe to proceed" from a write that appeared to succeed.

    Happy to put up a PR for the startup probe if that direction is useful, and equally happy to leave it with @SpencerCooley since the diagnosis and the fork are already theirs — just didn't want the capability question to get skipped on the way to a working binary.

  2. SpencerCooleymentioned3 months ago on GitHub

  3. SpencerCooleysubscribed3 months ago on GitHub

  4. SP
    Author

    Yeah. I like your fail fast idea of just at least making the reason why it is not working more obvious. Seems like more of a vendor issue. I just use digital ocean a lot and wanted to stay in the same dashboard for cloud stuff. Thought it was good to write something up so maybe the digital ocean people could take a look.

  5. MA

    Thanks @SpencerCooley — the diagnosis was right, and the debugging behind it (the fork, the 30 minutes of logs, the WebSocket app) is what made this quick to act on.

    celld now tests the conditional write against your bucket instead of assuming it works. `celld diagnose` runs a four-step contract — create, reject-create, update, reject-stale — and a store that applies either of the two writes it owes a rejection for is named as the fault:

    ``` fail bucket conditional write: the store applied a conditional write that carried a stale token; the store accepts If-Match / If-None-Match and does not enforce it, so two nodes can own one cell ```

    Each node runs the same test once at startup and refuses to serve when it fails, so the restart loop you saw becomes one message on the first boot. `CELLD_STORAGE_PROBE=0` skips the startup test, and `celld diagnose --read-only` skips the write probe for a read-only credential.

    This is not in a release yet — it lands in the next one.

    To be clear about what this does and does not do: it detects the DigitalOcean Spaces behaviour before it costs you a fleet. It does not make celld correct on Spaces. Spaces still cannot fence a cell, so celld cannot run on it.

  6. magurotunaclosed3 months ago on GitHub

  7. SpencerCooleymentioned3 months ago on GitHub

  8. SpencerCooleysubscribed3 months ago on GitHub