No supported way for a bundle to learn which node it is running on after CELLD_VAR_* removal
samquarm opened ·
We run a self-hosted multi-region fleet on v0.4.1 and are blocked from moving to v0.5.0 by the removal of `CELLD_VAR_*`. I want to describe the use case in case it isn't one you had in view, and be concrete about the cost. I'm not arguing the removal was wrong.
## The requirement: one string per node
A bundle needs to know the name of the node it is running on. Nothing else.
Clients connect to a per-region endpoint that terminates on the node for that region. A short-lived credential is minted by our control plane and bound to the region it was minted for, so the serving node can reject one minted elsewhere:
```ts // simplified const ok = verifyCredential(secret, token, { subject, region: env.REGION, // the only per-node value we need }); if (!ok) return deny(); ```
This is defence in depth on top of the signature and subject checks. It stops a validly minted credential being replayed against a different region.
The shared secret is fleet-wide and moves to `vars` cleanly. The region name is the problem, because it is the one value that differs per node.
## Why the documented replacement doesn't cover it
`vars` is deployment-wide, and a fleet runs one application with one `deploy/current.json`, so there is no per-node override. We also can't infer the node from the request: `docs/cloudflare-compat.md` states the inbound `Request` has "a `cf` object without Cloudflare edge fields".
On v0.4.1 there was a second route, `globalThis.__cell.node`. v0.5.0 closes it deliberately (`docs/security.md`: "a bundle cannot name a host function, and `globalThis` carries no `__`-prefixed property"). **We think that hardening is correct and are not asking for it back.** We confirmed it is genuinely closed: on v0.5.0, `typeof globalThis.__celld` is `"undefined"` from bundle code, both under `celld dev` and on a real fleet node.
We understand the model behind it. `bootstrap.rs` says "Every cell scope routes through the host, whichever node owns it", so a bundle asking where it runs in order to make placement decisions is a category error.
Our use is narrower than placement. We do not want node identity to decide where anything goes, since request routing happens entirely in our own control plane before celld is involved. We want it only so a node can refuse a credential issued for somewhere else.
## The broader shape: Durable Objects at the edge
Stepping back from our own case, this seems inherent to what celld is for.
The product is self-hosted, distributed Durable Objects. The natural way to deploy that is the way Cloudflare deploys the thing it replicates: many nodes spread across locations, each one a distinct place. Operators adopting celld to get DO semantics without DO billing will tend to scale by adding nodes in more locations, not by making one node bigger. Every node added makes per-node identity more useful, not less.
There is also a capability gap worth naming. On Cloudflare, a Worker can learn something about where it is executing from `request.cf`. On celld the `cf` object carries no edge fields, which is entirely reasonable since celld can't prove geolocation it doesn't have. But the combination is what bites: celld provides no edge context on the request, and v0.5.0 removed the last remaining way for a bundle to learn its own node. A self-hosted fleet therefore has strictly less location awareness available to application code than the platform it is replicating, with no supported path to add it back.
We don't think the answer is to synthesise `cf` fields, which would be claiming knowledge celld doesn't have. A node's own name, which celld does know for certain and already injects into the isolate, seems like the honest primitive: not geolocation, not placement input, just "which of the operator's nodes am I".
## The cost of staying on v0.4.1
`docs/security.md` states that "Security fixes apply to the latest release only, so older alpha builds do not receive fixes". So pinning is not merely forgoing features. It means knowingly forgoing security fixes on a fleet that terminates public traffic. That is what makes this urgent rather than cosmetic for us.
## What would unblock us
Any supported way for a bundle to learn its own node name. Roughly in order of how well each seems to fit your model:
1. **Expose the node id on `env`.** You already compute and inject it (`bootstrap.rs`, `inject_routing`), so it is close to hand. Read-only, no new configuration surface, and it cannot be used to influence placement. 2. **A `ctx` field** such as `ctx.node`, if `env` is meant to stay purely config-derived. 3. **Restore the per-node overlay, but opt-in.** Details below.
## On option 3, with a working implementation
We prototyped this against v0.5.0 to check it was feasible before suggesting it. Roughly 60 lines across `env_vars.rs` and `fleet.rs`, gated behind an explicit `CELLD_ALLOW_NODE_VARS`, default off.
Opt-in specifically answers the reason you gave for the removal. `env_vars.rs` explains that a node started with a silently ignored `CELLD_VAR_` "would run with no override at all, which reads inside a Worker as a missing secret with nothing in the log", which is a real and nasty failure mode. An operator who had to explicitly enable the feature cannot be surprised by it, so the default stays safe and the sharp edge is only reachable by someone who asked for it.
Verified behaviour of the prototype:
| case | result | |---|---| | `CELLD_VAR_` set, no opt-in | hard fail, current message unchanged | | `CELLD_VAR_` set, opt-in `=1` | proceeds, value arrives in `env` | | `CELLD_VAR_` set, opt-in `=0` | hard fail | | opt-in set to a malformed value | loud reject via the existing `parse_flag`, `must be 0 or 1` | | no `CELLD_VAR_`, opt-in on | unchanged |
Two details we found worth getting right, in case they're useful regardless of what you decide:
* The overlay must read `std::env::vars_os()`, not `vars()`. The latter panics if any name or value in the environment is not valid unicode, including variables unrelated to celld. `validate()` already uses `vars_os()`. * The opt-in itself should go through `flag()` rather than a lenient match, so a typo is rejected loudly rather than silently leaving the feature off.
We deliberately did not restore `CELLD_VARS_FILE`. One string per node is the need, and leaving the file loader out keeps `worker_vars()`'s signature and its single caller unchanged.
Happy to share the diff if it is useful, or to test any of the three options on a real multi-region fleet and report back with per-node timing and lease behaviour.
## Environment
* celld v0.4.1 in production, v0.5.0 for the prototype above * Multi-region self-hosted fleet, Linux x86_64 and aarch64 * S3-compatible object store with conditional write support * Cell classes holding hibernatable WebSockets (`acceptWebSocket`, `getWebSockets`, `serializeAttachment`); one also uses `ctx.storage`
- RY
There's no equivalent CF api for this so I'm hesitant to add it.
- SA
## How we use celld, and what the per-node var is for
We run celld as a self-hosted Durable Objects replacement across a ten-node, multi-region fleet spanning several continents. It backs per-user realtime inboxes and per-community fanout channels. The same worker bundle is deployed to every node.
### Why we need a per-node variable
Each node has to know which region it is. Our bundle reads `CELLD_REGION` at startup and uses it to mint and validate realtime tickets. Without it the bundle cannot identify itself and returns 503.
On 0.4.x we set that with the `CELLD_VAR_<NAME>` overlay: one `CELLD_VAR_CELLD_REGION=<region>` per node, and the bundle sees `CELLD_REGION` set accordingly. One string per node, with the binary, the bundle and every other setting identical fleet-wide.
0.5.0 removed the overlay and fails at startup when `CELLD_VAR_*` is present.
### The patch, already forked and already running
We have forked 0.5.0, added this, and it is live on all ten nodes. It is not a proposal we are asking you to evaluate on paper.
About 60 lines across two files. It restores the overlay behind an explicit `CELLD_ALLOW_NODE_VARS=1`, default OFF. Without the opt-in, 0.5.0 behaves exactly as it does today, including the startup failure, so nobody who has not asked for this sees any change.
We deliberately did not restore `CELLD_VARS_FILE`. One string per node is all we need, and omitting it leaves `worker_vars()`'s signature and its single caller untouched, which keeps the merge small.
Running on 10/10 nodes on patched 0.5.0: zero restarts, health 200 on every node, and the per-node identity confirmed reaching the bundle.
We are happy to keep carrying the fork. We would rather not, mainly because a default-OFF opt-in seemed like the kind of thing that costs you nothing and saves every self-hoster the same 60 lines.
**Happy to share the code.** We can open a PR against 0.5.0 with the patch as it stands, or just paste the diff here if you would rather look before there is a PR open. Either way it is yours to take, modify or reject, and we are not precious about the shape: if you want it written differently, or gated differently, or named something else, tell us what you would accept and we will send that instead.
We can also share how we exercised it, since an opt-in like this is mostly a question of proving it does not change the default path. Our gate matrix covers the opt-in on and off, the variable present and absent, and the 0.5.0 startup failure still firing untouched when the flag is not set.
### Why a celld node needs this when a Cloudflare Worker does not
This may be the actual shape of the gap, and it could explain why there is no equivalent API to point at.
On Cloudflare, placement is Cloudflare's job. An isolate never needs to be told where it is, because it did not choose and cannot influence it, and anything it does need arrives per request as `request.cf.colo`.
Self-hosting moves placement to the operator. We decide which machine serves which region, so the process has to be told something Cloudflare would never have to tell it.
So it is not that Cloudflare has no notion of "where am I". It does, through `request.cf.*`. It is that Cloudflare delivers it per request, because the platform owns placement, while a self-hosted fleet needs it per process, because the operator does.
### How we route to the nearest node, for context
A Cloudflare Worker at the edge picks the node per request: `cf.colo` first, `cf.country` as a fallback, and a global default when neither resolves.
`colo` deliberately outranks `country`. The hop being optimised is worker to node, and the worker executes at the colo, whereas `country` is where the user is. Those two diverge exactly where the cost is highest. Traffic from some countries is routinely backhauled to PoPs on another continent, so country routing would dispatch a worker thousands of kilometres to a node in the user's home region while a far closer node sat in the same building as the executing isolate.
### R2 Local Uploads, worth knowing about, plus one open question for you
Unrelated to the patch, but probably the most useful thing we can hand back.
celld uses the bucket as its consensus substrate, and a bucket has a single immutable `location` hint. Our writes originate from ten nodes across several continents, so every node lease renewal paid the leg from its local PoP to the bucket's home region. That is the textbook worst case for this design, and it dominated our fencing behaviour.
Enabling R2 Local Uploads (Beta) on the celld bucket (`wrangler r2 bucket local-uploads enable <bucket>`) removed most of it. On our worst node, the conditional write that the node lease performs went from roughly 1.7s to 875 ms, and plain PUT p50 to 905 ms over 25 samples. A hardware change on that node landed in the same window and contributed roughly as much, so attribute about half of that to Local Uploads rather than all of it.
**The open question, and the reason we are raising it here rather than just being pleased.** The dashboard only advertises "improve performance for cross-region uploads", but `wrangler r2 bucket local-uploads get` spells out the semantics:
> "Object data is written to the nearest region first, then *asynchronously > replicated* to the bucket's primary region."
That is eventual consistency on the data path, underneath a consensus substrate. The concrete worry: node A writes its lease locally, node B reads that lease from another continent to decide whether A is alive. If replication lag ever exceeded the read, a peer could see a stale lease and conclude a healthy node is dead.
What we verified across five nodes: a deliberately stale etag still returns 412 everywhere, so the conditional-write contract is still serialised against something authoritative. What we did not verify: cross-region read freshness. Those are different guarantees, and celld appears to need both.
So, the question for you. Is a Local-Uploads-enabled bucket a configuration celld can safely sit on, or does the async replication break an assumption in the lease and fencing design that we would only discover under partition? We are watching our fence count and would disable it on the celld bucket first if it rises, but you would know from the design side far faster than we will learn it from the metrics.
### If the env-var shape is the objection
Would you be more open to exposing node identity at runtime instead, something on the request or context mirroring `request.cf.colo`, rather than as an environment overlay? That is the same information in the shape Cloudflare already uses, it stays closer to the Durable Objects model you are tracking, and it would solve this for us just as well.
We are glad to do that work rather than hand you a design and wait. Point us at the shape you want and we will write it, test it on the fleet we already have running, and send you the results alongside the patch.
- DI
We're also having issues with the new approach. Currently our services decouple deployment from env changes, meaning we'd now need a `celld deploy` in order to propagate any env var changes, instead of relying on setting the env var in the celld service.
I think an opt-in to CELLD_VAR_* variables would be great, but I'm not too sure what was the rationale for the removal, it was a genuinely useful way to set these values.
shynomesubscribedlast month on GitHub
shynomesubscribedlast month on GitHub