denoland/celldPublic

self-hosted, distributed Durable Objects

celld deploy can silently pretend the fleet adopted it

LE

lemeb opened ·

`celld` adopts a new deployment without restarting the process. This is great, but it also means that during deployment and version upgrades, it removes the signal a k8 orchestrator would rely on. Therefore, correct deployment and stale fleet look like the same.

In our testing, we had a three-node 0.4.1 fleet where `celld deploy` exited 0, the pointer advanced, and yet the three nodes were still serving the previous version for 45 minutes. The only way to see that that nodes were refusing to serve the new version was in the `/reload` response body:

> HTTP 422 stateless Worker failed to load: instantiate: SyntaxError: The requested module 'cloudflare:workers' does not provide an export named 'DurableObject'

(To be clear, the underlying error above was fixed in 0.5.0, but the general deploy problem stays.)

An implementation idea would be to put the deployment version in the bucker under `nodes/node_<id>.json`, since it doesn't carry it yet. It would ensure that orchestrators wanting to check adoption just read the bucket, and don't have to probe the pods or the internal listener or stuff like that.

  1. lemebrenamed celld deploy can silently fail to celld deploy can silently pretend the fleet adopted itlast month on GitHub