Safe node retirement through the control plane
ewhauser opened ·
I'm working on a Kubernetes operator for celld (https://github.com/ewhauser/celld-operator) and need a way to know when a node's disk can safely be removed during scale-down or replacement.
The existing shutdown endpoint acknowledges the request, but neither that response nor process exit tells us whether the disk is still needed for recovery. In particular, the node may hold follower data for other nodes. I'd like celld to own that decision rather than have the operator inspect and interpret its recovery state.
I implemented a workaround in my fork that extends the shutdown endpoint with a disk-removal mode and exposes progress through the state endpoint. It stops accepting new work, drains outstanding work, and checks that both the node's own data and its follower obligations are recoverable without the disk. It then stays alive in a control-only state so the supervisor can observe completion before stopping the process and removing the disk. Requests are tied to a specific runtime incarnation and can be retried safely. If completion can't be confirmed, we retain the disk.
Would you be open to a patch for this? I'm happy to adapt the approach to how you'd like it to fit into celld's existing shutdown mechanism and control plane.
- PA
Thanks for sharing!
FYI, another project called celld-operator that started Aug 13 (last commit 2 weeks ago): https://celld-operator.io/docs/
Also, for anyone else curious: deno seems to be working toward launching their own [?]: https://celld.dev/control
- PA
Also, if you're looking to contribute a fix @ewhauser, I'd just make it as minimal and general as you can, and forward a patch to ry@deno.com as the Contributing section recommends. While it might feel polite to ask first, I'd say it might show your earnestness if you just submit with your best effort, and be willing to have it go unmerged with no hard feelings :)
ewhausermentionedlast month on GitHub
ewhausersubscribedlast month on GitHub
- EW
Yes, I was trying to be polite but sure can email the patch