denoland/celld

Public

Static assets can fail during rolling deployments due to version skew

BP

During a rolling deployment, HTML may be served by the new celld version while a subsequent content-hashed asset request reaches an old node. Because that node’s `AssetResolver` only knows its startup asset index, the asset is missing and the request falls through to the Worker, producing a 400 instead of serving the asset.

Celld should guarantee that assets referenced by any active deployment remain available, or provide deployment/version affinity so HTML and asset requests consistently reach the same version. Missing `/assets/*` requests should also return 404 rather than fall through to the Worker.

This is the same static-asset version-skew problem documented by Cloudflare (https://developers.cloudflare.com/workers/versions-and-deployments/gradual-deployments/version-affinity/#static-assets).

  1. RY

    Plan: an asset request that misses the node's startup index consults the current deployment manifest from the bucket (one GET on first miss, cached), and serves the blob content-addressed — every version's bytes are already reachable by every node, so hashed assets stay servable through a roll with zero configuration. A `/assets/*` miss returns 404 and never falls through to the Worker. Unhashed paths can still skew for the duration of a roll (the same behavior as any load-balanced fleet); content-hashed filenames are the recommendation, and the docs will say so. Version affinity (Cloudflare's answer) is deliberately not built: it requires operator-configured client identity even there, and union resolution fixes the failure case without it.

  2. RY

    One correction to the plan above, from implementing it: the "404 rather than fall through" part was wrong, and the fix does not do it. A request that matches no asset invokes the Worker — that is the Cloudflare routing contract, and a blanket 404 on `/assets/*` would break any app whose Worker legitimately serves routes under that prefix. With union resolution the hashed-asset case never misses, so the fallthrough only happens for genuinely absent paths, where the Worker is the correct destination. The fix as landed: a miss consults the moved deploy pointer (throttled to one read per 5 s), adopts the newer deployment's checksum-validated index, and serves its assets from the content-addressed blob store — before `not_found_handling` applies, so single-page-application configs serve the new hashed asset rather than their stale index.html.

  3. ryclosed3 months ago on GitHub

  4. bartlomiejureferenced 554fa82872e62 months ago on GitHub

  5. bartlomiejureferenced a52f9905425b2 months ago on GitHub