denoland/celldPublic

self-hosted, distributed Durable Objects

Paged restore fails under an instance-role credential chain: "paged reads over an ambient credential chain need a store this client built"

CO

celld v0.4.1, two-node fleet on EC2 (t4g.large, arm64, AL2023), bucket in us-west-2, credentials from the EC2 instance role (no `AWS_ACCESS_KEY_ID`), `CELLD_DURABILITY=fleet`, `CELLD_LTX_PAGED` at its default.

A session cell whose chain grew past 256 MiB (see my other issue about compaction never succeeding) was released after a durability timeout. Every restore since then fails:

``` celld restore failed for InvestigationDO:ee68a0ea…: paged reader InvestigationDO:ee68a0ea… e17: paged reads over an ambient credential chain need a store this client built ```

35 failures in the first five minutes, still failing an hour later. Clients get `503 route failed: RestoreFailed`. Neither node has a local copy (16 KB on one, empty on the other), so the cell is gone until this path works.

From `crates/ltx/src/client/object_store.rs`, `blocking_range_reader` takes the static keys from the config when they exist, and otherwise wants a credential provider from an `AmazonS3` this client built itself. celld's per-cell clients are constructed with `with_store` and share one pre-built store, so `self.s3` is never set and the ambient branch always errors. In other words, any fleet that authenticates through an instance role (or any other ambient chain) cannot restore a cell above the paged threshold. Below the threshold, clones work fine with the same credentials.

Building the provider from `self.config.build_s3()` in that branch looks like it would do it, at the cost of one extra client per paged restore. Setting static keys would work around it, but that's a step backwards from the instance role for us.

The 5 GB clone comment in `ltx_repl.rs` is why I'd rather not switch paged restores off on 8 GB nodes as the workaround.

  1. RY

    Thanks for the report. Will investigate!

  2. RY

    The fix has landed and will ship in 0.5.0 soon.

  3. ryclosed2 months ago on GitHub

  4. bartlomiejureferenced 9c2d299dd6ablast month on GitHub

  5. ryreferenced e519694581dclast month on GitHub

  6. ryreferenced 4250d366aceblast month on GitHub

  7. ryreferenced 3127cf8e307elast month on GitHub

  8. ryreferenced 12d5b6333fe5last month on GitHub