denoland/celldPublic

self-hosted, distributed Durable Objects

Recovery cannot read a follower tail larger than 10 s of transfer

LU

lukeed opened ·

Node-log recovery cannot read a large follower tail, because `/peer/log/tail` uses the shared log peer client with a 10 s whole-request timeout (`SignedPeerTransport::new`, `node_log.rs`, `.timeout(Duration::from_secs(10))`). `post_tail` passes no deadline, and the response contains the whole retained fragment.

## What happened (0.6.1, 3 nodes, cross-region)

A dead session's follower fragment was 607 MB (5,291 entries) after a heavy import. The member answered `/peer/log/seal`, then every tail read failed after about 10 s:

``` WARN celld::node_log: recovery could not read a sealed member's tail member="node_4c022e98…" dead="node_941128…" epoch=4 error=error decoding response body ```

At the same moment, the member logged `incomplete_message` from the recovering node. Recovery then refuses to seal ("no complete true witness"), so the session stays open. A node whose own predecessor is that session cannot start: `recover_self` fails four times, then "refusing to install a lease over an unrecovered predecessor log".

## Suggestion

Page the tail by sequence range (the request already carries the range evidence), or stream it with a progress timeout instead of a whole-request timeout. The bundle drain already reads in 512 MiB windows (`RECOVERY_GATHER_WINDOW_BYTES`); the follower gather could use the same bound.

  1. lukeedreferenced this in Fleet restart cannot recover without a stable CELLD_NODE #249last month on GitHub

  2. lukeedreferenced this in fix: name nodes by hostname #153last month on GitHub

  3. CN
    cnicolov

    Experiencing similar behavior

  4. TO
    toluwaanimi

    Another option is to keep large recovery reads off the peer path.

    Instead of fetching a whole retained follower tail from another node, recovery could read immutable restore inputs from object storage, with a small root object pointing at the current restore set. Peer traffic then stays mostly coordination/control, and large state transfer goes through the storage backend.

    For celld’s current design, paging the tail by sequence or byte range is probably the smaller fix. But object-store-led restore seems like a useful alternative model for avoiding this class of timeout.

  5. lukeedreferenced this in celld: whole-fleet stop leaves node logs open (upstream #250) #217last month on GitHub

  6. lukeedreferenced this in fix: seal logs on fleet stop #219last month on GitHub

  7. lukeedreferencedlast month on GitHub