denoland/celld

Public

LTX ignores AWS_ENDPOINT and falls back to the non-FIPS S3 endpoint

CP

# LTX ignores AWS_ENDPOINT and falls back to non-FIPS S3 endpoint

## Problem

In an AWS GovCloud environment that requires virtual-hosted FIPS S3 endpoints, Celld can deploy and load a Worker but cannot activate SQLite Durable Objects.

Environment:

- Celld `v0.3.0` - AWS GovCloud `us-gov-west-1` - OIDC/web-identity authentication - FIPS S3 endpoint required

We configure the bucket-qualified virtual-hosted endpoint:

```text AWS_ENDPOINT=https://BUCKET.s3-fips.us-gov-west-1.amazonaws.com ```

The primary S3 client honors this through `AmazonS3Builder::from_env()`. The LTX replication client does not read `AWS_ENDPOINT` and instead derives the standard virtual-hosted, non-FIPS endpoint:

```text https://BUCKET.s3.us-gov-west-1.amazonaws.com ```

That endpoint is blocked by the environment, causing Durable Object activation to fail with `RestoreFailed`.

## Attempted `S3_ENDPOINT` workaround

We also tested the generic FIPS service endpoint:

```text S3_ENDPOINT=https://s3-fips.us-gov-west-1.amazonaws.com ```

Celld gives this setting precedence for both S3 clients and forces path-style addressing:

```text https://s3-fips.us-gov-west-1.amazonaws.com/BUCKET/KEY ```

With this configuration, Celld starts and stateless Worker routes such as `/health` and `/version` work. However:

- Static asset requests remain pending indefinitely. - Requests that activate a SQLite Durable Object remain pending indefinitely.

Therefore, `S3_ENDPOINT` is not a viable workaround in this environment. This behavior is consistent with the environment documentation requiring bucket-qualified, virtual-hosted FIPS endpoints. We did not observe an explicit AWS rejection because the affected requests remained pending.

## Expected behavior

When `--endpoint`/`S3_ENDPOINT` is not explicitly configured, the LTX client should honor `AWS_ENDPOINT` using virtual-hosted addressing, consistent with the primary S3 client.

## Proposed solution

When no explicit `--endpoint`/`S3_ENDPOINT` is provided, make LTX fall back to `AWS_ENDPOINT` with `force_path_style=false`.

Explicit endpoints should retain their current path-style behavior and precedence.

[Proposed patch](https://github.com/cdemetriou-prod/celld/commit/e8de903b77cd4c62ab64a8910cc4cbd72124764c)

The patch adds endpoint-resolution tests and does not change storage formats, bucket layout, or existing explicit-endpoint behavior.

  1. cdemetriou-prodclosed2 months ago on GitHub

  2. CP
    Author

    red herring, was able to get it working