denoland/celld

Public

celld cannot use EKS Pod Identity: object_store has no container-credentials provider (0.3.0)

TB

## Summary

celld cannot use **EKS Pod Identity**, AWS's current recommendation for giving pods AWS credentials and the documented successor to IRSA. The bundled `object_store` build supports the environment, web-identity, and IMDS credential providers, but not the container-credentials provider that Pod Identity uses. There is no code path for it: the relevant environment variables do not appear in the binary at all.

This is a missing credential source rather than a detection-order bug, so I have filed it as a feature request rather than a defect.

IRSA works and is our workaround, so this is not blocking us. Reporting it because Pod Identity is what AWS now steers users to, and the failure mode is quiet: celld skips the injected credentials, falls through to IMDS, and reports the bucket as inaccessible after a timeout, which reads like a network fault rather than missing configuration.

## Version

`celld 0.3.0` (latest release at time of writing). Reproduced with both the `v0.3.0` `celld-x86_64-unknown-linux-gnu.gz` release asset and the `celld` binary inside `ghcr.io/denoland/celld:0.3.0`. The two are different builds of the same version but behave identically here.

## What happened

`celld deploy` on an EKS pod bound to an IAM role through a Pod Identity association. Bundling succeeds, then the bucket write fails:

``` celld 0.3.0 Bundled swarm-plane (0.22 sec) INFO object_store::aws::builder: Using Instance credential provider INFO object_store::client::retry: Encountered transport error backing off ... url (http://169.254.169.254/latest/api/token) Error: bucket unavailable or inaccessible: s3://<bucket> 1: Generic S3 error: Error after 1 retries in 30.1s ... error sending request for url (http://169.254.169.254/latest/api/token) ```

The `Using Instance credential provider` line is the significant part: celld never considered the injected credentials, it went straight to IMDS. IMDS is unreachable from that pod, and had it answered it would have returned the node's role rather than the role the pod is bound to.

## The platform side was correct

The pod had everything Pod Identity injects. From the live pod spec, on both the init container and the main container:

``` env: AWS_CONTAINER_CREDENTIALS_FULL_URI AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE AWS_STS_REGIONAL_ENDPOINTS, AWS_REGION, AWS_DEFAULT_REGION volumeMounts: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount ```

The association was active for the pod's `(namespace, serviceaccount)`, and the role's policy allowed the bucket operations. Switching the same role to IRSA, changing nothing else, made it work.

## Why it cannot work today

Literal string counts in the binary, identical in the release asset and in the image build:

| String | Count | | --- | --- | | `AWS_WEB_IDENTITY_TOKEN_FILE` | 1 | | `AWS_ROLE_ARN` | 1 | | `AWS_ROLE_SESSION_NAME` | 1 | | `WebIdentity` | 7 | | `AWS_ACCESS_KEY_ID` | 2 | | `Instance credential provider` | 1 | | **`AWS_CONTAINER_CREDENTIALS_FULL_URI`** | **0** | | **`AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`** | **0** | | **`AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`** | **0** | | **`ContainerCredentials`** | **0** |

``` strings -a celld | grep -c AWS_CONTAINER_CREDENTIALS_FULL_URI # => 0 ```

So the web-identity provider is compiled in with all three of its variables, and no code path could read the container-credential ones. That also rules out the ECS-style `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` bridge, which would otherwise have been an alternative workaround.

## Reproducing without a cluster

The provider selection is visible from a plain container, no EKS required.

Pod Identity shape (what an EKS pod gets). Note it reports the instance provider:

``` docker run --rm \ -e AWS_REGION=us-east-1 \ -e AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials \ -e AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/token \ ghcr.io/denoland/celld:0.3.0 deploy /app/bundle --bucket s3://<bucket> # INFO object_store::aws::builder: Using Instance credential provider ```

IRSA shape, for contrast. This one is picked up:

``` docker run --rm \ -e AWS_REGION=us-east-1 \ -e AWS_ROLE_ARN=arn:aws:iam::<account>:role/<role> \ -e AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token \ -v /path/to/token:/var/run/secrets/eks.amazonaws.com/serviceaccount:ro \ ghcr.io/denoland/celld:0.3.0 deploy /app/bundle --bucket s3://<bucket> # INFO object_store::aws::builder: Using WebIdentity credential provider # ... then a real sts:AssumeRoleWithWebIdentity call ```

## What would help

1. Support `AWS_CONTAINER_CREDENTIALS_FULL_URI` with `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`, which is what EKS Pod Identity injects. In the Rust ecosystem `object_store`'s `AmazonS3Builder` picks this up when the upstream crate's container-credentials support is enabled, so this may be a dependency version or feature-flag change rather than new code. 2. Failing that, a line in the docs saying Pod Identity is unsupported and IRSA is the mechanism to use on EKS. The current failure surfaces as a 30 second IMDS timeout and an "inaccessible bucket" error, which sends you looking at networking and IAM policy instead of at the credential source.

Happy to test a build.

  1. ryassigned nathanwhit3 months ago on GitHub

  2. nathanwhitclosed3 months ago on GitHub

  3. HM

    @nathanwhit I'm hitting the same thing (also on `0.3.0`) - is a fix already planned/completed?

    If not, happy to send a small patch. But don't want to waste your time. Thanks!

  4. nathanwhitmentioned2 months ago on GitHub

  5. nathanwhitsubscribed2 months ago on GitHub

  6. BA

    It's already fixed and will be included in the release

  7. bartlomiejureferenced 554fa82872e62 months ago on GitHub

  8. bartlomiejureferenced a52f9905425b2 months ago on GitHub