- Issues
- #139
Expanding S3-Compatibility to other object stores (i.e. Azure Blob Storage)
tjrivera opened 路
Hi Ry, celld maintainers, etc.,
I'd like to propose Azure Blob Storage as a native storage backend for celld.
Unfortunately, Azure Blob Storage does not use the S3 API. Its close - but not quite the same. It therefore needs a separate backend, not an S3 endpoint configuration.
## Proposed scope
The Azure backend will:
- keep the current S3 behavior unchanged - use a provider-neutral object store interface for celld and LTX data - support range reads, listing, deletion, metadata, and large uploads - support the Azure credential chain (i.e CLI credentials, managed identity, workload identity, storage account keys, SAS tokens) -- current approach is focused on CLI creds + managed identities FWIW. - add opt-in integration tests against a real Azure storage account
## Important design concerns
A few storage details I uncovered need explicit handling or are worth mentioning:
- Azure custom metadata names cannot contain hyphens. The Litestream key `litestream-timestamp` needs a provider-safe form, such as `litestream_timestamp`. This could be handled by the adapter, but wanted to point it out. - celld relies on S3 conditional writes so that only one node can claim a database at a time. Azure Blob Storage provides equivalent conditional writes through a different API. The Azure adapter must preserve the same ownership guarantees and translate Azure conflicts into the result celld already expects. The implementation differs in shape, but not in behavior. - Azure handles large uploads with block blobs rather than S3 multipart uploads. The Azure adapter must preserve the same LTX upload, cleanup, metadata, and range-read behavior. Similar to the last bullet - different shape, same behavior.
The implementation keeps celld's ownership protocol, LTX format, and existing S3 behavior unchanged.
## Contribution
If this direction fits the project, I am happy to prepare and send a focused patch. I can adjust the scope or configuration based onfeedback.
- CR
Vibe-coded here: https://github.com/crloz/celld/tree/azure-blob-storage-backend I just wanted to see it working... and it does
dwilson6subscribed3 months ago on GitHub
- BA
We've added support for Azure and it will be released this week. Thanks for raising this
bartlomiejureferenced 9c0ca09ef5d53 months ago on GitHub
bartlomiejureferenced bc0e29cddf303 months ago on GitHub
bartlomiejureferenced 89e4ffc53a143 months ago on GitHub
fhdhsnireferenced this in R2 writes fail with Azure Blob Storage due to invalid metadata key #2092 months ago on GitHub