Hi Austen, Nice discussion! Thanks for that, and welcome aboard :)
I think it makes sense to have R2 built under the existing S3 storage configuration. It matches how Polaris is already structured: StorageType is keyed on the URI scheme (s3://, s3a://), and AwsStorageConfigurationInfo already carries endpoint, stsEndpoint, pathStyleAccess and, importantly, an stsUnavailable flag. So, what you need is largely there already: correct me if I'm wrong, R2 is essentially "S3 compatible store where STS is unavailable, plus a way to actually vend credentials in that case". Going this route also keeps this work "isolated" from the FileIO-per-storage-type discussion. The interesting design question is a pluggable credential vending path for S3 compatible stores when STS is not available, with Cloudflare's local signing as the first concrete implementation, rather than R2 specific code. It would be good to sketch how MinIO would slot into the same config fields before the vending path hardens around R2. I believe we have to check if Cloudflare account ID and jurisdiction really need dedicated config fields, or can we use the endpoint value (with documentation)? Keeping the configuration surface minimal will help the design generalize. I would love to discuss two things: 1. Polaris shipping a Cloudflare API token and locally minting scoped credentials makes it effectively an STS for R2, which increases the blast radius of a server compromise. Your "reject rather than boarden" handling of unrepresentable scopes is the right call. 2. The PyIceberg/DuckDB/Java results are promising. Just curious about Spark or Trino. We love small, well-scoped PRs. I would suggest to start with the S3 config and the management API as first shoot, and we can follow with the credential vending implementation. Regards JB On Wed, Sep 9, 2026 at 8:18 PM Austen Tomek <[email protected]> wrote: > > Hi all, > > I’m Austen Tomek from Chicago Trading Company. We use Polaris for our Iceberg > catalogs and we're evaluating Cloudflare R2 as an object store (the no egress > fees make it a very compelling product). I’d like to discuss contributing R2 > support and credential vending to Polaris with the goal of getting feedback > before opening a ready-for-review upstream PR. This is my first contribution, > so trying to do this right. > > R2 exposes S3-compatible APIs but it does not provide AWS STS. Cloudflare > instead supports these short-lived, scoped credentials generated by locally > signed JWT with the server holding the parent API token [1]. This allows us > to retain Polaris’s authorization and credential-vending model while storing > our table data in R2. > > I have a prototype with the following design: > > > * > An opt-in R2 storage type, with catalog configuration identifying the > Cloudflare account and optional jurisdiction. Parent credentials remain in > server configuration and can be selected through storageName. > * > Local credential generation using Cloudflare’s documented format. Credentials > expire, are restricted to one bucket, and carry prefix and access scopes > derived from Polaris’s location grants. The integration uses Polaris’s > existing credential cache. > * > Credentials returned through the existing Iceberg REST vending flow as s3.* > properties, including the endpoint, session token, and expiry. Compatible > clients use their existing S3 FileIO. > * > Conservative handling of scopes the implementation cannot represent: > cross-bucket and mixed read/write grants are rejected rather than broadened. > R2 is excluded from the default supported-storage-types list. > > The main design questions I feel this runs into: > > * > Should this be a distinct storage type or a credential-vending mechanism > selected through the existing S3 configuration? > * > I chose a separate type because the identity, credential generation, and > scoping model differ from AWS STS, as this seemed more appropriate > > There is a draft preview PR against my fork [2], including implementation, > management API changes, CLI support, tests, and documentation. It also > discloses the AI assistance used during implementation. I leaned heavily on > Fable 5.1 for this implementation so I want to be upfront about it. > > We are running the implementation in a non-production environment right now > as a POC. Validation includes live R2 reads/writes, multipart operations, > negative scope-boundary tests, our internal Python client regression matrix, > and credential rotation/recovery exercises. The upstream CI workflow has > passed on the fork. > > The change also encounters the existing assumption that a FileIO > implementation identifies exactly one storage type: both S3 and R2 use > S3FileIO. This overlaps with #4486 [3]. > > I’d particularly appreciate feedback on: > > > 1. > Should R2 have its own storage configuration/type, or should non-STS vending > be modeled within the existing S3 integration? > 2. > Does the local-signing approach fit Polaris’s credential-vending model? Any > additional constraints or validations necessary? > 3. > Would you prefer the shared FileIO validation change to land separately, and > how should we sequence the management API and R2 integration changes? > > Happy to adapt the design and split the contribution into more focused PRs. > Wanted to start the discussion > > Thanks, > Austen > > [1] https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > [2] https://github.com/deepdishgary/polaris/pull/1 > [3] https://github.com/apache/polaris/issues/4486 > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > This electronic mail message and any attached files contain information > intended for the exclusive use of the individual or entity to whom it is > addressed and may contain information that is proprietary, confidential > and/or exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any viewing, copying, > disclosure or distribution of this information may be subject to legal > restriction or sanction. Please notify the sender, by electronic mail or > telephone, of any unintended recipients and delete the original message > without making any copies.
