Thanks for kicking this off, Austen. R2 interest is clearly picking up, so it's great to see this worked out in the open.
One thing I noticed is that the design sits between two approaches, and I wonder whether committing to one direction is cleaner than blending them. *Option A:* R2 as a fully first-class storage type. Own config, own credential vending (the local JWT signing), and its own R2FileIO rather than reusing S3FileIO. My thinking is that a dedicated FileIO keeps the boundaries clean and sidesteps the "one FileIO maps to one storage type" question in #4486, since R2 wouldn't share S3FileIO at all. The cost is real though: a new FileIO probably means contributing R2FileIO upstream to Iceberg, not just changes in Polaris. Curious whether that upstream path seems reasonable to folks who've done it before. *Option B:* R2 as S3-compatible storage. If we go here, I think the main design question is defining the lowest common denominator of credential vending that the majority of s3.* contract can actually express, not just for R2 but across S3-compatible backends generally (MinIO, Ceph RGW, Blackblaze, Wasabi and so on). R2 would then fit inside that shared vending path and accept its limits, rather than each backend bending s3.* in its own direction. Would it make sense to scope that LCD first, before wiring up R2 specifically? What I'd personally want to avoid is the middle ground: a distinct type that still leans on S3FileIO and S3 assumptions underneath. That seems like it carries the maintenance cost of a new type without the clean separation that justifies it, though I might be underestimating how much the two can safely share. On process, +1 to Yufei on splitting into small PRs. And I think his framing is right: the local-signing path does look a lot like STS-style vending from the client's side, so much of this may come down to whether the config differences are big enough to warrant a separate type. Thanks again for putting this together. Happy to pitch in however's useful as it moves forward. Sushant On Wed, Sep 9, 2026 at 3:57 PM Yufei Gu <[email protected]> wrote: > Hi Austen, > > Just to confirm my understanding: “local signing” happens on the Polaris > server, which generates scoped, temporary credentials without calling > Cloudflare’s credential API. The client then uses those credentials to sign > its own S3 requests, and R2 enforces the scope and expiry. This sounds like > STS-style credential vending, where Polaris performs the issuance without > talking to an object storage server. If this is the case, I'd lean toward > considering R2 an S3-compatible storage, which may not need a separate > storage config type. However, we will still need to check all configuration > details to ensure no incompatible changes are introduced. > > Does this work out of the box with existing Iceberg REST Catalog clients > that support credential vending and S3FileIO, or does it require a special > client impl. or changes? You mentioned validation with your internal Python > client. Have you also tested unmodified OSS clients(e.g., Spark with > Iceberg plugins, Trino)? > > > Would you prefer the shared FileIO validation change to land separately, > and how should we sequence the management API and R2 integration changes? > > Yes, landing the changes separately would be preferable. Small and > well-scoped PRs usually merge faster. > > Yufei > > > On Wed, Sep 9, 2026 at 11:18 AM Austen Tomek < > [email protected]> wrote: > > > Hi all, > > > > I’m Austen Tomek from Chicago Trading Company. We use Polaris for our > > Iceberg catalogs and we're evaluating Cloudflare R2 as an object store > (the > > no egress fees make it a very compelling product). I’d like to discuss > > contributing R2 support and credential vending to Polaris with the goal > of > > getting feedback before opening a ready-for-review upstream PR. This is > my > > first contribution, so trying to do this right. > > > > R2 exposes S3-compatible APIs but it does not provide AWS STS. Cloudflare > > instead supports these short-lived, scoped credentials generated by > locally > > signed JWT with the server holding the parent API token [1]. This allows > us > > to retain Polaris’s authorization and credential-vending model while > > storing our table data in R2. > > > > I have a prototype with the following design: > > > > > > * > > An opt-in R2 storage type, with catalog configuration identifying the > > Cloudflare account and optional jurisdiction. Parent credentials remain > in > > server configuration and can be selected through storageName. > > * > > Local credential generation using Cloudflare’s documented format. > > Credentials expire, are restricted to one bucket, and carry prefix and > > access scopes derived from Polaris’s location grants. The integration > uses > > Polaris’s existing credential cache. > > * > > Credentials returned through the existing Iceberg REST vending flow as > > s3.* properties, including the endpoint, session token, and expiry. > > Compatible clients use their existing S3 FileIO. > > * > > Conservative handling of scopes the implementation cannot represent: > > cross-bucket and mixed read/write grants are rejected rather than > > broadened. R2 is excluded from the default supported-storage-types list. > > > > The main design questions I feel this runs into: > > > > * > > Should this be a distinct storage type or a credential-vending mechanism > > selected through the existing S3 configuration? > > * > > I chose a separate type because the identity, credential generation, and > > scoping model differ from AWS STS, as this seemed more appropriate > > > > There is a draft preview PR against my fork [2], including > implementation, > > management API changes, CLI support, tests, and documentation. It also > > discloses the AI assistance used during implementation. I leaned heavily > on > > Fable 5.1 for this implementation so I want to be upfront about it. > > > > We are running the implementation in a non-production environment right > > now as a POC. Validation includes live R2 reads/writes, multipart > > operations, negative scope-boundary tests, our internal Python client > > regression matrix, and credential rotation/recovery exercises. The > upstream > > CI workflow has passed on the fork. > > > > The change also encounters the existing assumption that a FileIO > > implementation identifies exactly one storage type: both S3 and R2 use > > S3FileIO. This overlaps with #4486 [3]. > > > > I’d particularly appreciate feedback on: > > > > > > 1. > > Should R2 have its own storage configuration/type, or should non-STS > > vending be modeled within the existing S3 integration? > > 2. > > Does the local-signing approach fit Polaris’s credential-vending model? > > Any additional constraints or validations necessary? > > 3. > > Would you prefer the shared FileIO validation change to land separately, > > and how should we sequence the management API and R2 integration changes? > > > > Happy to adapt the design and split the contribution into more focused > > PRs. Wanted to start the discussion > > > > Thanks, > > Austen > > > > [1] https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > [2] https://github.com/deepdishgary/polaris/pull/1 > > [3] https://github.com/apache/polaris/issues/4486 > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > intended for the exclusive use of the individual or entity to whom it is > > addressed and may contain information that is proprietary, confidential > > and/or exempt from disclosure under applicable law. If you are not the > > intended recipient, you are hereby notified that any viewing, copying, > > disclosure or distribution of this information may be subject to legal > > restriction or sanction. Please notify the sender, by electronic mail or > > telephone, of any unintended recipients and delete the original message > > without making any copies. > > >
