Just to confirm my understanding: "local signing" happens on the Polaris server, which generates scoped, temporary credentials without calling Cloudflare's credential API. Yes that’s correct. Polaris generates the temporary credentials without calling Cloudflare’s credential API. The client uses them to sign its S3 requests, and R2 checks the permissions, prefix scope, and expiry. Does this work out of the box with existing Iceberg REST Catalog clients that support credential vending and S3FileIO, or does it require a special client impl. or changes? No R2-specific client implementation, signer, or plugin was needed for basic reads and writes. Here’s what I’ve tested against a local patched Polaris backed by real R2 buckets:
* PyIceberg 0.11.1 and 0.12.0, with PyArrow 25.0.1: create, append, and read on 0.11.1; append and read on 0.12.0. * DuckDB 1.5.5, with the Iceberg and httpfs extensions: create, insert, and read. * Iceberg Java 1.11.0, with AWS SDK 2.44.7: reads through the unmodified RESTCatalog and S3FileIO. Separate tests through our Java client covered writes and multipart uploads, including the SDK’s CRC32 trailers. * Spark, Trino, and Flink: not tested yet. The clients consumed the normal Iceberg REST credential response, no separately configured R2 storage credentials were necessary. Option B: R2 as S3-compatible storage … If this is the case, I'd lean toward considering R2 an S3-compatible storage, which may not need a separate storage config type I agree that the current approach does seem to be straddling two concepts as Sushant pointed out. If we go with Option A, R2FileIO would basically be a wrapper around S3FileIO given the R2 data plane is the S3 API and nothing R2 specific happens on the client. I’m happy to explore R2 under the existing S3 storage configuration. I originally chose a separate type because credential issuance differs from AWS STS, but that difference seems clearly server side. My next step would be to prototype Option B on my fork: put local signing behind the S3 configuration, work through how to represent the account ID and jurisdiction, and rerun the client test matrix.. I’ll check that the configuration changes preserve existing S3 and STS behavior. Agreed on keeping the PRs small. Using the existing S3 type should remove R2’s need for the StorageTypeFileIO change in #4486. I expect we can separate the configuration/API changes from the credential-vending implementation, but I want to confirm that with the prototype before actually proposing the split. Does that sound OK to everyone? Thanks, Austen Get Outlook for Mac<https://aka.ms/GetOutlookForMac> From: Sushant Raikar via dev <[email protected]> Date: Wednesday, September 9, 2026 at 19:16 To: [email protected] <[email protected]> Cc: Sushant Raikar <[email protected]> Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential vending [You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Thanks for kicking this off, Austen. R2 interest is clearly picking up, so it's great to see this worked out in the open. One thing I noticed is that the design sits between two approaches, and I wonder whether committing to one direction is cleaner than blending them. *Option A:* R2 as a fully first-class storage type. Own config, own credential vending (the local JWT signing), and its own R2FileIO rather than reusing S3FileIO. My thinking is that a dedicated FileIO keeps the boundaries clean and sidesteps the "one FileIO maps to one storage type" question in #4486, since R2 wouldn't share S3FileIO at all. The cost is real though: a new FileIO probably means contributing R2FileIO upstream to Iceberg, not just changes in Polaris. Curious whether that upstream path seems reasonable to folks who've done it before. *Option B:* R2 as S3-compatible storage. If we go here, I think the main design question is defining the lowest common denominator of credential vending that the majority of s3.* contract can actually express, not just for R2 but across S3-compatible backends generally (MinIO, Ceph RGW, Blackblaze, Wasabi and so on). R2 would then fit inside that shared vending path and accept its limits, rather than each backend bending s3.* in its own direction. Would it make sense to scope that LCD first, before wiring up R2 specifically? What I'd personally want to avoid is the middle ground: a distinct type that still leans on S3FileIO and S3 assumptions underneath. That seems like it carries the maintenance cost of a new type without the clean separation that justifies it, though I might be underestimating how much the two can safely share. On process, +1 to Yufei on splitting into small PRs. And I think his framing is right: the local-signing path does look a lot like STS-style vending from the client's side, so much of this may come down to whether the config differences are big enough to warrant a separate type. Thanks again for putting this together. Happy to pitch in however's useful as it moves forward. On Wed, Sep 9, 2026 at 3:57 PM Yufei Gu <[email protected]> wrote: > Hi Austen, > > Just to confirm my understanding: “local signing” happens on the Polaris > server, which generates scoped, temporary credentials without calling > Cloudflare’s credential API. The client then uses those credentials to sign > its own S3 requests, and R2 enforces the scope and expiry. This sounds like > STS-style credential vending, where Polaris performs the issuance without > talking to an object storage server. If this is the case, I'd lean toward > considering R2 an S3-compatible storage, which may not need a separate > storage config type. However, we will still need to check all configuration > details to ensure no incompatible changes are introduced. > > Does this work out of the box with existing Iceberg REST Catalog clients > that support credential vending and S3FileIO, or does it require a special > client impl. or changes? You mentioned validation with your internal Python > client. Have you also tested unmodified OSS clients(e.g., Spark with > Iceberg plugins, Trino)? > > > Would you prefer the shared FileIO validation change to land separately, > and how should we sequence the management API and R2 integration changes? > > Yes, landing the changes separately would be preferable. Small and > well-scoped PRs usually merge faster. > > Yufei > > > On Wed, Sep 9, 2026 at 11:18 AM Austen Tomek < > [email protected]> wrote: > > > Hi all, > > > > I’m Austen Tomek from Chicago Trading Company. We use Polaris for our > > Iceberg catalogs and we're evaluating Cloudflare R2 as an object store > (the > > no egress fees make it a very compelling product). I’d like to discuss > > contributing R2 support and credential vending to Polaris with the goal > of > > getting feedback before opening a ready-for-review upstream PR. This is > my > > first contribution, so trying to do this right. > > > > R2 exposes S3-compatible APIs but it does not provide AWS STS. Cloudflare > > instead supports these short-lived, scoped credentials generated by > locally > > signed JWT with the server holding the parent API token [1]. This allows > us > > to retain Polaris’s authorization and credential-vending model while > > storing our table data in R2. > > > > I have a prototype with the following design: > > > > > > * > > An opt-in R2 storage type, with catalog configuration identifying the > > Cloudflare account and optional jurisdiction. Parent credentials remain > in > > server configuration and can be selected through storageName. > > * > > Local credential generation using Cloudflare’s documented format. > > Credentials expire, are restricted to one bucket, and carry prefix and > > access scopes derived from Polaris’s location grants. The integration > uses > > Polaris’s existing credential cache. > > * > > Credentials returned through the existing Iceberg REST vending flow as > > s3.* properties, including the endpoint, session token, and expiry. > > Compatible clients use their existing S3 FileIO. > > * > > Conservative handling of scopes the implementation cannot represent: > > cross-bucket and mixed read/write grants are rejected rather than > > broadened. R2 is excluded from the default supported-storage-types list. > > > > The main design questions I feel this runs into: > > > > * > > Should this be a distinct storage type or a credential-vending mechanism > > selected through the existing S3 configuration? > > * > > I chose a separate type because the identity, credential generation, and > > scoping model differ from AWS STS, as this seemed more appropriate > > > > There is a draft preview PR against my fork [2], including > implementation, > > management API changes, CLI support, tests, and documentation. It also > > discloses the AI assistance used during implementation. I leaned heavily > on > > Fable 5.1 for this implementation so I want to be upfront about it. > > > > We are running the implementation in a non-production environment right > > now as a POC. Validation includes live R2 reads/writes, multipart > > operations, negative scope-boundary tests, our internal Python client > > regression matrix, and credential rotation/recovery exercises. The > upstream > > CI workflow has passed on the fork. > > > > The change also encounters the existing assumption that a FileIO > > implementation identifies exactly one storage type: both S3 and R2 use > > S3FileIO. This overlaps with #4486 [3]. > > > > I’d particularly appreciate feedback on: > > > > > > 1. > > Should R2 have its own storage configuration/type, or should non-STS > > vending be modeled within the existing S3 integration? > > 2. > > Does the local-signing approach fit Polaris’s credential-vending model? > > Any additional constraints or validations necessary? > > 3. > > Would you prefer the shared FileIO validation change to land separately, > > and how should we sequence the management API and R2 integration changes? > > > > Happy to adapt the design and split the contribution into more focused > > PRs. Wanted to start the discussion > > > > Thanks, > > Austen > > > > [1] https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > [2] https://github.com/deepdishgary/polaris/pull/1 > > [3] https://github.com/apache/polaris/issues/4486 > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > intended for the exclusive use of the individual or entity to whom it is > > addressed and may contain information that is proprietary, confidential > > and/or exempt from disclosure under applicable law. If you are not the > > intended recipient, you are hereby notified that any viewing, copying, > > disclosure or distribution of this information may be subject to legal > > restriction or sanction. Please notify the sender, by electronic mail or > > telephone, of any unintended recipients and delete the original message > > without making any copies. > > > This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies.
