I don't think we have consensus on the PR, esp. a client should NOT depend on a specific deployment. Using an enum will clarify the REST spec contract.
Yufei On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > PR [5513] has been in review for quite some time. I believe we have general > consensus on the Management API changes and java code changes. > > The only point that may still have divergent opinions is whether to use > free text or enum for "credentialVendingMechanism". I personally support > free text (rationale explained in previous comments). This conversation has > been dormant for a few days, so I assume lazy consensus on the current > state of the PR (free text). In any case switching from free text to enum > is a non-breaking API change before the next release in case the consensus > shifts and we need to adjust. We have about one month for followup API > changes. > > The java code introduced in this PR is not rigid and can change at any time > per our evolution guidelines [1]. > > I propose merging PR [5513] on Sept 30 unless fresh blocking concerns are > raised. > > [1] https://polaris.apache.org/releases/1.8.0/evolution/ > > [5513] https://github.com/apache/polaris/pull/5513 > > Cheers, > Dmitri. > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek < > [email protected]> wrote: > > > Hey all, > > > > Any chance I can get further reviews this week on the PR? I’d like to get > > moving onto Phase 2 here if there are no further issues to address. > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > From: Dmitri Bourlatchkov <[email protected]> > > Date: Thursday, September 24, 2026 at 17:33 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > vending > > > > Hi Yufei, > > > > > The REST spec is a contract between Polaris servers and clients [...] > > > > I tend to think that this statement is a bit too strong. The Management > API > > spec defines a protocol for clients and servers to communicate. > > > > > [...] This would help avoid a situation where a Polaris client works > > > only with certain deployments. > > > > Since Polaris has been developed as a customizable system, a REST API > > specification defined by Apache Polaris cannot control the behaviour of > all > > possible downstream environments. The purpose of the spec is to ensure > > protocol compatibility. > > > > Note that the value of `credentialVendingMechanism` has no bearing on > > any logic clients need to execute. It is an opaque parameter controlled > > by the Polaris administrator. > > > > > The new credentialVendingMechanism field in the REST spec looks > > > reasonable. My only suggestion is to use an enum instead of a string. > > [...] > > > > The above points are exactly the reason why the Management API cannot > > enumerate all possible vending mechanisms. > > > > The same applies to storage types too. However, we do not have any > > practical > > examples of 3rd party storage systems that go beyond S3/GCS/ADLS (yet). > > > > Cheers, > > Dmitri. > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu <[email protected]> wrote: > > > > > Thanks Austen. It's fair that endpoints are not reliable given that > some > > > s3-compatible can use arbitrary hostnames. > > > I also think stsUnavailable is fine to keep as-is. I don’t think we > need > > to > > > change its scope or repurpose it. > > > > > > The new credentialVendingMechanism field in the REST spec looks > > reasonable. > > > My only suggestion is to use an enum instead of a string. The REST spec > > is > > > a contract between Polaris servers and clients, so I’d expect it to > > define > > > the valid values and their meanings rather than leaving them up to each > > > deployment. This would help avoid a situation where a Polaris client > > works > > > only with certain deployments. Deployment-specific mechanisms shouldn’t > > be > > > part of the standard REST contract; users who need a custom mechanism > can > > > always extend the server in their own deployment. > > > > > > > Either way the operator needs a per-realm switch like > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control which realm can > > mint > > > R2 credentials from the operators Cloudflare account. > > > > > > Could you clarify why we need a separate per-realm switch? Once the > > storage > > > configuration is set at the catalog level to select a specific > mechanism, > > > the catalog will have no choice but to support it. Do we need another > > layer > > > of gating? > > > > > > Yufei > > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek < > > > [email protected]> wrote: > > > > > > > Hey Yufei, > > > > > > > > Apologies for missing your question. > > > > > > > > I don’t think Endpoint is sufficient. > > > > > > > > Endpoint matching only works where the hostname identifies the vendor > > > like > > > > R2’s public hostnames that are recognizable, MinIO and other > > self-hosted > > > > S3-compatible stores use arbitrary hostnames so there would be > nothing > > to > > > > match on. On top of that, if a match fails it would just silently > fall > > > back > > > > to STS. > > > > > > > > Right now the endpoint just says how you’ll connect not how the > > > > credentials are vended. There's endpoint and also endpointInternal > for > > > > server-side access, so it would be unclear which of the two decides. > > > > > > > > For your other question, there is no second mechanism for R2 today, > but > > > > for other self-hosted stores the same endpoint could be served by STS > > or > > > by > > > > a different token mechanism chosen per catalog. > > > > > > > > stsUnavailable keeps it’s current behavior in the PR (that was > > Prithvi’s > > > > point earlier in the thread) > > > > > > > > Either way the operator needs a per-realm switch like > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control which realm can > > > mint > > > > R2 credentials from the operators Cloudflare account. > > > > > > > > Thanks, > > > > Austen > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > From: Yufei Gu <[email protected]> > > > > Date: Monday, September 21, 2026 at 17:53 > > > > To: [email protected] <[email protected]> > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > > > vending > > > > > > > > Hi Austen, > > > > > > > > Thanks for the update. After looking at PR #5513, I think we should > > first > > > > clarify whether we need a new credentialVendingMechanism field in the > > > > spec/config. > > > > > > > > Austen, you mentioned that the R2 account ID and jurisdiction already > > > come > > > > from the endpoint. Could we also use the existing field like > > `endpoint`, > > > to > > > > recognize R2 and select its credential-vending mechanism, while > keeping > > > the > > > > current stsUnavailable behavior unchanged? > > > > > > > > Are there cases where the endpoint isn’t enough, or where users need > to > > > > choose different vending mechanisms for the same endpoint(e.g., R2) > per > > > > catalog? > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek < > > > > [email protected]> wrote: > > > > > > > > > Hi Yufei, Dimitri, > > > > > > > > > > Thanks, both. PR 1 is ready for review: > > > > > https://github.com/apache/polaris/pull/5513 > > > > > > > > > > I pushed a revision last night that takes in Dimitri’s comments. > > > > > > > > > > > > > > > * > > > > > I changed from issuer to credentialVendingMechanism. It’s also now > a > > > > plain > > > > > string and the realm allowlist is > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which still defaults to > > > [STS] > > > > > * > > > > > Mechanisms are CDI beans found by their identifier, as Dimitri > > > proposed. > > > > > * > > > > > The CLOUDFLARE_R2 bean and the R2 vending code will follow in the > > next > > > PR > > > > > > > > > > > > > > > CI workflow is waiting approval to run. > > > > > > > > > > Thanks, > > > > > Austen > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > Date: Tuesday, September 15, 2026 at 20:00 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > > > > vending > > > > > > > > > > Hi All, > > > > > > > > > > > Having getCloudflareR2Endpoint on AwsStorageConfigurationInfo > > > > > > > > > > Yes, this does feel awkward. > > > > > > > > > > From my POV pursuing the option with the "credentialIssuers" > property > > > > looks > > > > > promising. This property is fairly generic and fits naturally into > > the > > > > > config, given that we already target multiple backends. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu <[email protected]> > > wrote: > > > > > > > > > > > Hi Austen, Dmitri, > > > > > > > > > > > > > Having getCloudflareR2Endpoint on AwsStorageConfigurationInfo > > > > > > > > > > > > That does feel awkward. However, AwsStorageConfigurationInfo has > > > > > supported > > > > > > S3-compatible backends beyond AWS for a while, so there is > > precedent > > > > > (e.g., > > > > > > MinIO, etc.). The underlying mismatch is that we call the storage > > > type > > > > > S3, > > > > > > but its configuration mixes shared S3 settings with AWS-specific > > > ones: > > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: S3 > > > > > > ├── Common S3 settings > > > > > > │ endpoint, region, pathStyleAccess, allowedLocations > > > > > > └── AWS-specific settings > > > > > > roleArn, externalId, STS, KMS > > > > > > > > > > > > I’m open to separating the shared S3 settings from > > provider-specific > > > > > > configuration if that helps, but I don’t think we need to settle > > on a > > > > > > broader refactor now. > > > > > > > > > > > > Let me know when the PR is ready for review. I’d be happy to > take a > > > > > closer > > > > > > look. > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > +1 to adding "credentialIssues" or similar (renamed) property > to > > S3 > > > > > > config. > > > > > > > In fact, I was looking exactly for something like that last > week > > > but > > > > > the > > > > > > > idea did not quite condense in my mind :) > > > > > > > > > > > > > > I posted some preliminary comments on the draft PR. > > > > > > > > > > > > > > Just for community awareness, I'd like to propose here that we > > use > > > > CDI > > > > > > for > > > > > > > finding actual "issuer" / "mechanism" implementations. The > > codebase > > > > has > > > > > > > many examples of this. The closest match in this case is > > > > > > > probably PolarisStorageIntegrationProvider. > > > > > > > > > > > > > > This way the API does not have to enumerate all possible > values. > > > > > > Downstream > > > > > > > builds are free to add different implementations without having > > to > > > > > change > > > > > > > the API spec. The "allowed" list still makes sense. > > > > > > > > > > > > > > In general, I like the direction this feature is taking, > > although I > > > > > > expect > > > > > > > PR reviews may take a few rounds :) > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > Closing the loop since my last note and answering the > questions > > > > that > > > > > > have > > > > > > > > come up. > > > > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t have AWS-style IAM > > > roles > > > > > or > > > > > > > > ARNs. Our implementation signs temporary credentials with a > > > parent > > > > > > > token’s > > > > > > > > secret, scoped to a bucket and a set of prefixes. The account > > ID > > > > and > > > > > > > > jurisdiction come from the R2 endpoint. Putting the account > ID > > in > > > > > > roleArn > > > > > > > > or userArn would give those fields a different meaning, so > I’ve > > > > left > > > > > > them > > > > > > > > absent for CLOUDFLARE_R2. > > > > > > > > > > > > > > > > I pursued Option B. The S3 configuration gains one new field, > > > > > > > > credentialIssuer, with values STS > > > > > > > > (default) and CLOUDFLARE_R2 and a realm allowlist > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults > > > > > > > > to [STS]. I left stsUnavailable untouched (Prithvi) so it > still > > > > means > > > > > > > > "vend nothing". The > > > > > > > > credentialIssuer is the explicit sign. The R2 config rejects > > > > > > > > stsUnavailable along with a host of other STS and KMS fields. > > I'm > > > > > > running > > > > > > > > this in our non-production environment at the moment. > > > > > > > > > > > > > > > > Draft PR for stage 1 (config field, API, allowlist, gates): > > > > > > > > https://github.com/apache/polaris/pull/5513. > > > > > > > > > > > > > > > > I still see the appeal of a separate R2 configuration, as > > Dmitri > > > > > > > > suggested. My original reason for separating it was to keep > the > > > > > > > R2-specific > > > > > > > > configuration and validation together. Having > > > > getCloudflareR2Endpoint > > > > > > on > > > > > > > > AwsStorageConfigurationInfo is one part I’d still like > feedback > > > on. > > > > > > > > > > > > > > > > I wanted to put a working version of Option B in front of > > > everyone > > > > so > > > > > > we > > > > > > > > could discuss the actual changes. In my earlier > implementation, > > > the > > > > > > > > separate config also meant a separate storage type sharing > > > > S3FileIO. > > > > > > That > > > > > > > > was the tradeoff that led me to try Option B. > > > > > > > > > > > > > > > > Do the diffs change anyone’s lean here? And are the names > > > > > > > credentialIssuer > > > > > > > > / STS / CLOUDFLARE_R2 / SUPPORTED_S3_CREDENTIAL_ISSUERS > > > acceptable? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > Date: Sunday, September 13, 2026 at 19:50 > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped > > > credential > > > > > > > vending > > > > > > > > > > > > > > > > Building on top of Ayush's and JB's suggestion, I agree that > > > > > > jurisdiction > > > > > > > > can live inside the existing endpoint config field. > > > > > > > > > > > > > > > > That leaves only Account ID, so potentially we could store it > > in > > > > user > > > > > > or > > > > > > > > role ARN. Note that userArn is currently unused in OSS code. > > > > > > > > > > > > > > > > That should allow us to keep existing > > AwsStorageConfigurationInfo > > > > for > > > > > > all > > > > > > > > S3 storage backend, but will require special parsing of > > user/role > > > > ARN > > > > > > > field > > > > > > > > to distinguish R2 from the other systems. > > > > > > > > > > > > > > > > I'm still leaning towards the separate > > > R2StorageConfigurationInfo, > > > > > but > > > > > > > I'd > > > > > > > > be ok with overloading user/role ARN too if other people > prefer > > > > > that. I > > > > > > > > guess a key question in this context is whether R2 has the > > > concept > > > > of > > > > > > > role > > > > > > > > ARN at all.... but TBH I have no idea about that. I hope > Austen > > > > might > > > > > > be > > > > > > > > able to clarify that. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > Welcome, and thanks for starting with a [DISCUSS]. Best way > > to > > > > > make a > > > > > > > > first > > > > > > > > > contribution :) > > > > > > > > > > > > > > > > > > I think the thread is talking about two layers, and we can > > keep > > > > > both > > > > > > > > > answers. > > > > > > > > > > > > > > > > > > On the client side this should stay S3: s3:// locations, > s3.* > > > > > > > properties, > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB / Iceberg Java results > > > already > > > > > show > > > > > > > > that > > > > > > > > > unmodified clients work, so a dedicated R2FileIO does not > > seem > > > > > worth > > > > > > > it. > > > > > > > > > On the server side I lean toward Dmitri: a distinct > > > > > > > > > R2StorageConfigurationInfo and storage integration. > Issuance > > is > > > > not > > > > > > > STS, > > > > > > > > > and folding Cloudflare fields into > > AwsStorageConfigurationInfo > > > > > mixes > > > > > > > > > unrelated concepts. The middle ground to avoid is a new > type > > > that > > > > > is > > > > > > > > still > > > > > > > > > AWS-shaped underneath. > > > > > > > > > > > > > > > > > > One concrete caution if the Option B prototype stays on the > > S3 > > > > > > config: > > > > > > > > > please do not overload stsUnavailable. Today that flag > skips > > > > > > AssumeRole > > > > > > > > and > > > > > > > > > disables vending. R2 wants the opposite. vend, just not via > > > STS. > > > > so > > > > > > > that > > > > > > > > > flag would change meaning for existing MinIO / NetApp > > catalogs. > > > > If > > > > > we > > > > > > > > stay > > > > > > > > > on S3 config, we need an explicit vending signal; a > dedicated > > > R2 > > > > > type > > > > > > > > > avoids that trap. > > > > > > > > > > > > > > > > > > On config surface, jurisdiction can live in endpoint > > > (<account>. > > > > > > > > > r2.cloudflarestorage.com vs <account>.eu.r2...). Account > ID > > is > > > > > still > > > > > > > > > needed > > > > > > > > > for the JWT sub, so keeping an explicit accountId and > > deriving > > > > > > audience > > > > > > > > > from the endpoint seems enough. Parent tokens should stay > in > > > > server > > > > > > > > config > > > > > > > > > via storageName. > > > > > > > > > Local signing fits the vending model. Fail-closed on > > > cross-bucket > > > > > and > > > > > > > > mixed > > > > > > > > > read/write grants is the right call. Prefix claims should > > > follow > > > > > the > > > > > > > same > > > > > > > > > path comparison Polaris already uses, so we do not mint a > > > > > credential > > > > > > > > wider > > > > > > > > > than the grant. > > > > > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator (#4486) separately if > > R2 > > > > > needs > > > > > > > it, > > > > > > > > > then management API + Java config, then vending, Python CLI > > on > > > > its > > > > > > own. > > > > > > > > > Sketching MinIO / Backblaze against the same fields is a > > useful > > > > > > design > > > > > > > > > check; a generic non-STS SPI can wait for a second backend. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri Bourlatchkov < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > Thanks for starting this contribution! > > > > > > > > > > > > > > > > > > > > After briefly looking through your PR [1] I think > > > > > > > > > > adding R2StorageConfigInfo makes sense. > > > > > > > > > > > > > > > > > > > > Even though "aws" in AwsStorageConfigurationInfo is > longer > > > > > implies > > > > > > > AWS > > > > > > > > > > services, and MinIO, Rust, Ozone are supported just as > > well, > > > R2 > > > > > > > appears > > > > > > > > > to > > > > > > > > > > be significantly different from STS-based S3 systems > that a > > > > > > separate > > > > > > > > > config > > > > > > > > > > is probably the most natural way to support it in the > > Polaris > > > > > > > codebase. > > > > > > > > > > > > > > > > > > > > The alternative, I guess, is to add R2 accountId and > > > > jurisdiction > > > > > > as > > > > > > > > > > optional properties to AwsStorageConfigurationInfo (or > > > overload > > > > > > > > existing > > > > > > > > > > properties), but that looks like mixing unrelated > concepts > > > > > > together. > > > > > > > > > > > > > > > > > > > > Adding a generic property bag to > > AwsStorageConfigurationInfo > > > is > > > > > not > > > > > > > > > > convenient because PolarisStorageIntegrationProviderImpl > > will > > > > > > > probably > > > > > > > > > have > > > > > > > > > > to know how to interpret it in order to redirect > credential > > > > > vending > > > > > > > > > > requests to R2-specific code. > > > > > > > > > > > > > > > > > > > > I wonder whether R2StorageConfigurationInfo and > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > (and corresponding "storage integration" classes) could > > > share a > > > > > > > common > > > > > > > > > base > > > > > > > > > > class for common properties (e.g. endpoint). Even though > > > > > "endpoint" > > > > > > > > might > > > > > > > > > > not be relevant to R2 in the cloud, the fact that > S3FileIO > > > uses > > > > > it, > > > > > > > and > > > > > > > > > R2 > > > > > > > > > > is accessed via S3FileIO means we should probably allow > > users > > > > to > > > > > > > > > configure > > > > > > > > > > the full set of FileIO properties just in case. > > > > > > > > > > > > > > > > > > > > I think it is fine for different > PolarisStorageIntegration > > > > > > > > > implementations > > > > > > > > > > to produce the same kind of StorageAccessConfig (S3 in > this > > > > case) > > > > > > and > > > > > > > > > thus > > > > > > > > > > cause S3FileIO to be used by clients (including Polaris' > > own > > > > > > storage > > > > > > > > > access > > > > > > > > > > paths). > > > > > > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] look a bit concerning > > to > > > > me. > > > > > I > > > > > > > > wonder > > > > > > > > > > if this code could be refactored to avoid dependencies on > > > > storage > > > > > > > > config > > > > > > > > > > types completely.... but TBH, I did not look too deeply > > into > > > > this > > > > > > > > today. > > > > > > > > > > > > > > > > > > > > Anticipating future PRs for this in the Polaris repo, I'd > > > like > > > > to > > > > > > ask > > > > > > > > to > > > > > > > > > > separate Python code changes from java changes since > these > > > > areas > > > > > > > > usually > > > > > > > > > > attract different reviewers :) > > > > > > > > > > > > > > > > > > > > From my POV, management API and matching java changes can > > be > > > in > > > > > the > > > > > > > > same > > > > > > > > > > PR. I'd say docs should come later (to keep PRs small). > > > > > > > > > > > > > > > > > > > > All in all, thank you again and I hope this contribution > > > lands > > > > in > > > > > > > > Polaris > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > [1] https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM Jean-Baptiste Onofré < > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and welcome aboard :) > > > > > > > > > > > > > > > > > > > > > > I think it makes sense to have R2 built under the > > existing > > > S3 > > > > > > > storage > > > > > > > > > > > configuration. It matches how Polaris is already > > > structured: > > > > > > > > > > > StorageType is keyed on the URI scheme (s3://, s3a://), > > and > > > > > > > > > > > AwsStorageConfigurationInfo already carries endpoint, > > > > > > stsEndpoint, > > > > > > > > > > > pathStyleAccess and, importantly, an stsUnavailable > flag. > > > > > > > > > > > So, what you need is largely there already: correct me > if > > > I'm > > > > > > > wrong, > > > > > > > > > > > R2 is essentially "S3 compatible store where STS is > > > > > unavailable, > > > > > > > plus > > > > > > > > > > > a way to actually vend credentials in that case". Going > > > this > > > > > > route > > > > > > > > > > > also keeps this work "isolated" from the > > > > > FileIO-per-storage-type > > > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > The interesting design question is a pluggable > credential > > > > > vending > > > > > > > > path > > > > > > > > > > > for S3 compatible stores when STS is not available, > with > > > > > > > Cloudflare's > > > > > > > > > > > local signing as the first concrete implementation, > > rather > > > > than > > > > > > R2 > > > > > > > > > > > specific code. It would be good to sketch how MinIO > would > > > > slot > > > > > > into > > > > > > > > > > > the same config fields before the vending path hardens > > > around > > > > > R2. > > > > > > > > > > > > > > > > > > > > > > I believe we have to check if Cloudflare account ID and > > > > > > > jurisdiction > > > > > > > > > > > really need dedicated config fields, or can we use the > > > > endpoint > > > > > > > value > > > > > > > > > > > (with documentation)? Keeping the configuration surface > > > > minimal > > > > > > > will > > > > > > > > > > > help the design generalize. > > > > > > > > > > > > > > > > > > > > > > I would love to discuss two things: > > > > > > > > > > > 1. Polaris shipping a Cloudflare API token and locally > > > > minting > > > > > > > scoped > > > > > > > > > > > credentials makes it effectively an STS for R2, which > > > > increases > > > > > > the > > > > > > > > > > > blast radius of a server compromise. Your "reject > rather > > > than > > > > > > > > boarden" > > > > > > > > > > > handling of unrepresentable scopes is the right call. > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java results are promising. > Just > > > > > curious > > > > > > > > about > > > > > > > > > > > Spark or Trino. > > > > > > > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I would suggest to > start > > > with > > > > > the > > > > > > > S3 > > > > > > > > > > > config and the management API as first shoot, and we > can > > > > follow > > > > > > > with > > > > > > > > > > > the credential vending implementation. > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen Tomek > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago Trading Company. We use > > > > Polaris > > > > > > for > > > > > > > > our > > > > > > > > > > > Iceberg catalogs and we're evaluating Cloudflare R2 as > an > > > > > object > > > > > > > > store > > > > > > > > > > (the > > > > > > > > > > > no egress fees make it a very compelling product). I’d > > like > > > > to > > > > > > > > discuss > > > > > > > > > > > contributing R2 support and credential vending to > Polaris > > > > with > > > > > > the > > > > > > > > goal > > > > > > > > > > of > > > > > > > > > > > getting feedback before opening a ready-for-review > > upstream > > > > PR. > > > > > > > This > > > > > > > > is > > > > > > > > > > my > > > > > > > > > > > first contribution, so trying to do this right. > > > > > > > > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it does not provide > > AWS > > > > > STS. > > > > > > > > > > > Cloudflare instead supports these short-lived, scoped > > > > > credentials > > > > > > > > > > generated > > > > > > > > > > > by locally signed JWT with the server holding the > parent > > > API > > > > > > token > > > > > > > > [1]. > > > > > > > > > > > This allows us to retain Polaris’s authorization and > > > > > > > > credential-vending > > > > > > > > > > > model while storing our table data in R2. > > > > > > > > > > > > > > > > > > > > > > > > I have a prototype with the following design: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > An opt-in R2 storage type, with catalog configuration > > > > > > identifying > > > > > > > > the > > > > > > > > > > > Cloudflare account and optional jurisdiction. Parent > > > > > credentials > > > > > > > > remain > > > > > > > > > > in > > > > > > > > > > > server configuration and can be selected through > > > storageName. > > > > > > > > > > > > * > > > > > > > > > > > > Local credential generation using Cloudflare’s > > documented > > > > > > format. > > > > > > > > > > > Credentials expire, are restricted to one bucket, and > > carry > > > > > > prefix > > > > > > > > and > > > > > > > > > > > access scopes derived from Polaris’s location grants. > The > > > > > > > integration > > > > > > > > > > uses > > > > > > > > > > > Polaris’s existing credential cache. > > > > > > > > > > > > * > > > > > > > > > > > > Credentials returned through the existing Iceberg > REST > > > > > vending > > > > > > > flow > > > > > > > > > as > > > > > > > > > > > s3.* properties, including the endpoint, session token, > > and > > > > > > expiry. > > > > > > > > > > > Compatible clients use their existing S3 FileIO. > > > > > > > > > > > > * > > > > > > > > > > > > Conservative handling of scopes the implementation > > cannot > > > > > > > > represent: > > > > > > > > > > > cross-bucket and mixed read/write grants are rejected > > > rather > > > > > than > > > > > > > > > > > broadened. R2 is excluded from the default > > > > > > supported-storage-types > > > > > > > > > list. > > > > > > > > > > > > > > > > > > > > > > > > The main design questions I feel this runs into: > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > Should this be a distinct storage type or a > > > > > credential-vending > > > > > > > > > > mechanism > > > > > > > > > > > selected through the existing S3 configuration? > > > > > > > > > > > > * > > > > > > > > > > > > I chose a separate type because the identity, > > credential > > > > > > > > generation, > > > > > > > > > > and > > > > > > > > > > > scoping model differ from AWS STS, as this seemed more > > > > > > appropriate > > > > > > > > > > > > > > > > > > > > > > > > There is a draft preview PR against my fork [2], > > > including > > > > > > > > > > > implementation, management API changes, CLI support, > > tests, > > > > and > > > > > > > > > > > documentation. It also discloses the AI assistance used > > > > during > > > > > > > > > > > implementation. I leaned heavily on Fable 5.1 for this > > > > > > > implementation > > > > > > > > > so > > > > > > > > > > I > > > > > > > > > > > want to be upfront about it. > > > > > > > > > > > > > > > > > > > > > > > > We are running the implementation in a non-production > > > > > > environment > > > > > > > > > right > > > > > > > > > > > now as a POC. Validation includes live R2 reads/writes, > > > > > multipart > > > > > > > > > > > operations, negative scope-boundary tests, our internal > > > > Python > > > > > > > client > > > > > > > > > > > regression matrix, and credential rotation/recovery > > > > exercises. > > > > > > The > > > > > > > > > > upstream > > > > > > > > > > > CI workflow has passed on the fork. > > > > > > > > > > > > > > > > > > > > > > > > The change also encounters the existing assumption > > that a > > > > > > FileIO > > > > > > > > > > > implementation identifies exactly one storage type: > both > > S3 > > > > and > > > > > > R2 > > > > > > > > use > > > > > > > > > > > S3FileIO. This overlaps with #4486 [3]. > > > > > > > > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback on: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > Should R2 have its own storage configuration/type, or > > > > should > > > > > > > > non-STS > > > > > > > > > > > vending be modeled within the existing S3 integration? > > > > > > > > > > > > 2. > > > > > > > > > > > > Does the local-signing approach fit Polaris’s > > > > > > credential-vending > > > > > > > > > model? > > > > > > > > > > > Any additional constraints or validations necessary? > > > > > > > > > > > > 3. > > > > > > > > > > > > Would you prefer the shared FileIO validation change > to > > > > land > > > > > > > > > > separately, > > > > > > > > > > > and how should we sequence the management API and R2 > > > > > integration > > > > > > > > > changes? > > > > > > > > > > > > > > > > > > > > > > > > Happy to adapt the design and split the > contribution > > > into > > > > > > more > > > > > > > > > > focused > > > > > > > > > > > PRs. Wanted to start the discussion > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > > > > > > > > > > > [2] https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > [3] https://github.com/apache/polaris/issues/4486 > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > > > contain > > > > > > > > > information > > > > > > > > > > > intended for the exclusive use of the individual or > > entity > > > to > > > > > > whom > > > > > > > it > > > > > > > > > is > > > > > > > > > > > addressed and may contain information that is > > proprietary, > > > > > > > > confidential > > > > > > > > > > > and/or exempt from disclosure under applicable law. If > > you > > > > are > > > > > > not > > > > > > > > the > > > > > > > > > > > intended recipient, you are hereby notified that any > > > viewing, > > > > > > > > copying, > > > > > > > > > > > disclosure or distribution of this information may be > > > subject > > > > > to > > > > > > > > legal > > > > > > > > > > > restriction or sanction. Please notify the sender, by > > > > > electronic > > > > > > > mail > > > > > > > > > or > > > > > > > > > > > telephone, of any unintended recipients and delete the > > > > original > > > > > > > > message > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > > > > information > > > > > > > > intended for the exclusive use of the individual or entity to > > > whom > > > > it > > > > > > is > > > > > > > > addressed and may contain information that is proprietary, > > > > > confidential > > > > > > > > and/or exempt from disclosure under applicable law. If you > are > > > not > > > > > the > > > > > > > > intended recipient, you are hereby notified that any viewing, > > > > > copying, > > > > > > > > disclosure or distribution of this information may be subject > > to > > > > > legal > > > > > > > > restriction or sanction. Please notify the sender, by > > electronic > > > > mail > > > > > > or > > > > > > > > telephone, of any unintended recipients and delete the > original > > > > > message > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > information > > > > > intended for the exclusive use of the individual or entity to whom > it > > > is > > > > > addressed and may contain information that is proprietary, > > confidential > > > > > and/or exempt from disclosure under applicable law. If you are not > > the > > > > > intended recipient, you are hereby notified that any viewing, > > copying, > > > > > disclosure or distribution of this information may be subject to > > legal > > > > > restriction or sanction. Please notify the sender, by electronic > mail > > > or > > > > > telephone, of any unintended recipients and delete the original > > message > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > information > > > > intended for the exclusive use of the individual or entity to whom it > > is > > > > addressed and may contain information that is proprietary, > confidential > > > > and/or exempt from disclosure under applicable law. If you are not > the > > > > intended recipient, you are hereby notified that any viewing, > copying, > > > > disclosure or distribution of this information may be subject to > legal > > > > restriction or sanction. Please notify the sender, by electronic mail > > or > > > > telephone, of any unintended recipients and delete the original > message > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > intended for the exclusive use of the individual or entity to whom it is > > addressed and may contain information that is proprietary, confidential > > and/or exempt from disclosure under applicable law. If you are not the > > intended recipient, you are hereby notified that any viewing, copying, > > disclosure or distribution of this information may be subject to legal > > restriction or sanction. Please notify the sender, by electronic mail or > > telephone, of any unintended recipients and delete the original message > > without making any copies. > > >
