Hi Yufei, > Using an enum will clarify the REST spec contract.
I'm not sure I understand your point. Current PR clearly describes the meaning of "credentialVendingMechanism" and lists options available out-of-the-box. What does an enum achieve, that (current) runtime validation does not? Conversely, an enum will require defining all possible vending mechanisms in the Polaris API spec. This will prevent downstream projects from using custom vending mechanisms. I believe Polaris as a project currently promotes downstream customizability. Cheers, Dmitri. On Tue, Sep 29, 2026 at 1:21 PM Yufei Gu <[email protected]> wrote: > I don't think we have consensus on the PR, esp. a client should NOT depend > on a specific deployment. Using an enum will clarify the REST spec > contract. > > Yufei > > > On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi All, > > > > PR [5513] has been in review for quite some time. I believe we have > general > > consensus on the Management API changes and java code changes. > > > > The only point that may still have divergent opinions is whether to use > > free text or enum for "credentialVendingMechanism". I personally support > > free text (rationale explained in previous comments). This conversation > has > > been dormant for a few days, so I assume lazy consensus on the current > > state of the PR (free text). In any case switching from free text to enum > > is a non-breaking API change before the next release in case the > consensus > > shifts and we need to adjust. We have about one month for followup API > > changes. > > > > The java code introduced in this PR is not rigid and can change at any > time > > per our evolution guidelines [1]. > > > > I propose merging PR [5513] on Sept 30 unless fresh blocking concerns are > > raised. > > > > [1] https://polaris.apache.org/releases/1.8.0/evolution/ > > > > [5513] https://github.com/apache/polaris/pull/5513 > > > > Cheers, > > Dmitri. > > > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek < > > [email protected]> wrote: > > > > > Hey all, > > > > > > Any chance I can get further reviews this week on the PR? I’d like to > get > > > moving onto Phase 2 here if there are no further issues to address. > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > Date: Thursday, September 24, 2026 at 17:33 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > > vending > > > > > > Hi Yufei, > > > > > > > The REST spec is a contract between Polaris servers and clients [...] > > > > > > I tend to think that this statement is a bit too strong. The Management > > API > > > spec defines a protocol for clients and servers to communicate. > > > > > > > [...] This would help avoid a situation where a Polaris client works > > > > only with certain deployments. > > > > > > Since Polaris has been developed as a customizable system, a REST API > > > specification defined by Apache Polaris cannot control the behaviour of > > all > > > possible downstream environments. The purpose of the spec is to ensure > > > protocol compatibility. > > > > > > Note that the value of `credentialVendingMechanism` has no bearing on > > > any logic clients need to execute. It is an opaque parameter controlled > > > by the Polaris administrator. > > > > > > > The new credentialVendingMechanism field in the REST spec looks > > > > reasonable. My only suggestion is to use an enum instead of a string. > > > [...] > > > > > > The above points are exactly the reason why the Management API cannot > > > enumerate all possible vending mechanisms. > > > > > > The same applies to storage types too. However, we do not have any > > > practical > > > examples of 3rd party storage systems that go beyond S3/GCS/ADLS (yet). > > > > > > Cheers, > > > Dmitri. > > > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu <[email protected]> wrote: > > > > > > > Thanks Austen. It's fair that endpoints are not reliable given that > > some > > > > s3-compatible can use arbitrary hostnames. > > > > I also think stsUnavailable is fine to keep as-is. I don’t think we > > need > > > to > > > > change its scope or repurpose it. > > > > > > > > The new credentialVendingMechanism field in the REST spec looks > > > reasonable. > > > > My only suggestion is to use an enum instead of a string. The REST > spec > > > is > > > > a contract between Polaris servers and clients, so I’d expect it to > > > define > > > > the valid values and their meanings rather than leaving them up to > each > > > > deployment. This would help avoid a situation where a Polaris client > > > works > > > > only with certain deployments. Deployment-specific mechanisms > shouldn’t > > > be > > > > part of the standard REST contract; users who need a custom mechanism > > can > > > > always extend the server in their own deployment. > > > > > > > > > Either way the operator needs a per-realm switch like > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control which realm can > > > mint > > > > R2 credentials from the operators Cloudflare account. > > > > > > > > Could you clarify why we need a separate per-realm switch? Once the > > > storage > > > > configuration is set at the catalog level to select a specific > > mechanism, > > > > the catalog will have no choice but to support it. Do we need another > > > layer > > > > of gating? > > > > > > > > Yufei > > > > > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek < > > > > [email protected]> wrote: > > > > > > > > > Hey Yufei, > > > > > > > > > > Apologies for missing your question. > > > > > > > > > > I don’t think Endpoint is sufficient. > > > > > > > > > > Endpoint matching only works where the hostname identifies the > vendor > > > > like > > > > > R2’s public hostnames that are recognizable, MinIO and other > > > self-hosted > > > > > S3-compatible stores use arbitrary hostnames so there would be > > nothing > > > to > > > > > match on. On top of that, if a match fails it would just silently > > fall > > > > back > > > > > to STS. > > > > > > > > > > Right now the endpoint just says how you’ll connect not how the > > > > > credentials are vended. There's endpoint and also endpointInternal > > for > > > > > server-side access, so it would be unclear which of the two > decides. > > > > > > > > > > For your other question, there is no second mechanism for R2 today, > > but > > > > > for other self-hosted stores the same endpoint could be served by > STS > > > or > > > > by > > > > > a different token mechanism chosen per catalog. > > > > > > > > > > stsUnavailable keeps it’s current behavior in the PR (that was > > > Prithvi’s > > > > > point earlier in the thread) > > > > > > > > > > Either way the operator needs a per-realm switch like > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control which realm > can > > > > mint > > > > > R2 credentials from the operators Cloudflare account. > > > > > > > > > > Thanks, > > > > > Austen > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > From: Yufei Gu <[email protected]> > > > > > Date: Monday, September 21, 2026 at 17:53 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > > > > vending > > > > > > > > > > Hi Austen, > > > > > > > > > > Thanks for the update. After looking at PR #5513, I think we should > > > first > > > > > clarify whether we need a new credentialVendingMechanism field in > the > > > > > spec/config. > > > > > > > > > > Austen, you mentioned that the R2 account ID and jurisdiction > already > > > > come > > > > > from the endpoint. Could we also use the existing field like > > > `endpoint`, > > > > to > > > > > recognize R2 and select its credential-vending mechanism, while > > keeping > > > > the > > > > > current stsUnavailable behavior unchanged? > > > > > > > > > > Are there cases where the endpoint isn’t enough, or where users > need > > to > > > > > choose different vending mechanisms for the same endpoint(e.g., R2) > > per > > > > > catalog? > > > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek < > > > > > [email protected]> wrote: > > > > > > > > > > > Hi Yufei, Dimitri, > > > > > > > > > > > > Thanks, both. PR 1 is ready for review: > > > > > > https://github.com/apache/polaris/pull/5513 > > > > > > > > > > > > I pushed a revision last night that takes in Dimitri’s comments. > > > > > > > > > > > > > > > > > > * > > > > > > I changed from issuer to credentialVendingMechanism. It’s also > now > > a > > > > > plain > > > > > > string and the realm allowlist is > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which still defaults > to > > > > [STS] > > > > > > * > > > > > > Mechanisms are CDI beans found by their identifier, as Dimitri > > > > proposed. > > > > > > * > > > > > > The CLOUDFLARE_R2 bean and the R2 vending code will follow in the > > > next > > > > PR > > > > > > > > > > > > > > > > > > CI workflow is waiting approval to run. > > > > > > > > > > > > Thanks, > > > > > > Austen > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > Date: Tuesday, September 15, 2026 at 20:00 > > > > > > To: [email protected] <[email protected]> > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped > credential > > > > > vending > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > Having getCloudflareR2Endpoint on AwsStorageConfigurationInfo > > > > > > > > > > > > Yes, this does feel awkward. > > > > > > > > > > > > From my POV pursuing the option with the "credentialIssuers" > > property > > > > > looks > > > > > > promising. This property is fairly generic and fits naturally > into > > > the > > > > > > config, given that we already target multiple backends. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu <[email protected]> > > > wrote: > > > > > > > > > > > > > Hi Austen, Dmitri, > > > > > > > > > > > > > > > Having getCloudflareR2Endpoint on AwsStorageConfigurationInfo > > > > > > > > > > > > > > That does feel awkward. However, AwsStorageConfigurationInfo > has > > > > > > supported > > > > > > > S3-compatible backends beyond AWS for a while, so there is > > > precedent > > > > > > (e.g., > > > > > > > MinIO, etc.). The underlying mismatch is that we call the > storage > > > > type > > > > > > S3, > > > > > > > but its configuration mixes shared S3 settings with > AWS-specific > > > > ones: > > > > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: S3 > > > > > > > ├── Common S3 settings > > > > > > > │ endpoint, region, pathStyleAccess, allowedLocations > > > > > > > └── AWS-specific settings > > > > > > > roleArn, externalId, STS, KMS > > > > > > > > > > > > > > I’m open to separating the shared S3 settings from > > > provider-specific > > > > > > > configuration if that helps, but I don’t think we need to > settle > > > on a > > > > > > > broader refactor now. > > > > > > > > > > > > > > Let me know when the PR is ready for review. I’d be happy to > > take a > > > > > > closer > > > > > > > look. > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri Bourlatchkov < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > +1 to adding "credentialIssues" or similar (renamed) property > > to > > > S3 > > > > > > > config. > > > > > > > > In fact, I was looking exactly for something like that last > > week > > > > but > > > > > > the > > > > > > > > idea did not quite condense in my mind :) > > > > > > > > > > > > > > > > I posted some preliminary comments on the draft PR. > > > > > > > > > > > > > > > > Just for community awareness, I'd like to propose here that > we > > > use > > > > > CDI > > > > > > > for > > > > > > > > finding actual "issuer" / "mechanism" implementations. The > > > codebase > > > > > has > > > > > > > > many examples of this. The closest match in this case is > > > > > > > > probably PolarisStorageIntegrationProvider. > > > > > > > > > > > > > > > > This way the API does not have to enumerate all possible > > values. > > > > > > > Downstream > > > > > > > > builds are free to add different implementations without > having > > > to > > > > > > change > > > > > > > > the API spec. The "allowed" list still makes sense. > > > > > > > > > > > > > > > > In general, I like the direction this feature is taking, > > > although I > > > > > > > expect > > > > > > > > PR reviews may take a few rounds :) > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > Closing the loop since my last note and answering the > > questions > > > > > that > > > > > > > have > > > > > > > > > come up. > > > > > > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t have AWS-style > IAM > > > > roles > > > > > > or > > > > > > > > > ARNs. Our implementation signs temporary credentials with a > > > > parent > > > > > > > > token’s > > > > > > > > > secret, scoped to a bucket and a set of prefixes. The > account > > > ID > > > > > and > > > > > > > > > jurisdiction come from the R2 endpoint. Putting the account > > ID > > > in > > > > > > > roleArn > > > > > > > > > or userArn would give those fields a different meaning, so > > I’ve > > > > > left > > > > > > > them > > > > > > > > > absent for CLOUDFLARE_R2. > > > > > > > > > > > > > > > > > > I pursued Option B. The S3 configuration gains one new > field, > > > > > > > > > credentialIssuer, with values STS > > > > > > > > > (default) and CLOUDFLARE_R2 and a realm allowlist > > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults > > > > > > > > > to [STS]. I left stsUnavailable untouched (Prithvi) so it > > still > > > > > means > > > > > > > > > "vend nothing". The > > > > > > > > > credentialIssuer is the explicit sign. The R2 config > rejects > > > > > > > > > stsUnavailable along with a host of other STS and KMS > fields. > > > I'm > > > > > > > running > > > > > > > > > this in our non-production environment at the moment. > > > > > > > > > > > > > > > > > > Draft PR for stage 1 (config field, API, allowlist, gates): > > > > > > > > > https://github.com/apache/polaris/pull/5513. > > > > > > > > > > > > > > > > > > I still see the appeal of a separate R2 configuration, as > > > Dmitri > > > > > > > > > suggested. My original reason for separating it was to keep > > the > > > > > > > > R2-specific > > > > > > > > > configuration and validation together. Having > > > > > getCloudflareR2Endpoint > > > > > > > on > > > > > > > > > AwsStorageConfigurationInfo is one part I’d still like > > feedback > > > > on. > > > > > > > > > > > > > > > > > > I wanted to put a working version of Option B in front of > > > > everyone > > > > > so > > > > > > > we > > > > > > > > > could discuss the actual changes. In my earlier > > implementation, > > > > the > > > > > > > > > separate config also meant a separate storage type sharing > > > > > S3FileIO. > > > > > > > That > > > > > > > > > was the tradeoff that led me to try Option B. > > > > > > > > > > > > > > > > > > Do the diffs change anyone’s lean here? And are the names > > > > > > > > credentialIssuer > > > > > > > > > / STS / CLOUDFLARE_R2 / SUPPORTED_S3_CREDENTIAL_ISSUERS > > > > acceptable? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > Date: Sunday, September 13, 2026 at 19:50 > > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped > > > > credential > > > > > > > > vending > > > > > > > > > > > > > > > > > > Building on top of Ayush's and JB's suggestion, I agree > that > > > > > > > jurisdiction > > > > > > > > > can live inside the existing endpoint config field. > > > > > > > > > > > > > > > > > > That leaves only Account ID, so potentially we could store > it > > > in > > > > > user > > > > > > > or > > > > > > > > > role ARN. Note that userArn is currently unused in OSS > code. > > > > > > > > > > > > > > > > > > That should allow us to keep existing > > > AwsStorageConfigurationInfo > > > > > for > > > > > > > all > > > > > > > > > S3 storage backend, but will require special parsing of > > > user/role > > > > > ARN > > > > > > > > field > > > > > > > > > to distinguish R2 from the other systems. > > > > > > > > > > > > > > > > > > I'm still leaning towards the separate > > > > R2StorageConfigurationInfo, > > > > > > but > > > > > > > > I'd > > > > > > > > > be ok with overloading user/role ARN too if other people > > prefer > > > > > > that. I > > > > > > > > > guess a key question in this context is whether R2 has the > > > > concept > > > > > of > > > > > > > > role > > > > > > > > > ARN at all.... but TBH I have no idea about that. I hope > > Austen > > > > > might > > > > > > > be > > > > > > > > > able to clarify that. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > Welcome, and thanks for starting with a [DISCUSS]. Best > way > > > to > > > > > > make a > > > > > > > > > first > > > > > > > > > > contribution :) > > > > > > > > > > > > > > > > > > > > I think the thread is talking about two layers, and we > can > > > keep > > > > > > both > > > > > > > > > > answers. > > > > > > > > > > > > > > > > > > > > On the client side this should stay S3: s3:// locations, > > s3.* > > > > > > > > properties, > > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB / Iceberg Java results > > > > already > > > > > > show > > > > > > > > > that > > > > > > > > > > unmodified clients work, so a dedicated R2FileIO does not > > > seem > > > > > > worth > > > > > > > > it. > > > > > > > > > > On the server side I lean toward Dmitri: a distinct > > > > > > > > > > R2StorageConfigurationInfo and storage integration. > > Issuance > > > is > > > > > not > > > > > > > > STS, > > > > > > > > > > and folding Cloudflare fields into > > > AwsStorageConfigurationInfo > > > > > > mixes > > > > > > > > > > unrelated concepts. The middle ground to avoid is a new > > type > > > > that > > > > > > is > > > > > > > > > still > > > > > > > > > > AWS-shaped underneath. > > > > > > > > > > > > > > > > > > > > One concrete caution if the Option B prototype stays on > the > > > S3 > > > > > > > config: > > > > > > > > > > please do not overload stsUnavailable. Today that flag > > skips > > > > > > > AssumeRole > > > > > > > > > and > > > > > > > > > > disables vending. R2 wants the opposite. vend, just not > via > > > > STS. > > > > > so > > > > > > > > that > > > > > > > > > > flag would change meaning for existing MinIO / NetApp > > > catalogs. > > > > > If > > > > > > we > > > > > > > > > stay > > > > > > > > > > on S3 config, we need an explicit vending signal; a > > dedicated > > > > R2 > > > > > > type > > > > > > > > > > avoids that trap. > > > > > > > > > > > > > > > > > > > > On config surface, jurisdiction can live in endpoint > > > > (<account>. > > > > > > > > > > r2.cloudflarestorage.com vs <account>.eu.r2...). Account > > ID > > > is > > > > > > still > > > > > > > > > > needed > > > > > > > > > > for the JWT sub, so keeping an explicit accountId and > > > deriving > > > > > > > audience > > > > > > > > > > from the endpoint seems enough. Parent tokens should stay > > in > > > > > server > > > > > > > > > config > > > > > > > > > > via storageName. > > > > > > > > > > Local signing fits the vending model. Fail-closed on > > > > cross-bucket > > > > > > and > > > > > > > > > mixed > > > > > > > > > > read/write grants is the right call. Prefix claims should > > > > follow > > > > > > the > > > > > > > > same > > > > > > > > > > path comparison Polaris already uses, so we do not mint a > > > > > > credential > > > > > > > > > wider > > > > > > > > > > than the grant. > > > > > > > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator (#4486) separately > if > > > R2 > > > > > > needs > > > > > > > > it, > > > > > > > > > > then management API + Java config, then vending, Python > CLI > > > on > > > > > its > > > > > > > own. > > > > > > > > > > Sketching MinIO / Backblaze against the same fields is a > > > useful > > > > > > > design > > > > > > > > > > check; a generic non-STS SPI can wait for a second > backend. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri Bourlatchkov < > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > Thanks for starting this contribution! > > > > > > > > > > > > > > > > > > > > > > After briefly looking through your PR [1] I think > > > > > > > > > > > adding R2StorageConfigInfo makes sense. > > > > > > > > > > > > > > > > > > > > > > Even though "aws" in AwsStorageConfigurationInfo is > > longer > > > > > > implies > > > > > > > > AWS > > > > > > > > > > > services, and MinIO, Rust, Ozone are supported just as > > > well, > > > > R2 > > > > > > > > appears > > > > > > > > > > to > > > > > > > > > > > be significantly different from STS-based S3 systems > > that a > > > > > > > separate > > > > > > > > > > config > > > > > > > > > > > is probably the most natural way to support it in the > > > Polaris > > > > > > > > codebase. > > > > > > > > > > > > > > > > > > > > > > The alternative, I guess, is to add R2 accountId and > > > > > jurisdiction > > > > > > > as > > > > > > > > > > > optional properties to AwsStorageConfigurationInfo (or > > > > overload > > > > > > > > > existing > > > > > > > > > > > properties), but that looks like mixing unrelated > > concepts > > > > > > > together. > > > > > > > > > > > > > > > > > > > > > > Adding a generic property bag to > > > AwsStorageConfigurationInfo > > > > is > > > > > > not > > > > > > > > > > > convenient because > PolarisStorageIntegrationProviderImpl > > > will > > > > > > > > probably > > > > > > > > > > have > > > > > > > > > > > to know how to interpret it in order to redirect > > credential > > > > > > vending > > > > > > > > > > > requests to R2-specific code. > > > > > > > > > > > > > > > > > > > > > > I wonder whether R2StorageConfigurationInfo and > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > (and corresponding "storage integration" classes) could > > > > share a > > > > > > > > common > > > > > > > > > > base > > > > > > > > > > > class for common properties (e.g. endpoint). Even > though > > > > > > "endpoint" > > > > > > > > > might > > > > > > > > > > > not be relevant to R2 in the cloud, the fact that > > S3FileIO > > > > uses > > > > > > it, > > > > > > > > and > > > > > > > > > > R2 > > > > > > > > > > > is accessed via S3FileIO means we should probably allow > > > users > > > > > to > > > > > > > > > > configure > > > > > > > > > > > the full set of FileIO properties just in case. > > > > > > > > > > > > > > > > > > > > > > I think it is fine for different > > PolarisStorageIntegration > > > > > > > > > > implementations > > > > > > > > > > > to produce the same kind of StorageAccessConfig (S3 in > > this > > > > > case) > > > > > > > and > > > > > > > > > > thus > > > > > > > > > > > cause S3FileIO to be used by clients (including > Polaris' > > > own > > > > > > > storage > > > > > > > > > > access > > > > > > > > > > > paths). > > > > > > > > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] look a bit > concerning > > > to > > > > > me. > > > > > > I > > > > > > > > > wonder > > > > > > > > > > > if this code could be refactored to avoid dependencies > on > > > > > storage > > > > > > > > > config > > > > > > > > > > > types completely.... but TBH, I did not look too deeply > > > into > > > > > this > > > > > > > > > today. > > > > > > > > > > > > > > > > > > > > > > Anticipating future PRs for this in the Polaris repo, > I'd > > > > like > > > > > to > > > > > > > ask > > > > > > > > > to > > > > > > > > > > > separate Python code changes from java changes since > > these > > > > > areas > > > > > > > > > usually > > > > > > > > > > > attract different reviewers :) > > > > > > > > > > > > > > > > > > > > > > From my POV, management API and matching java changes > can > > > be > > > > in > > > > > > the > > > > > > > > > same > > > > > > > > > > > PR. I'd say docs should come later (to keep PRs small). > > > > > > > > > > > > > > > > > > > > > > All in all, thank you again and I hope this > contribution > > > > lands > > > > > in > > > > > > > > > Polaris > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > [1] https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM Jean-Baptiste Onofré < > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and welcome aboard > :) > > > > > > > > > > > > > > > > > > > > > > > > I think it makes sense to have R2 built under the > > > existing > > > > S3 > > > > > > > > storage > > > > > > > > > > > > configuration. It matches how Polaris is already > > > > structured: > > > > > > > > > > > > StorageType is keyed on the URI scheme (s3://, > s3a://), > > > and > > > > > > > > > > > > AwsStorageConfigurationInfo already carries endpoint, > > > > > > > stsEndpoint, > > > > > > > > > > > > pathStyleAccess and, importantly, an stsUnavailable > > flag. > > > > > > > > > > > > So, what you need is largely there already: correct > me > > if > > > > I'm > > > > > > > > wrong, > > > > > > > > > > > > R2 is essentially "S3 compatible store where STS is > > > > > > unavailable, > > > > > > > > plus > > > > > > > > > > > > a way to actually vend credentials in that case". > Going > > > > this > > > > > > > route > > > > > > > > > > > > also keeps this work "isolated" from the > > > > > > FileIO-per-storage-type > > > > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > The interesting design question is a pluggable > > credential > > > > > > vending > > > > > > > > > path > > > > > > > > > > > > for S3 compatible stores when STS is not available, > > with > > > > > > > > Cloudflare's > > > > > > > > > > > > local signing as the first concrete implementation, > > > rather > > > > > than > > > > > > > R2 > > > > > > > > > > > > specific code. It would be good to sketch how MinIO > > would > > > > > slot > > > > > > > into > > > > > > > > > > > > the same config fields before the vending path > hardens > > > > around > > > > > > R2. > > > > > > > > > > > > > > > > > > > > > > > > I believe we have to check if Cloudflare account ID > and > > > > > > > > jurisdiction > > > > > > > > > > > > really need dedicated config fields, or can we use > the > > > > > endpoint > > > > > > > > value > > > > > > > > > > > > (with documentation)? Keeping the configuration > surface > > > > > minimal > > > > > > > > will > > > > > > > > > > > > help the design generalize. > > > > > > > > > > > > > > > > > > > > > > > > I would love to discuss two things: > > > > > > > > > > > > 1. Polaris shipping a Cloudflare API token and > locally > > > > > minting > > > > > > > > scoped > > > > > > > > > > > > credentials makes it effectively an STS for R2, which > > > > > increases > > > > > > > the > > > > > > > > > > > > blast radius of a server compromise. Your "reject > > rather > > > > than > > > > > > > > > boarden" > > > > > > > > > > > > handling of unrepresentable scopes is the right call. > > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java results are promising. > > Just > > > > > > curious > > > > > > > > > about > > > > > > > > > > > > Spark or Trino. > > > > > > > > > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I would suggest to > > start > > > > with > > > > > > the > > > > > > > > S3 > > > > > > > > > > > > config and the management API as first shoot, and we > > can > > > > > follow > > > > > > > > with > > > > > > > > > > > > the credential vending implementation. > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen Tomek > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago Trading Company. We > use > > > > > Polaris > > > > > > > for > > > > > > > > > our > > > > > > > > > > > > Iceberg catalogs and we're evaluating Cloudflare R2 > as > > an > > > > > > object > > > > > > > > > store > > > > > > > > > > > (the > > > > > > > > > > > > no egress fees make it a very compelling product). > I’d > > > like > > > > > to > > > > > > > > > discuss > > > > > > > > > > > > contributing R2 support and credential vending to > > Polaris > > > > > with > > > > > > > the > > > > > > > > > goal > > > > > > > > > > > of > > > > > > > > > > > > getting feedback before opening a ready-for-review > > > upstream > > > > > PR. > > > > > > > > This > > > > > > > > > is > > > > > > > > > > > my > > > > > > > > > > > > first contribution, so trying to do this right. > > > > > > > > > > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it does not > provide > > > AWS > > > > > > STS. > > > > > > > > > > > > Cloudflare instead supports these short-lived, scoped > > > > > > credentials > > > > > > > > > > > generated > > > > > > > > > > > > by locally signed JWT with the server holding the > > parent > > > > API > > > > > > > token > > > > > > > > > [1]. > > > > > > > > > > > > This allows us to retain Polaris’s authorization and > > > > > > > > > credential-vending > > > > > > > > > > > > model while storing our table data in R2. > > > > > > > > > > > > > > > > > > > > > > > > > > I have a prototype with the following design: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > An opt-in R2 storage type, with catalog > configuration > > > > > > > identifying > > > > > > > > > the > > > > > > > > > > > > Cloudflare account and optional jurisdiction. Parent > > > > > > credentials > > > > > > > > > remain > > > > > > > > > > > in > > > > > > > > > > > > server configuration and can be selected through > > > > storageName. > > > > > > > > > > > > > * > > > > > > > > > > > > > Local credential generation using Cloudflare’s > > > documented > > > > > > > format. > > > > > > > > > > > > Credentials expire, are restricted to one bucket, and > > > carry > > > > > > > prefix > > > > > > > > > and > > > > > > > > > > > > access scopes derived from Polaris’s location grants. > > The > > > > > > > > integration > > > > > > > > > > > uses > > > > > > > > > > > > Polaris’s existing credential cache. > > > > > > > > > > > > > * > > > > > > > > > > > > > Credentials returned through the existing Iceberg > > REST > > > > > > vending > > > > > > > > flow > > > > > > > > > > as > > > > > > > > > > > > s3.* properties, including the endpoint, session > token, > > > and > > > > > > > expiry. > > > > > > > > > > > > Compatible clients use their existing S3 FileIO. > > > > > > > > > > > > > * > > > > > > > > > > > > > Conservative handling of scopes the implementation > > > cannot > > > > > > > > > represent: > > > > > > > > > > > > cross-bucket and mixed read/write grants are rejected > > > > rather > > > > > > than > > > > > > > > > > > > broadened. R2 is excluded from the default > > > > > > > supported-storage-types > > > > > > > > > > list. > > > > > > > > > > > > > > > > > > > > > > > > > > The main design questions I feel this runs into: > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > Should this be a distinct storage type or a > > > > > > credential-vending > > > > > > > > > > > mechanism > > > > > > > > > > > > selected through the existing S3 configuration? > > > > > > > > > > > > > * > > > > > > > > > > > > > I chose a separate type because the identity, > > > credential > > > > > > > > > generation, > > > > > > > > > > > and > > > > > > > > > > > > scoping model differ from AWS STS, as this seemed > more > > > > > > > appropriate > > > > > > > > > > > > > > > > > > > > > > > > > > There is a draft preview PR against my fork [2], > > > > including > > > > > > > > > > > > implementation, management API changes, CLI support, > > > tests, > > > > > and > > > > > > > > > > > > documentation. It also discloses the AI assistance > used > > > > > during > > > > > > > > > > > > implementation. I leaned heavily on Fable 5.1 for > this > > > > > > > > implementation > > > > > > > > > > so > > > > > > > > > > > I > > > > > > > > > > > > want to be upfront about it. > > > > > > > > > > > > > > > > > > > > > > > > > > We are running the implementation in a > non-production > > > > > > > environment > > > > > > > > > > right > > > > > > > > > > > > now as a POC. Validation includes live R2 > reads/writes, > > > > > > multipart > > > > > > > > > > > > operations, negative scope-boundary tests, our > internal > > > > > Python > > > > > > > > client > > > > > > > > > > > > regression matrix, and credential rotation/recovery > > > > > exercises. > > > > > > > The > > > > > > > > > > > upstream > > > > > > > > > > > > CI workflow has passed on the fork. > > > > > > > > > > > > > > > > > > > > > > > > > > The change also encounters the existing assumption > > > that a > > > > > > > FileIO > > > > > > > > > > > > implementation identifies exactly one storage type: > > both > > > S3 > > > > > and > > > > > > > R2 > > > > > > > > > use > > > > > > > > > > > > S3FileIO. This overlaps with #4486 [3]. > > > > > > > > > > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback on: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > Should R2 have its own storage configuration/type, > or > > > > > should > > > > > > > > > non-STS > > > > > > > > > > > > vending be modeled within the existing S3 > integration? > > > > > > > > > > > > > 2. > > > > > > > > > > > > > Does the local-signing approach fit Polaris’s > > > > > > > credential-vending > > > > > > > > > > model? > > > > > > > > > > > > Any additional constraints or validations necessary? > > > > > > > > > > > > > 3. > > > > > > > > > > > > > Would you prefer the shared FileIO validation > change > > to > > > > > land > > > > > > > > > > > separately, > > > > > > > > > > > > and how should we sequence the management API and R2 > > > > > > integration > > > > > > > > > > changes? > > > > > > > > > > > > > > > > > > > > > > > > > > Happy to adapt the design and split the > > contribution > > > > into > > > > > > > more > > > > > > > > > > > focused > > > > > > > > > > > > PRs. Wanted to start the discussion > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > > > > > > > > > > > > [2] > https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > [3] > https://github.com/apache/polaris/issues/4486 > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > > > > contain > > > > > > > > > > information > > > > > > > > > > > > intended for the exclusive use of the individual or > > > entity > > > > to > > > > > > > whom > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > > addressed and may contain information that is > > > proprietary, > > > > > > > > > confidential > > > > > > > > > > > > and/or exempt from disclosure under applicable law. > If > > > you > > > > > are > > > > > > > not > > > > > > > > > the > > > > > > > > > > > > intended recipient, you are hereby notified that any > > > > viewing, > > > > > > > > > copying, > > > > > > > > > > > > disclosure or distribution of this information may be > > > > subject > > > > > > to > > > > > > > > > legal > > > > > > > > > > > > restriction or sanction. Please notify the sender, by > > > > > > electronic > > > > > > > > mail > > > > > > > > > > or > > > > > > > > > > > > telephone, of any unintended recipients and delete > the > > > > > original > > > > > > > > > message > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > > > > > information > > > > > > > > > intended for the exclusive use of the individual or entity > to > > > > whom > > > > > it > > > > > > > is > > > > > > > > > addressed and may contain information that is proprietary, > > > > > > confidential > > > > > > > > > and/or exempt from disclosure under applicable law. If you > > are > > > > not > > > > > > the > > > > > > > > > intended recipient, you are hereby notified that any > viewing, > > > > > > copying, > > > > > > > > > disclosure or distribution of this information may be > subject > > > to > > > > > > legal > > > > > > > > > restriction or sanction. Please notify the sender, by > > > electronic > > > > > mail > > > > > > > or > > > > > > > > > telephone, of any unintended recipients and delete the > > original > > > > > > message > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > > information > > > > > > intended for the exclusive use of the individual or entity to > whom > > it > > > > is > > > > > > addressed and may contain information that is proprietary, > > > confidential > > > > > > and/or exempt from disclosure under applicable law. If you are > not > > > the > > > > > > intended recipient, you are hereby notified that any viewing, > > > copying, > > > > > > disclosure or distribution of this information may be subject to > > > legal > > > > > > restriction or sanction. Please notify the sender, by electronic > > mail > > > > or > > > > > > telephone, of any unintended recipients and delete the original > > > message > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > information > > > > > intended for the exclusive use of the individual or entity to whom > it > > > is > > > > > addressed and may contain information that is proprietary, > > confidential > > > > > and/or exempt from disclosure under applicable law. If you are not > > the > > > > > intended recipient, you are hereby notified that any viewing, > > copying, > > > > > disclosure or distribution of this information may be subject to > > legal > > > > > restriction or sanction. Please notify the sender, by electronic > mail > > > or > > > > > telephone, of any unintended recipients and delete the original > > message > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > > intended for the exclusive use of the individual or entity to whom it > is > > > addressed and may contain information that is proprietary, confidential > > > and/or exempt from disclosure under applicable law. If you are not the > > > intended recipient, you are hereby notified that any viewing, copying, > > > disclosure or distribution of this information may be subject to legal > > > restriction or sanction. Please notify the sender, by electronic mail > or > > > telephone, of any unintended recipients and delete the original message > > > without making any copies. > > > > > >
