Hi Austen, Your plan matches my understanding.
Also +1 to "CLOUDFLARE_R2" as the new mechanism name. Cheers, Dmitri. On Thu, Oct 1, 2026 at 3:56 PM Austen Tomek <[email protected]> wrote: > Thanks for the discussion today. My understanding of where we landed was > to keep credentialVendingMechanism a string and have the spec define the > values Apache Polaris accepts and what they mean. In this PR that is just > STS and the empty value still signifies the server default. In PR 2 I’ll be > adding CLOUDFLARE_R2 to the same list. The server already refuses any other > value with a 400, so it seems like this is just a documentation change. Let > me know if I am missing anything. > > Thanks, > Austen > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > From: Dmitri Bourlatchkov <[email protected]> > Date: Wednesday, September 30, 2026 at 12:31 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential vending > > Hi Yufei, > > Thanks for providing the details! Replying inline. > > > When a client creates a catalog with credentialVendingMechanism: "m1", > > what is the expected behavior? > > This depends on the meaning of "m1" and the specific server build. > > In Apache Polaris "m1" is not meaningful and is not supported. > The server will reject this value on the corresponding catalog > update request. This is handled in PolarisAdminService in the current PR > state. > > > 1. The server supports it: This is the ideal scenario (happy path). > > Indeed. > > > 2. The server does not support it: This creates inconsistency if a > client > > that works with Server A fails on Server B. > > In my view this is not an inconsistency as different servers do not have > to provide the same capabilities. > > Creating a catalog with a specific credentialVendingMechanism is an > administrative task. The admin user must be aware of what the server > supports. This is a deployment / configuration concern. Not a runtime > compatibility concern. > > > 3. The server supports "m1" differently: This is the worst-case > scenario, > > where the client's expectations conflict with the server's actual > behavior. > > This is a good point! > > My take is that if "m1" is defined in Apache Polaris, then all server > builds must implement it the same way. > > The definition will be expressed by means of S3CredentialVendingMechanism > classes included in standard Apache Polaris releases (and hopefully docs > too). > > Currently only "STS" and "DEFAULT" (falling back on STS) are defined now. > > If "m1" is not defined in Apache Polaris, custom builds are free to assign > any meaning to it and do not have to be mutually compatible. This is the > main point for extensibility. > > Note that custom builds are not "Apache Polaris" anymore and cannot > claim to be the "same" as Apache Polaris. This is a matter of ASF > trademarks. > Normally, custom builds say something like "Powered by Apache Polaris". > > Again, the admin user setting "m1" as a vending mechanism in a particular > server must be aware of what that server provides and understand > the meaning of "m1" in that context. The onus to describe the meaning of > "m1" to users in this case rests with the custom server developers. > > We can talk about preventing name overlaps, but I do not think establishing > any kind of name registry is worth the trouble. In this day and age, I > believe > all downstream project developers are aware of name overlap issues. > and will follow normal practices to avoid name collisions. > > I hope this clears your concerns. > > Cheers, > Dmitri. > > On Wed, Sep 30, 2026 at 1:02 PM Yufei Gu <[email protected]> wrote: > > > Hi Dmitri, > > > > When a client creates a catalog with credentialVendingMechanism: "m1", > what > > is the expected behavior? > > > > Here is a suitaion if we allow a freeform string, the client might > > encounter three different scenarios depending on the server: > > 1. The server supports it: This is the ideal scenario (happy path). > > 2. The server does not support it: This creates inconsistency if a > client > > that works with Server A fails on Server B. > > 3. The server supports "m1" differently: This is the worst-case > scenario, > > where the client's expectations conflict with the server's actual > behavior. > > > > I don't think this is the situation we want users to run into. The fix is > > quite simple, using a specified values instead of a freefrom string. > > > > Yufei > > > > > > On Tue, Sep 29, 2026 at 9:48 PM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Since concerns were raised, it's important to take a moment to align > > > rather than rushing. That's the way we drive consensus :) > > > > > > On the technical question of string vs enum for > > > credentialVendingMechanism, here is my take: > > > > > > 1. This field is part of the Management API for catalog > > > administrators. Iceberg REST clients (Spark, Trino, ...) never see or > > > interact with credentialVendingMechanism. They only consume standard > > > vended credentials (s3.* session tokens/keys) during table > > > load/commit. There is no risk of breaking query engine clients across > > > deployments. > > > 2. One of our primary goals when discussing S3-compatible stores (R2, > > > MinIO, ...) was to keep credential vending pluggable via SPI rather > > > than hardcoding vendor-specific paths in core. If we use a closed enum > > > in the OpenAPI spec, it means: > > > * any new mechanism (including R2 in Phase 2 or MinIO) requires a > spec > > > change. > > > * any downstream deployment or plugin providing a custom mechanism > > > would break OpenAPI-generated client SDKs due to strict enum > > > validation. > > > * runtime validation on the server (returning HTTP 400 with a clear > > > message) combined with the realm allowlist provides the right > > > server-side guardrail without restricting extensibility. > > > 3. Yufei's concerns about a clear contract and avoiding confusion is > > > valid. We can address contract clarity without closing the enum: > > > * we keep type: string, but document the well-known/standard values > > > (STS, DEFAULT) explicitly in the description and examples in the > > > OpenAPI spec (as PR #5513 does). > > > * we ensure the server validates cleanly with case-insensitivity (or > > > canonical normalization) so administrators get immediate, actionable > > > feedback on typos. > > > > > > PR #5513 does a solid job laying this foundation so Austen can move > > > forward with Phase 2. If we keep it as a string with well-documented > > > standard values and server-siide validation, I think it's the right > > > balance between spec clarify and pluggability. > > > > > > Thoughts? > > > > > > Regards > > > JB > > > > > > On Wed, Sep 30, 2026 at 12:50 AM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > > > > Hi Yufei, > > > > > > > > > > Conversely, an enum will require defining all possible vending > > > > mechanisms in > > > > > > the Polaris API spec. > > > > > > > > > Yes. Otherwise, a client may failed in certain deployments. > > > > > > > > Could you describe this failure mode in terms of API request/response > > > flow? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > On Tue, Sep 29, 2026 at 2:06 PM Yufei Gu <[email protected]> > wrote: > > > > > > > > > > Conversely, an enum will require defining all possible vending > > > > > mechanisms in > > > > > the Polaris API spec. > > > > > > > > > > Yes. Otherwise, a client may failed in certain deployments. > > > > > > > > > > > This will prevent downstream projects from using custom vending > > > > > mechanisms. > > > > > > > > > > No. if downstream needs custom vending mechanisms, they can always > > > build > > > > > their own Polaris. > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 11:01 AM Dmitri Bourlatchkov < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > Using an enum will clarify the REST spec contract. > > > > > > > > > > > > I'm not sure I understand your point. Current PR clearly > describes > > > the > > > > > > meaning of "credentialVendingMechanism" and lists options > available > > > > > > out-of-the-box. > > > > > > > > > > > > What does an enum achieve, that (current) runtime validation does > > > not? > > > > > > > > > > > > Conversely, an enum will require defining all possible vending > > > mechanisms > > > > > > in the Polaris API spec. This will prevent downstream projects > from > > > using > > > > > > custom vending mechanisms. > > > > > > > > > > > > I believe Polaris as a project currently promotes downstream > > > > > > customizability. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Tue, Sep 29, 2026 at 1:21 PM Yufei Gu <[email protected]> > > > wrote: > > > > > > > > > > > > > I don't think we have consensus on the PR, esp. a client should > > NOT > > > > > > depend > > > > > > > on a specific deployment. Using an enum will clarify the REST > > spec > > > > > > > contract. > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > PR [5513] has been in review for quite some time. I believe > we > > > have > > > > > > > general > > > > > > > > consensus on the Management API changes and java code > changes. > > > > > > > > > > > > > > > > The only point that may still have divergent opinions is > > whether > > > to > > > > > use > > > > > > > > free text or enum for "credentialVendingMechanism". I > > personally > > > > > > support > > > > > > > > free text (rationale explained in previous comments). This > > > > > conversation > > > > > > > has > > > > > > > > been dormant for a few days, so I assume lazy consensus on > the > > > > > current > > > > > > > > state of the PR (free text). In any case switching from free > > > text to > > > > > > enum > > > > > > > > is a non-breaking API change before the next release in case > > the > > > > > > > consensus > > > > > > > > shifts and we need to adjust. We have about one month for > > > followup > > > > > API > > > > > > > > changes. > > > > > > > > > > > > > > > > The java code introduced in this PR is not rigid and can > change > > > at > > > > > any > > > > > > > time > > > > > > > > per our evolution guidelines [1]. > > > > > > > > > > > > > > > > I propose merging PR [5513] on Sept 30 unless fresh blocking > > > concerns > > > > > > are > > > > > > > > raised. > > > > > > > > > > > > > > > > [1] https://polaris.apache.org/releases/1.8.0/evolution/ > > > > > > > > > > > > > > > > [5513] https://github.com/apache/polaris/pull/5513 > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > Hey all, > > > > > > > > > > > > > > > > > > Any chance I can get further reviews this week on the PR? > I’d > > > like > > > > > to > > > > > > > get > > > > > > > > > moving onto Phase 2 here if there are no further issues to > > > address. > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > Date: Thursday, September 24, 2026 at 17:33 > > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped > > > credential > > > > > > > > vending > > > > > > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > > > > > > The REST spec is a contract between Polaris servers and > > > clients > > > > > > [...] > > > > > > > > > > > > > > > > > > I tend to think that this statement is a bit too strong. > The > > > > > > Management > > > > > > > > API > > > > > > > > > spec defines a protocol for clients and servers to > > communicate. > > > > > > > > > > > > > > > > > > > [...] This would help avoid a situation where a Polaris > > > client > > > > > > works > > > > > > > > > > only with certain deployments. > > > > > > > > > > > > > > > > > > Since Polaris has been developed as a customizable > system, a > > > REST > > > > > > API > > > > > > > > > specification defined by Apache Polaris cannot control the > > > > > behaviour > > > > > > of > > > > > > > > all > > > > > > > > > possible downstream environments. The purpose of the spec > is > > to > > > > > > ensure > > > > > > > > > protocol compatibility. > > > > > > > > > > > > > > > > > > Note that the value of `credentialVendingMechanism` has no > > > bearing > > > > > on > > > > > > > > > any logic clients need to execute. It is an opaque > parameter > > > > > > controlled > > > > > > > > > by the Polaris administrator. > > > > > > > > > > > > > > > > > > > The new credentialVendingMechanism field in the REST spec > > > looks > > > > > > > > > > reasonable. My only suggestion is to use an enum instead > > of a > > > > > > string. > > > > > > > > > [...] > > > > > > > > > > > > > > > > > > The above points are exactly the reason why the Management > > API > > > > > cannot > > > > > > > > > enumerate all possible vending mechanisms. > > > > > > > > > > > > > > > > > > The same applies to storage types too. However, we do not > > have > > > any > > > > > > > > > practical > > > > > > > > > examples of 3rd party storage systems that go beyond > > > S3/GCS/ADLS > > > > > > (yet). > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks Austen. It's fair that endpoints are not reliable > > > given > > > > > that > > > > > > > > some > > > > > > > > > > s3-compatible can use arbitrary hostnames. > > > > > > > > > > I also think stsUnavailable is fine to keep as-is. I > don’t > > > think > > > > > we > > > > > > > > need > > > > > > > > > to > > > > > > > > > > change its scope or repurpose it. > > > > > > > > > > > > > > > > > > > > The new credentialVendingMechanism field in the REST spec > > > looks > > > > > > > > > reasonable. > > > > > > > > > > My only suggestion is to use an enum instead of a string. > > The > > > > > REST > > > > > > > spec > > > > > > > > > is > > > > > > > > > > a contract between Polaris servers and clients, so I’d > > > expect it > > > > > to > > > > > > > > > define > > > > > > > > > > the valid values and their meanings rather than leaving > > them > > > up > > > > > to > > > > > > > each > > > > > > > > > > deployment. This would help avoid a situation where a > > Polaris > > > > > > client > > > > > > > > > works > > > > > > > > > > only with certain deployments. Deployment-specific > > mechanisms > > > > > > > shouldn’t > > > > > > > > > be > > > > > > > > > > part of the standard REST contract; users who need a > custom > > > > > > mechanism > > > > > > > > can > > > > > > > > > > always extend the server in their own deployment. > > > > > > > > > > > > > > > > > > > > > Either way the operator needs a per-realm switch like > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control > which > > > realm > > > > > > can > > > > > > > > > mint > > > > > > > > > > R2 credentials from the operators Cloudflare account. > > > > > > > > > > > > > > > > > > > > Could you clarify why we need a separate per-realm > switch? > > > Once > > > > > the > > > > > > > > > storage > > > > > > > > > > configuration is set at the catalog level to select a > > > specific > > > > > > > > mechanism, > > > > > > > > > > the catalog will have no choice but to support it. Do we > > need > > > > > > another > > > > > > > > > layer > > > > > > > > > > of gating? > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > Hey Yufei, > > > > > > > > > > > > > > > > > > > > > > Apologies for missing your question. > > > > > > > > > > > > > > > > > > > > > > I don’t think Endpoint is sufficient. > > > > > > > > > > > > > > > > > > > > > > Endpoint matching only works where the hostname > > identifies > > > the > > > > > > > vendor > > > > > > > > > > like > > > > > > > > > > > R2’s public hostnames that are recognizable, MinIO and > > > other > > > > > > > > > self-hosted > > > > > > > > > > > S3-compatible stores use arbitrary hostnames so there > > > would be > > > > > > > > nothing > > > > > > > > > to > > > > > > > > > > > match on. On top of that, if a match fails it would > just > > > > > silently > > > > > > > > fall > > > > > > > > > > back > > > > > > > > > > > to STS. > > > > > > > > > > > > > > > > > > > > > > Right now the endpoint just says how you’ll connect not > > > how the > > > > > > > > > > > credentials are vended. There's endpoint and also > > > > > > endpointInternal > > > > > > > > for > > > > > > > > > > > server-side access, so it would be unclear which of the > > two > > > > > > > decides. > > > > > > > > > > > > > > > > > > > > > > For your other question, there is no second mechanism > for > > > R2 > > > > > > today, > > > > > > > > but > > > > > > > > > > > for other self-hosted stores the same endpoint could be > > > served > > > > > by > > > > > > > STS > > > > > > > > > or > > > > > > > > > > by > > > > > > > > > > > a different token mechanism chosen per catalog. > > > > > > > > > > > > > > > > > > > > > > stsUnavailable keeps it’s current behavior in the PR > > (that > > > was > > > > > > > > > Prithvi’s > > > > > > > > > > > point earlier in the thread) > > > > > > > > > > > > > > > > > > > > > > Either way the operator needs a per-realm switch like > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control > > which > > > > > realm > > > > > > > can > > > > > > > > > > mint > > > > > > > > > > > R2 credentials from the operators Cloudflare account. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > From: Yufei Gu <[email protected]> > > > > > > > > > > > Date: Monday, September 21, 2026 at 17:53 > > > > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with > scoped > > > > > > credential > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > Thanks for the update. After looking at PR #5513, I > think > > > we > > > > > > should > > > > > > > > > first > > > > > > > > > > > clarify whether we need a new > credentialVendingMechanism > > > field > > > > > in > > > > > > > the > > > > > > > > > > > spec/config. > > > > > > > > > > > > > > > > > > > > > > Austen, you mentioned that the R2 account ID and > > > jurisdiction > > > > > > > already > > > > > > > > > > come > > > > > > > > > > > from the endpoint. Could we also use the existing field > > > like > > > > > > > > > `endpoint`, > > > > > > > > > > to > > > > > > > > > > > recognize R2 and select its credential-vending > mechanism, > > > while > > > > > > > > keeping > > > > > > > > > > the > > > > > > > > > > > current stsUnavailable behavior unchanged? > > > > > > > > > > > > > > > > > > > > > > Are there cases where the endpoint isn’t enough, or > where > > > users > > > > > > > need > > > > > > > > to > > > > > > > > > > > choose different vending mechanisms for the same > > > endpoint(e.g., > > > > > > R2) > > > > > > > > per > > > > > > > > > > > catalog? > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek < > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Yufei, Dimitri, > > > > > > > > > > > > > > > > > > > > > > > > Thanks, both. PR 1 is ready for review: > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513 > > > > > > > > > > > > > > > > > > > > > > > > I pushed a revision last night that takes in > Dimitri’s > > > > > > comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > I changed from issuer to credentialVendingMechanism. > > It’s > > > > > also > > > > > > > now > > > > > > > > a > > > > > > > > > > > plain > > > > > > > > > > > > string and the realm allowlist is > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which > still > > > > > > defaults > > > > > > > to > > > > > > > > > > [STS] > > > > > > > > > > > > * > > > > > > > > > > > > Mechanisms are CDI beans found by their identifier, > as > > > > > Dimitri > > > > > > > > > > proposed. > > > > > > > > > > > > * > > > > > > > > > > > > The CLOUDFLARE_R2 bean and the R2 vending code will > > > follow in > > > > > > the > > > > > > > > > next > > > > > > > > > > PR > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > CI workflow is waiting approval to run. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > > > > Date: Tuesday, September 15, 2026 at 20:00 > > > > > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with > > scoped > > > > > > > credential > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > Having getCloudflareR2Endpoint on > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > > > > > > > > Yes, this does feel awkward. > > > > > > > > > > > > > > > > > > > > > > > > From my POV pursuing the option with the > > > "credentialIssuers" > > > > > > > > property > > > > > > > > > > > looks > > > > > > > > > > > > promising. This property is fairly generic and fits > > > naturally > > > > > > > into > > > > > > > > > the > > > > > > > > > > > > config, given that we already target multiple > backends. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, Dmitri, > > > > > > > > > > > > > > > > > > > > > > > > > > > Having getCloudflareR2Endpoint on > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > > > > > > > > > > That does feel awkward. However, > > > > > AwsStorageConfigurationInfo > > > > > > > has > > > > > > > > > > > > supported > > > > > > > > > > > > > S3-compatible backends beyond AWS for a while, so > > > there is > > > > > > > > > precedent > > > > > > > > > > > > (e.g., > > > > > > > > > > > > > MinIO, etc.). The underlying mismatch is that we > call > > > the > > > > > > > storage > > > > > > > > > > type > > > > > > > > > > > > S3, > > > > > > > > > > > > > but its configuration mixes shared S3 settings with > > > > > > > AWS-specific > > > > > > > > > > ones: > > > > > > > > > > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: S3 > > > > > > > > > > > > > ├── Common S3 settings > > > > > > > > > > > > > │ endpoint, region, pathStyleAccess, > > > allowedLocations > > > > > > > > > > > > > └── AWS-specific settings > > > > > > > > > > > > > roleArn, externalId, STS, KMS > > > > > > > > > > > > > > > > > > > > > > > > > > I’m open to separating the shared S3 settings from > > > > > > > > > provider-specific > > > > > > > > > > > > > configuration if that helps, but I don’t think we > > need > > > to > > > > > > > settle > > > > > > > > > on a > > > > > > > > > > > > > broader refactor now. > > > > > > > > > > > > > > > > > > > > > > > > > > Let me know when the PR is ready for review. I’d be > > > happy > > > > > to > > > > > > > > take a > > > > > > > > > > > > closer > > > > > > > > > > > > > look. > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri > Bourlatchkov < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 to adding "credentialIssues" or similar > > (renamed) > > > > > > property > > > > > > > > to > > > > > > > > > S3 > > > > > > > > > > > > > config. > > > > > > > > > > > > > > In fact, I was looking exactly for something like > > > that > > > > > last > > > > > > > > week > > > > > > > > > > but > > > > > > > > > > > > the > > > > > > > > > > > > > > idea did not quite condense in my mind :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > I posted some preliminary comments on the draft > PR. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just for community awareness, I'd like to propose > > > here > > > > > that > > > > > > > we > > > > > > > > > use > > > > > > > > > > > CDI > > > > > > > > > > > > > for > > > > > > > > > > > > > > finding actual "issuer" / "mechanism" > > > implementations. > > > > > The > > > > > > > > > codebase > > > > > > > > > > > has > > > > > > > > > > > > > > many examples of this. The closest match in this > > > case is > > > > > > > > > > > > > > probably PolarisStorageIntegrationProvider. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This way the API does not have to enumerate all > > > possible > > > > > > > > values. > > > > > > > > > > > > > Downstream > > > > > > > > > > > > > > builds are free to add different implementations > > > without > > > > > > > having > > > > > > > > > to > > > > > > > > > > > > change > > > > > > > > > > > > > > the API spec. The "allowed" list still makes > sense. > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I like the direction this feature is > > > taking, > > > > > > > > > although I > > > > > > > > > > > > > expect > > > > > > > > > > > > > > PR reviews may take a few rounds :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek < > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Closing the loop since my last note and > answering > > > the > > > > > > > > questions > > > > > > > > > > > that > > > > > > > > > > > > > have > > > > > > > > > > > > > > > come up. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t have > > > > > AWS-style > > > > > > > IAM > > > > > > > > > > roles > > > > > > > > > > > > or > > > > > > > > > > > > > > > ARNs. Our implementation signs temporary > > > credentials > > > > > > with a > > > > > > > > > > parent > > > > > > > > > > > > > > token’s > > > > > > > > > > > > > > > secret, scoped to a bucket and a set of > prefixes. > > > The > > > > > > > account > > > > > > > > > ID > > > > > > > > > > > and > > > > > > > > > > > > > > > jurisdiction come from the R2 endpoint. Putting > > the > > > > > > account > > > > > > > > ID > > > > > > > > > in > > > > > > > > > > > > > roleArn > > > > > > > > > > > > > > > or userArn would give those fields a different > > > meaning, > > > > > > so > > > > > > > > I’ve > > > > > > > > > > > left > > > > > > > > > > > > > them > > > > > > > > > > > > > > > absent for CLOUDFLARE_R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I pursued Option B. The S3 configuration gains > > one > > > new > > > > > > > field, > > > > > > > > > > > > > > > credentialIssuer, with values STS > > > > > > > > > > > > > > > (default) and CLOUDFLARE_R2 and a realm > allowlist > > > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults > > > > > > > > > > > > > > > to [STS]. I left stsUnavailable untouched > > > (Prithvi) so > > > > > it > > > > > > > > still > > > > > > > > > > > means > > > > > > > > > > > > > > > "vend nothing". The > > > > > > > > > > > > > > > credentialIssuer is the explicit sign. The R2 > > > config > > > > > > > rejects > > > > > > > > > > > > > > > stsUnavailable along with a host of other STS > and > > > KMS > > > > > > > fields. > > > > > > > > > I'm > > > > > > > > > > > > > running > > > > > > > > > > > > > > > this in our non-production environment at the > > > moment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Draft PR for stage 1 (config field, API, > > allowlist, > > > > > > gates): > > > > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still see the appeal of a separate R2 > > > configuration, > > > > > as > > > > > > > > > Dmitri > > > > > > > > > > > > > > > suggested. My original reason for separating it > > > was to > > > > > > keep > > > > > > > > the > > > > > > > > > > > > > > R2-specific > > > > > > > > > > > > > > > configuration and validation together. Having > > > > > > > > > > > getCloudflareR2Endpoint > > > > > > > > > > > > > on > > > > > > > > > > > > > > > AwsStorageConfigurationInfo is one part I’d > still > > > like > > > > > > > > feedback > > > > > > > > > > on. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wanted to put a working version of Option B > in > > > front > > > > > of > > > > > > > > > > everyone > > > > > > > > > > > so > > > > > > > > > > > > > we > > > > > > > > > > > > > > > could discuss the actual changes. In my earlier > > > > > > > > implementation, > > > > > > > > > > the > > > > > > > > > > > > > > > separate config also meant a separate storage > > type > > > > > > sharing > > > > > > > > > > > S3FileIO. > > > > > > > > > > > > > That > > > > > > > > > > > > > > > was the tradeoff that led me to try Option B. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do the diffs change anyone’s lean here? And are > > the > > > > > names > > > > > > > > > > > > > > credentialIssuer > > > > > > > > > > > > > > > / STS / CLOUDFLARE_R2 / > > > SUPPORTED_S3_CREDENTIAL_ISSUERS > > > > > > > > > > acceptable? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > > > > > > > Date: Sunday, September 13, 2026 at 19:50 > > > > > > > > > > > > > > > To: [email protected] < > > [email protected] > > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support > with > > > > > scoped > > > > > > > > > > credential > > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Building on top of Ayush's and JB's > suggestion, I > > > agree > > > > > > > that > > > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > > > can live inside the existing endpoint config > > field. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That leaves only Account ID, so potentially we > > > could > > > > > > store > > > > > > > it > > > > > > > > > in > > > > > > > > > > > user > > > > > > > > > > > > > or > > > > > > > > > > > > > > > role ARN. Note that userArn is currently unused > > in > > > OSS > > > > > > > code. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That should allow us to keep existing > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > for > > > > > > > > > > > > > all > > > > > > > > > > > > > > > S3 storage backend, but will require special > > > parsing of > > > > > > > > > user/role > > > > > > > > > > > ARN > > > > > > > > > > > > > > field > > > > > > > > > > > > > > > to distinguish R2 from the other systems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm still leaning towards the separate > > > > > > > > > > R2StorageConfigurationInfo, > > > > > > > > > > > > but > > > > > > > > > > > > > > I'd > > > > > > > > > > > > > > > be ok with overloading user/role ARN too if > other > > > > > people > > > > > > > > prefer > > > > > > > > > > > > that. I > > > > > > > > > > > > > > > guess a key question in this context is whether > > R2 > > > has > > > > > > the > > > > > > > > > > concept > > > > > > > > > > > of > > > > > > > > > > > > > > role > > > > > > > > > > > > > > > ARN at all.... but TBH I have no idea about > > that. I > > > > > hope > > > > > > > > Austen > > > > > > > > > > > might > > > > > > > > > > > > > be > > > > > > > > > > > > > > > able to clarify that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S < > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Welcome, and thanks for starting with a > > > [DISCUSS]. > > > > > Best > > > > > > > way > > > > > > > > > to > > > > > > > > > > > > make a > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > contribution :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think the thread is talking about two > layers, > > > and > > > > > we > > > > > > > can > > > > > > > > > keep > > > > > > > > > > > > both > > > > > > > > > > > > > > > > answers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On the client side this should stay S3: s3:// > > > > > > locations, > > > > > > > > s3.* > > > > > > > > > > > > > > properties, > > > > > > > > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB / Iceberg > > Java > > > > > > results > > > > > > > > > > already > > > > > > > > > > > > show > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > unmodified clients work, so a dedicated > > R2FileIO > > > does > > > > > > not > > > > > > > > > seem > > > > > > > > > > > > worth > > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > On the server side I lean toward Dmitri: a > > > distinct > > > > > > > > > > > > > > > > R2StorageConfigurationInfo and storage > > > integration. > > > > > > > > Issuance > > > > > > > > > is > > > > > > > > > > > not > > > > > > > > > > > > > > STS, > > > > > > > > > > > > > > > > and folding Cloudflare fields into > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > mixes > > > > > > > > > > > > > > > > unrelated concepts. The middle ground to > avoid > > > is a > > > > > new > > > > > > > > type > > > > > > > > > > that > > > > > > > > > > > > is > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > AWS-shaped underneath. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One concrete caution if the Option B > prototype > > > stays > > > > > on > > > > > > > the > > > > > > > > > S3 > > > > > > > > > > > > > config: > > > > > > > > > > > > > > > > please do not overload stsUnavailable. Today > > that > > > > > flag > > > > > > > > skips > > > > > > > > > > > > > AssumeRole > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > disables vending. R2 wants the opposite. > vend, > > > just > > > > > not > > > > > > > via > > > > > > > > > > STS. > > > > > > > > > > > so > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > flag would change meaning for existing MinIO > / > > > NetApp > > > > > > > > > catalogs. > > > > > > > > > > > If > > > > > > > > > > > > we > > > > > > > > > > > > > > > stay > > > > > > > > > > > > > > > > on S3 config, we need an explicit vending > > > signal; a > > > > > > > > dedicated > > > > > > > > > > R2 > > > > > > > > > > > > type > > > > > > > > > > > > > > > > avoids that trap. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On config surface, jurisdiction can live in > > > endpoint > > > > > > > > > > (<account>. > > > > > > > > > > > > > > > > r2.cloudflarestorage.com vs > > <account>.eu.r2...). > > > > > > Account > > > > > > > > ID > > > > > > > > > is > > > > > > > > > > > > still > > > > > > > > > > > > > > > > needed > > > > > > > > > > > > > > > > for the JWT sub, so keeping an explicit > > > accountId and > > > > > > > > > deriving > > > > > > > > > > > > > audience > > > > > > > > > > > > > > > > from the endpoint seems enough. Parent tokens > > > should > > > > > > stay > > > > > > > > in > > > > > > > > > > > server > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > via storageName. > > > > > > > > > > > > > > > > Local signing fits the vending model. > > > Fail-closed on > > > > > > > > > > cross-bucket > > > > > > > > > > > > and > > > > > > > > > > > > > > > mixed > > > > > > > > > > > > > > > > read/write grants is the right call. Prefix > > > claims > > > > > > should > > > > > > > > > > follow > > > > > > > > > > > > the > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > path comparison Polaris already uses, so we > do > > > not > > > > > > mint a > > > > > > > > > > > > credential > > > > > > > > > > > > > > > wider > > > > > > > > > > > > > > > > than the grant. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator (#4486) > > > > > > separately > > > > > > > if > > > > > > > > > R2 > > > > > > > > > > > > needs > > > > > > > > > > > > > > it, > > > > > > > > > > > > > > > > then management API + Java config, then > > vending, > > > > > Python > > > > > > > CLI > > > > > > > > > on > > > > > > > > > > > its > > > > > > > > > > > > > own. > > > > > > > > > > > > > > > > Sketching MinIO / Backblaze against the same > > > fields > > > > > is > > > > > > a > > > > > > > > > useful > > > > > > > > > > > > > design > > > > > > > > > > > > > > > > check; a generic non-STS SPI can wait for a > > > second > > > > > > > backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri > > > Bourlatchkov < > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for starting this contribution! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > After briefly looking through your PR [1] I > > > think > > > > > > > > > > > > > > > > > adding R2StorageConfigInfo makes sense. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Even though "aws" in > > > AwsStorageConfigurationInfo is > > > > > > > > longer > > > > > > > > > > > > implies > > > > > > > > > > > > > > AWS > > > > > > > > > > > > > > > > > services, and MinIO, Rust, Ozone are > > supported > > > just > > > > > > as > > > > > > > > > well, > > > > > > > > > > R2 > > > > > > > > > > > > > > appears > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > be significantly different from STS-based > S3 > > > > > systems > > > > > > > > that a > > > > > > > > > > > > > separate > > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > > is probably the most natural way to support > > it > > > in > > > > > the > > > > > > > > > Polaris > > > > > > > > > > > > > > codebase. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The alternative, I guess, is to add R2 > > > accountId > > > > > and > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > optional properties to > > > AwsStorageConfigurationInfo > > > > > > (or > > > > > > > > > > overload > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > properties), but that looks like mixing > > > unrelated > > > > > > > > concepts > > > > > > > > > > > > > together. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Adding a generic property bag to > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > is > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > convenient because > > > > > > > PolarisStorageIntegrationProviderImpl > > > > > > > > > will > > > > > > > > > > > > > > probably > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > to know how to interpret it in order to > > > redirect > > > > > > > > credential > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > requests to R2-specific code. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wonder whether R2StorageConfigurationInfo > > and > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > (and corresponding "storage integration" > > > classes) > > > > > > could > > > > > > > > > > share a > > > > > > > > > > > > > > common > > > > > > > > > > > > > > > > base > > > > > > > > > > > > > > > > > class for common properties (e.g. > endpoint). > > > Even > > > > > > > though > > > > > > > > > > > > "endpoint" > > > > > > > > > > > > > > > might > > > > > > > > > > > > > > > > > not be relevant to R2 in the cloud, the > fact > > > that > > > > > > > > S3FileIO > > > > > > > > > > uses > > > > > > > > > > > > it, > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > > is accessed via S3FileIO means we should > > > probably > > > > > > allow > > > > > > > > > users > > > > > > > > > > > to > > > > > > > > > > > > > > > > configure > > > > > > > > > > > > > > > > > the full set of FileIO properties just in > > case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is fine for different > > > > > > > > PolarisStorageIntegration > > > > > > > > > > > > > > > > implementations > > > > > > > > > > > > > > > > > to produce the same kind of > > > StorageAccessConfig (S3 > > > > > > in > > > > > > > > this > > > > > > > > > > > case) > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > thus > > > > > > > > > > > > > > > > > cause S3FileIO to be used by clients > > (including > > > > > > > Polaris' > > > > > > > > > own > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > access > > > > > > > > > > > > > > > > > paths). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] look a > > bit > > > > > > > concerning > > > > > > > > > to > > > > > > > > > > > me. > > > > > > > > > > > > I > > > > > > > > > > > > > > > wonder > > > > > > > > > > > > > > > > > if this code could be refactored to avoid > > > > > > dependencies > > > > > > > on > > > > > > > > > > > storage > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > > types completely.... but TBH, I did not > look > > > too > > > > > > deeply > > > > > > > > > into > > > > > > > > > > > this > > > > > > > > > > > > > > > today. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anticipating future PRs for this in the > > Polaris > > > > > repo, > > > > > > > I'd > > > > > > > > > > like > > > > > > > > > > > to > > > > > > > > > > > > > ask > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > separate Python code changes from java > > changes > > > > > since > > > > > > > > these > > > > > > > > > > > areas > > > > > > > > > > > > > > > usually > > > > > > > > > > > > > > > > > attract different reviewers :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From my POV, management API and matching > java > > > > > changes > > > > > > > can > > > > > > > > > be > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > > PR. I'd say docs should come later (to keep > > PRs > > > > > > small). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > All in all, thank you again and I hope this > > > > > > > contribution > > > > > > > > > > lands > > > > > > > > > > > in > > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM > Jean-Baptiste > > > > > Onofré > > > > > > < > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and > > welcome > > > > > > aboard > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it makes sense to have R2 built > > > under the > > > > > > > > > existing > > > > > > > > > > S3 > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > configuration. It matches how Polaris is > > > already > > > > > > > > > > structured: > > > > > > > > > > > > > > > > > > StorageType is keyed on the URI scheme > > > (s3://, > > > > > > > s3a://), > > > > > > > > > and > > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo already > carries > > > > > > endpoint, > > > > > > > > > > > > > stsEndpoint, > > > > > > > > > > > > > > > > > > pathStyleAccess and, importantly, an > > > > > stsUnavailable > > > > > > > > flag. > > > > > > > > > > > > > > > > > > So, what you need is largely there > already: > > > > > correct > > > > > > > me > > > > > > > > if > > > > > > > > > > I'm > > > > > > > > > > > > > > wrong, > > > > > > > > > > > > > > > > > > R2 is essentially "S3 compatible store > > where > > > STS > > > > > is > > > > > > > > > > > > unavailable, > > > > > > > > > > > > > > plus > > > > > > > > > > > > > > > > > > a way to actually vend credentials in > that > > > case". > > > > > > > Going > > > > > > > > > > this > > > > > > > > > > > > > route > > > > > > > > > > > > > > > > > > also keeps this work "isolated" from the > > > > > > > > > > > > FileIO-per-storage-type > > > > > > > > > > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The interesting design question is a > > > pluggable > > > > > > > > credential > > > > > > > > > > > > vending > > > > > > > > > > > > > > > path > > > > > > > > > > > > > > > > > > for S3 compatible stores when STS is not > > > > > available, > > > > > > > > with > > > > > > > > > > > > > > Cloudflare's > > > > > > > > > > > > > > > > > > local signing as the first concrete > > > > > implementation, > > > > > > > > > rather > > > > > > > > > > > than > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > > > specific code. It would be good to sketch > > how > > > > > MinIO > > > > > > > > would > > > > > > > > > > > slot > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > the same config fields before the vending > > > path > > > > > > > hardens > > > > > > > > > > around > > > > > > > > > > > > R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I believe we have to check if Cloudflare > > > account > > > > > ID > > > > > > > and > > > > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > > > > > > really need dedicated config fields, or > can > > > we > > > > > use > > > > > > > the > > > > > > > > > > > endpoint > > > > > > > > > > > > > > value > > > > > > > > > > > > > > > > > > (with documentation)? Keeping the > > > configuration > > > > > > > surface > > > > > > > > > > > minimal > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > help the design generalize. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would love to discuss two things: > > > > > > > > > > > > > > > > > > 1. Polaris shipping a Cloudflare API > token > > > and > > > > > > > locally > > > > > > > > > > > minting > > > > > > > > > > > > > > scoped > > > > > > > > > > > > > > > > > > credentials makes it effectively an STS > for > > > R2, > > > > > > which > > > > > > > > > > > increases > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > blast radius of a server compromise. Your > > > "reject > > > > > > > > rather > > > > > > > > > > than > > > > > > > > > > > > > > > boarden" > > > > > > > > > > > > > > > > > > handling of unrepresentable scopes is the > > > right > > > > > > call. > > > > > > > > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java results are > > > > > promising. > > > > > > > > Just > > > > > > > > > > > > curious > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > Spark or Trino. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I would > > > suggest > > > > > to > > > > > > > > start > > > > > > > > > > with > > > > > > > > > > > > the > > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > > > > > config and the management API as first > > > shoot, and > > > > > > we > > > > > > > > can > > > > > > > > > > > follow > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > the credential vending implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen > Tomek > > > > > > > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago Trading > > > Company. > > > > > We > > > > > > > use > > > > > > > > > > > Polaris > > > > > > > > > > > > > for > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > Iceberg catalogs and we're evaluating > > > Cloudflare > > > > > R2 > > > > > > > as > > > > > > > > an > > > > > > > > > > > > object > > > > > > > > > > > > > > > store > > > > > > > > > > > > > > > > > (the > > > > > > > > > > > > > > > > > > no egress fees make it a very compelling > > > > > product). > > > > > > > I’d > > > > > > > > > like > > > > > > > > > > > to > > > > > > > > > > > > > > > discuss > > > > > > > > > > > > > > > > > > contributing R2 support and credential > > > vending to > > > > > > > > Polaris > > > > > > > > > > > with > > > > > > > > > > > > > the > > > > > > > > > > > > > > > goal > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > getting feedback before opening a > > > > > ready-for-review > > > > > > > > > upstream > > > > > > > > > > > PR. > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > my > > > > > > > > > > > > > > > > > > first contribution, so trying to do this > > > right. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it > does > > > not > > > > > > > provide > > > > > > > > > AWS > > > > > > > > > > > > STS. > > > > > > > > > > > > > > > > > > Cloudflare instead supports these > > > short-lived, > > > > > > scoped > > > > > > > > > > > > credentials > > > > > > > > > > > > > > > > > generated > > > > > > > > > > > > > > > > > > by locally signed JWT with the server > > > holding the > > > > > > > > parent > > > > > > > > > > API > > > > > > > > > > > > > token > > > > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > > This allows us to retain Polaris’s > > > authorization > > > > > > and > > > > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > > > model while storing our table data in R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have a prototype with the following > > > design: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > An opt-in R2 storage type, with catalog > > > > > > > configuration > > > > > > > > > > > > > identifying > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > Cloudflare account and optional > > jurisdiction. > > > > > > Parent > > > > > > > > > > > > credentials > > > > > > > > > > > > > > > remain > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > server configuration and can be selected > > > through > > > > > > > > > > storageName. > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > Local credential generation using > > > Cloudflare’s > > > > > > > > > documented > > > > > > > > > > > > > format. > > > > > > > > > > > > > > > > > > Credentials expire, are restricted to one > > > bucket, > > > > > > and > > > > > > > > > carry > > > > > > > > > > > > > prefix > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > access scopes derived from Polaris’s > > location > > > > > > grants. > > > > > > > > The > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > uses > > > > > > > > > > > > > > > > > > Polaris’s existing credential cache. > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > Credentials returned through the > existing > > > > > Iceberg > > > > > > > > REST > > > > > > > > > > > > vending > > > > > > > > > > > > > > flow > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > s3.* properties, including the endpoint, > > > session > > > > > > > token, > > > > > > > > > and > > > > > > > > > > > > > expiry. > > > > > > > > > > > > > > > > > > Compatible clients use their existing S3 > > > FileIO. > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > Conservative handling of scopes the > > > > > > implementation > > > > > > > > > cannot > > > > > > > > > > > > > > > represent: > > > > > > > > > > > > > > > > > > cross-bucket and mixed read/write grants > > are > > > > > > rejected > > > > > > > > > > rather > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > broadened. R2 is excluded from the > default > > > > > > > > > > > > > supported-storage-types > > > > > > > > > > > > > > > > list. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The main design questions I feel this > > runs > > > > > into: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > Should this be a distinct storage type > > or a > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > > mechanism > > > > > > > > > > > > > > > > > > selected through the existing S3 > > > configuration? > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > I chose a separate type because the > > > identity, > > > > > > > > > credential > > > > > > > > > > > > > > > generation, > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > scoping model differ from AWS STS, as > this > > > seemed > > > > > > > more > > > > > > > > > > > > > appropriate > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is a draft preview PR against my > > fork > > > > > [2], > > > > > > > > > > including > > > > > > > > > > > > > > > > > > implementation, management API changes, > CLI > > > > > > support, > > > > > > > > > tests, > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > documentation. It also discloses the AI > > > > > assistance > > > > > > > used > > > > > > > > > > > during > > > > > > > > > > > > > > > > > > implementation. I leaned heavily on Fable > > > 5.1 for > > > > > > > this > > > > > > > > > > > > > > implementation > > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > want to be upfront about it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We are running the implementation in a > > > > > > > non-production > > > > > > > > > > > > > environment > > > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > > now as a POC. Validation includes live R2 > > > > > > > reads/writes, > > > > > > > > > > > > multipart > > > > > > > > > > > > > > > > > > operations, negative scope-boundary > tests, > > > our > > > > > > > internal > > > > > > > > > > > Python > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > > regression matrix, and credential > > > > > rotation/recovery > > > > > > > > > > > exercises. > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > upstream > > > > > > > > > > > > > > > > > > CI workflow has passed on the fork. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The change also encounters the existing > > > > > > assumption > > > > > > > > > that a > > > > > > > > > > > > > FileIO > > > > > > > > > > > > > > > > > > implementation identifies exactly one > > storage > > > > > type: > > > > > > > > both > > > > > > > > > S3 > > > > > > > > > > > and > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > > S3FileIO. This overlaps with #4486 [3]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback > on: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > > Should R2 have its own storage > > > > > > configuration/type, > > > > > > > or > > > > > > > > > > > should > > > > > > > > > > > > > > > non-STS > > > > > > > > > > > > > > > > > > vending be modeled within the existing S3 > > > > > > > integration? > > > > > > > > > > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > > Does the local-signing approach fit > > > Polaris’s > > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > model? > > > > > > > > > > > > > > > > > > Any additional constraints or validations > > > > > > necessary? > > > > > > > > > > > > > > > > > > > 3. > > > > > > > > > > > > > > > > > > > Would you prefer the shared FileIO > > > validation > > > > > > > change > > > > > > > > to > > > > > > > > > > > land > > > > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > > > > and how should we sequence the management > > > API and > > > > > > R2 > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > changes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Happy to adapt the design and split > the > > > > > > > > contribution > > > > > > > > > > into > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > focused > > > > > > > > > > > > > > > > > > PRs. Wanted to start the discussion > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > > > > > > > > > > > > > > > > > > [2] > > > > > > > https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > > [3] > > > > > > > https://github.com/apache/polaris/issues/4486 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > > > > > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any > > > attached > > > > > > files > > > > > > > > > > contain > > > > > > > > > > > > > > > > information > > > > > > > > > > > > > > > > > > intended for the exclusive use of the > > > individual > > > > > or > > > > > > > > > entity > > > > > > > > > > to > > > > > > > > > > > > > whom > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > addressed and may contain information > that > > is > > > > > > > > > proprietary, > > > > > > > > > > > > > > > confidential > > > > > > > > > > > > > > > > > > and/or exempt from disclosure under > > > applicable > > > > > law. > > > > > > > If > > > > > > > > > you > > > > > > > > > > > are > > > > > > > > > > > > > not > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > intended recipient, you are hereby > notified > > > that > > > > > > any > > > > > > > > > > viewing, > > > > > > > > > > > > > > > copying, > > > > > > > > > > > > > > > > > > disclosure or distribution of this > > > information > > > > > may > > > > > > be > > > > > > > > > > subject > > > > > > > > > > > > to > > > > > > > > > > > > > > > legal > > > > > > > > > > > > > > > > > > restriction or sanction. Please notify > the > > > > > sender, > > > > > > by > > > > > > > > > > > > electronic > > > > > > > > > > > > > > mail > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > telephone, of any unintended recipients > and > > > > > delete > > > > > > > the > > > > > > > > > > > original > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached > > files > > > > > > contain > > > > > > > > > > > > information > > > > > > > > > > > > > > > intended for the exclusive use of the > individual > > or > > > > > > entity > > > > > > > to > > > > > > > > > > whom > > > > > > > > > > > it > > > > > > > > > > > > > is > > > > > > > > > > > > > > > addressed and may contain information that is > > > > > > proprietary, > > > > > > > > > > > > confidential > > > > > > > > > > > > > > > and/or exempt from disclosure under applicable > > > law. If > > > > > > you > > > > > > > > are > > > > > > > > > > not > > > > > > > > > > > > the > > > > > > > > > > > > > > > intended recipient, you are hereby notified > that > > > any > > > > > > > viewing, > > > > > > > > > > > > copying, > > > > > > > > > > > > > > > disclosure or distribution of this information > > may > > > be > > > > > > > subject > > > > > > > > > to > > > > > > > > > > > > legal > > > > > > > > > > > > > > > restriction or sanction. Please notify the > > sender, > > > by > > > > > > > > > electronic > > > > > > > > > > > mail > > > > > > > > > > > > > or > > > > > > > > > > > > > > > telephone, of any unintended recipients and > > delete > > > the > > > > > > > > original > > > > > > > > > > > > message > > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > > > contain > > > > > > > > > information > > > > > > > > > > > > intended for the exclusive use of the individual or > > > entity to > > > > > > > whom > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > > addressed and may contain information that is > > > proprietary, > > > > > > > > > confidential > > > > > > > > > > > > and/or exempt from disclosure under applicable law. > If > > > you > > > > > are > > > > > > > not > > > > > > > > > the > > > > > > > > > > > > intended recipient, you are hereby notified that any > > > viewing, > > > > > > > > > copying, > > > > > > > > > > > > disclosure or distribution of this information may be > > > subject > > > > > > to > > > > > > > > > legal > > > > > > > > > > > > restriction or sanction. Please notify the sender, by > > > > > > electronic > > > > > > > > mail > > > > > > > > > > or > > > > > > > > > > > > telephone, of any unintended recipients and delete > the > > > > > original > > > > > > > > > message > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > > contain > > > > > > > > information > > > > > > > > > > > intended for the exclusive use of the individual or > > entity > > > to > > > > > > whom > > > > > > > it > > > > > > > > > is > > > > > > > > > > > addressed and may contain information that is > > proprietary, > > > > > > > > confidential > > > > > > > > > > > and/or exempt from disclosure under applicable law. If > > you > > > are > > > > > > not > > > > > > > > the > > > > > > > > > > > intended recipient, you are hereby notified that any > > > viewing, > > > > > > > > copying, > > > > > > > > > > > disclosure or distribution of this information may be > > > subject > > > > > to > > > > > > > > legal > > > > > > > > > > > restriction or sanction. Please notify the sender, by > > > > > electronic > > > > > > > mail > > > > > > > > > or > > > > > > > > > > > telephone, of any unintended recipients and delete the > > > original > > > > > > > > message > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > > > > > information > > > > > > > > > intended for the exclusive use of the individual or entity > to > > > whom > > > > > it > > > > > > > is > > > > > > > > > addressed and may contain information that is proprietary, > > > > > > confidential > > > > > > > > > and/or exempt from disclosure under applicable law. If you > > are > > > not > > > > > > the > > > > > > > > > intended recipient, you are hereby notified that any > viewing, > > > > > > copying, > > > > > > > > > disclosure or distribution of this information may be > subject > > > to > > > > > > legal > > > > > > > > > restriction or sanction. Please notify the sender, by > > > electronic > > > > > mail > > > > > > > or > > > > > > > > > telephone, of any unintended recipients and delete the > > original > > > > > > message > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > intended for the exclusive use of the individual or entity to whom it is > addressed and may contain information that is proprietary, confidential > and/or exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any viewing, copying, > disclosure or distribution of this information may be subject to legal > restriction or sanction. Please notify the sender, by electronic mail or > telephone, of any unintended recipients and delete the original message > without making any copies. >
