Thanks for the discussion today. My understanding of where we landed was to keep credentialVendingMechanism a string and have the spec define the values Apache Polaris accepts and what they mean. In this PR that is just STS and the empty value still signifies the server default. In PR 2 I’ll be adding CLOUDFLARE_R2 to the same list. The server already refuses any other value with a 400, so it seems like this is just a documentation change. Let me know if I am missing anything.
Thanks, Austen Get Outlook for Mac<https://aka.ms/GetOutlookForMac> From: Dmitri Bourlatchkov <[email protected]> Date: Wednesday, September 30, 2026 at 12:31 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential vending Hi Yufei, Thanks for providing the details! Replying inline. > When a client creates a catalog with credentialVendingMechanism: "m1", > what is the expected behavior? This depends on the meaning of "m1" and the specific server build. In Apache Polaris "m1" is not meaningful and is not supported. The server will reject this value on the corresponding catalog update request. This is handled in PolarisAdminService in the current PR state. > 1. The server supports it: This is the ideal scenario (happy path). Indeed. > 2. The server does not support it: This creates inconsistency if a client > that works with Server A fails on Server B. In my view this is not an inconsistency as different servers do not have to provide the same capabilities. Creating a catalog with a specific credentialVendingMechanism is an administrative task. The admin user must be aware of what the server supports. This is a deployment / configuration concern. Not a runtime compatibility concern. > 3. The server supports "m1" differently: This is the worst-case scenario, > where the client's expectations conflict with the server's actual behavior. This is a good point! My take is that if "m1" is defined in Apache Polaris, then all server builds must implement it the same way. The definition will be expressed by means of S3CredentialVendingMechanism classes included in standard Apache Polaris releases (and hopefully docs too). Currently only "STS" and "DEFAULT" (falling back on STS) are defined now. If "m1" is not defined in Apache Polaris, custom builds are free to assign any meaning to it and do not have to be mutually compatible. This is the main point for extensibility. Note that custom builds are not "Apache Polaris" anymore and cannot claim to be the "same" as Apache Polaris. This is a matter of ASF trademarks. Normally, custom builds say something like "Powered by Apache Polaris". Again, the admin user setting "m1" as a vending mechanism in a particular server must be aware of what that server provides and understand the meaning of "m1" in that context. The onus to describe the meaning of "m1" to users in this case rests with the custom server developers. We can talk about preventing name overlaps, but I do not think establishing any kind of name registry is worth the trouble. In this day and age, I believe all downstream project developers are aware of name overlap issues. and will follow normal practices to avoid name collisions. I hope this clears your concerns. Cheers, Dmitri. On Wed, Sep 30, 2026 at 1:02 PM Yufei Gu <[email protected]> wrote: > Hi Dmitri, > > When a client creates a catalog with credentialVendingMechanism: "m1", what > is the expected behavior? > > Here is a suitaion if we allow a freeform string, the client might > encounter three different scenarios depending on the server: > 1. The server supports it: This is the ideal scenario (happy path). > 2. The server does not support it: This creates inconsistency if a client > that works with Server A fails on Server B. > 3. The server supports "m1" differently: This is the worst-case scenario, > where the client's expectations conflict with the server's actual behavior. > > I don't think this is the situation we want users to run into. The fix is > quite simple, using a specified values instead of a freefrom string. > > Yufei > > > On Tue, Sep 29, 2026 at 9:48 PM Jean-Baptiste Onofré <[email protected]> > wrote: > > > Hi all, > > > > Since concerns were raised, it's important to take a moment to align > > rather than rushing. That's the way we drive consensus :) > > > > On the technical question of string vs enum for > > credentialVendingMechanism, here is my take: > > > > 1. This field is part of the Management API for catalog > > administrators. Iceberg REST clients (Spark, Trino, ...) never see or > > interact with credentialVendingMechanism. They only consume standard > > vended credentials (s3.* session tokens/keys) during table > > load/commit. There is no risk of breaking query engine clients across > > deployments. > > 2. One of our primary goals when discussing S3-compatible stores (R2, > > MinIO, ...) was to keep credential vending pluggable via SPI rather > > than hardcoding vendor-specific paths in core. If we use a closed enum > > in the OpenAPI spec, it means: > > * any new mechanism (including R2 in Phase 2 or MinIO) requires a spec > > change. > > * any downstream deployment or plugin providing a custom mechanism > > would break OpenAPI-generated client SDKs due to strict enum > > validation. > > * runtime validation on the server (returning HTTP 400 with a clear > > message) combined with the realm allowlist provides the right > > server-side guardrail without restricting extensibility. > > 3. Yufei's concerns about a clear contract and avoiding confusion is > > valid. We can address contract clarity without closing the enum: > > * we keep type: string, but document the well-known/standard values > > (STS, DEFAULT) explicitly in the description and examples in the > > OpenAPI spec (as PR #5513 does). > > * we ensure the server validates cleanly with case-insensitivity (or > > canonical normalization) so administrators get immediate, actionable > > feedback on typos. > > > > PR #5513 does a solid job laying this foundation so Austen can move > > forward with Phase 2. If we keep it as a string with well-documented > > standard values and server-siide validation, I think it's the right > > balance between spec clarify and pluggability. > > > > Thoughts? > > > > Regards > > JB > > > > On Wed, Sep 30, 2026 at 12:50 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > > Hi Yufei, > > > > > > > > Conversely, an enum will require defining all possible vending > > > mechanisms in > > > > > the Polaris API spec. > > > > > > > Yes. Otherwise, a client may failed in certain deployments. > > > > > > Could you describe this failure mode in terms of API request/response > > flow? > > > > > > Thanks, > > > Dmitri. > > > > > > On Tue, Sep 29, 2026 at 2:06 PM Yufei Gu <[email protected]> wrote: > > > > > > > > Conversely, an enum will require defining all possible vending > > > > mechanisms in > > > > the Polaris API spec. > > > > > > > > Yes. Otherwise, a client may failed in certain deployments. > > > > > > > > > This will prevent downstream projects from using custom vending > > > > mechanisms. > > > > > > > > No. if downstream needs custom vending mechanisms, they can always > > build > > > > their own Polaris. > > > > > > > > Yufei > > > > > > > > > > > > On Tue, Sep 29, 2026 at 11:01 AM Dmitri Bourlatchkov < > [email protected] > > > > > > > wrote: > > > > > > > > > Hi Yufei, > > > > > > > > > > > Using an enum will clarify the REST spec contract. > > > > > > > > > > I'm not sure I understand your point. Current PR clearly describes > > the > > > > > meaning of "credentialVendingMechanism" and lists options available > > > > > out-of-the-box. > > > > > > > > > > What does an enum achieve, that (current) runtime validation does > > not? > > > > > > > > > > Conversely, an enum will require defining all possible vending > > mechanisms > > > > > in the Polaris API spec. This will prevent downstream projects from > > using > > > > > custom vending mechanisms. > > > > > > > > > > I believe Polaris as a project currently promotes downstream > > > > > customizability. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Sep 29, 2026 at 1:21 PM Yufei Gu <[email protected]> > > wrote: > > > > > > > > > > > I don't think we have consensus on the PR, esp. a client should > NOT > > > > > depend > > > > > > on a specific deployment. Using an enum will clarify the REST > spec > > > > > > contract. > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 7:46 AM Dmitri Bourlatchkov < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > PR [5513] has been in review for quite some time. I believe we > > have > > > > > > general > > > > > > > consensus on the Management API changes and java code changes. > > > > > > > > > > > > > > The only point that may still have divergent opinions is > whether > > to > > > > use > > > > > > > free text or enum for "credentialVendingMechanism". I > personally > > > > > support > > > > > > > free text (rationale explained in previous comments). This > > > > conversation > > > > > > has > > > > > > > been dormant for a few days, so I assume lazy consensus on the > > > > current > > > > > > > state of the PR (free text). In any case switching from free > > text to > > > > > enum > > > > > > > is a non-breaking API change before the next release in case > the > > > > > > consensus > > > > > > > shifts and we need to adjust. We have about one month for > > followup > > > > API > > > > > > > changes. > > > > > > > > > > > > > > The java code introduced in this PR is not rigid and can change > > at > > > > any > > > > > > time > > > > > > > per our evolution guidelines [1]. > > > > > > > > > > > > > > I propose merging PR [5513] on Sept 30 unless fresh blocking > > concerns > > > > > are > > > > > > > raised. > > > > > > > > > > > > > > [1] https://polaris.apache.org/releases/1.8.0/evolution/ > > > > > > > > > > > > > > [5513] https://github.com/apache/polaris/pull/5513 > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 9:30 AM Austen Tomek < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hey all, > > > > > > > > > > > > > > > > Any chance I can get further reviews this week on the PR? I’d > > like > > > > to > > > > > > get > > > > > > > > moving onto Phase 2 here if there are no further issues to > > address. > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > Date: Thursday, September 24, 2026 at 17:33 > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped > > credential > > > > > > > vending > > > > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > > > > The REST spec is a contract between Polaris servers and > > clients > > > > > [...] > > > > > > > > > > > > > > > > I tend to think that this statement is a bit too strong. The > > > > > Management > > > > > > > API > > > > > > > > spec defines a protocol for clients and servers to > communicate. > > > > > > > > > > > > > > > > > [...] This would help avoid a situation where a Polaris > > client > > > > > works > > > > > > > > > only with certain deployments. > > > > > > > > > > > > > > > > Since Polaris has been developed as a customizable system, a > > REST > > > > > API > > > > > > > > specification defined by Apache Polaris cannot control the > > > > behaviour > > > > > of > > > > > > > all > > > > > > > > possible downstream environments. The purpose of the spec is > to > > > > > ensure > > > > > > > > protocol compatibility. > > > > > > > > > > > > > > > > Note that the value of `credentialVendingMechanism` has no > > bearing > > > > on > > > > > > > > any logic clients need to execute. It is an opaque parameter > > > > > controlled > > > > > > > > by the Polaris administrator. > > > > > > > > > > > > > > > > > The new credentialVendingMechanism field in the REST spec > > looks > > > > > > > > > reasonable. My only suggestion is to use an enum instead > of a > > > > > string. > > > > > > > > [...] > > > > > > > > > > > > > > > > The above points are exactly the reason why the Management > API > > > > cannot > > > > > > > > enumerate all possible vending mechanisms. > > > > > > > > > > > > > > > > The same applies to storage types too. However, we do not > have > > any > > > > > > > > practical > > > > > > > > examples of 3rd party storage systems that go beyond > > S3/GCS/ADLS > > > > > (yet). > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Austen. It's fair that endpoints are not reliable > > given > > > > that > > > > > > > some > > > > > > > > > s3-compatible can use arbitrary hostnames. > > > > > > > > > I also think stsUnavailable is fine to keep as-is. I don’t > > think > > > > we > > > > > > > need > > > > > > > > to > > > > > > > > > change its scope or repurpose it. > > > > > > > > > > > > > > > > > > The new credentialVendingMechanism field in the REST spec > > looks > > > > > > > > reasonable. > > > > > > > > > My only suggestion is to use an enum instead of a string. > The > > > > REST > > > > > > spec > > > > > > > > is > > > > > > > > > a contract between Polaris servers and clients, so I’d > > expect it > > > > to > > > > > > > > define > > > > > > > > > the valid values and their meanings rather than leaving > them > > up > > > > to > > > > > > each > > > > > > > > > deployment. This would help avoid a situation where a > Polaris > > > > > client > > > > > > > > works > > > > > > > > > only with certain deployments. Deployment-specific > mechanisms > > > > > > shouldn’t > > > > > > > > be > > > > > > > > > part of the standard REST contract; users who need a custom > > > > > mechanism > > > > > > > can > > > > > > > > > always extend the server in their own deployment. > > > > > > > > > > > > > > > > > > > Either way the operator needs a per-realm switch like > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control which > > realm > > > > > can > > > > > > > > mint > > > > > > > > > R2 credentials from the operators Cloudflare account. > > > > > > > > > > > > > > > > > > Could you clarify why we need a separate per-realm switch? > > Once > > > > the > > > > > > > > storage > > > > > > > > > configuration is set at the catalog level to select a > > specific > > > > > > > mechanism, > > > > > > > > > the catalog will have no choice but to support it. Do we > need > > > > > another > > > > > > > > layer > > > > > > > > > of gating? > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > Hey Yufei, > > > > > > > > > > > > > > > > > > > > Apologies for missing your question. > > > > > > > > > > > > > > > > > > > > I don’t think Endpoint is sufficient. > > > > > > > > > > > > > > > > > > > > Endpoint matching only works where the hostname > identifies > > the > > > > > > vendor > > > > > > > > > like > > > > > > > > > > R2’s public hostnames that are recognizable, MinIO and > > other > > > > > > > > self-hosted > > > > > > > > > > S3-compatible stores use arbitrary hostnames so there > > would be > > > > > > > nothing > > > > > > > > to > > > > > > > > > > match on. On top of that, if a match fails it would just > > > > silently > > > > > > > fall > > > > > > > > > back > > > > > > > > > > to STS. > > > > > > > > > > > > > > > > > > > > Right now the endpoint just says how you’ll connect not > > how the > > > > > > > > > > credentials are vended. There's endpoint and also > > > > > endpointInternal > > > > > > > for > > > > > > > > > > server-side access, so it would be unclear which of the > two > > > > > > decides. > > > > > > > > > > > > > > > > > > > > For your other question, there is no second mechanism for > > R2 > > > > > today, > > > > > > > but > > > > > > > > > > for other self-hosted stores the same endpoint could be > > served > > > > by > > > > > > STS > > > > > > > > or > > > > > > > > > by > > > > > > > > > > a different token mechanism chosen per catalog. > > > > > > > > > > > > > > > > > > > > stsUnavailable keeps it’s current behavior in the PR > (that > > was > > > > > > > > Prithvi’s > > > > > > > > > > point earlier in the thread) > > > > > > > > > > > > > > > > > > > > Either way the operator needs a per-realm switch like > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control > which > > > > realm > > > > > > can > > > > > > > > > mint > > > > > > > > > > R2 credentials from the operators Cloudflare account. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > From: Yufei Gu <[email protected]> > > > > > > > > > > Date: Monday, September 21, 2026 at 17:53 > > > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped > > > > > credential > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > Thanks for the update. After looking at PR #5513, I think > > we > > > > > should > > > > > > > > first > > > > > > > > > > clarify whether we need a new credentialVendingMechanism > > field > > > > in > > > > > > the > > > > > > > > > > spec/config. > > > > > > > > > > > > > > > > > > > > Austen, you mentioned that the R2 account ID and > > jurisdiction > > > > > > already > > > > > > > > > come > > > > > > > > > > from the endpoint. Could we also use the existing field > > like > > > > > > > > `endpoint`, > > > > > > > > > to > > > > > > > > > > recognize R2 and select its credential-vending mechanism, > > while > > > > > > > keeping > > > > > > > > > the > > > > > > > > > > current stsUnavailable behavior unchanged? > > > > > > > > > > > > > > > > > > > > Are there cases where the endpoint isn’t enough, or where > > users > > > > > > need > > > > > > > to > > > > > > > > > > choose different vending mechanisms for the same > > endpoint(e.g., > > > > > R2) > > > > > > > per > > > > > > > > > > catalog? > > > > > > > > > > > > > > > > > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > Hi Yufei, Dimitri, > > > > > > > > > > > > > > > > > > > > > > Thanks, both. PR 1 is ready for review: > > > > > > > > > > > https://github.com/apache/polaris/pull/5513 > > > > > > > > > > > > > > > > > > > > > > I pushed a revision last night that takes in Dimitri’s > > > > > comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > I changed from issuer to credentialVendingMechanism. > It’s > > > > also > > > > > > now > > > > > > > a > > > > > > > > > > plain > > > > > > > > > > > string and the realm allowlist is > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which still > > > > > defaults > > > > > > to > > > > > > > > > [STS] > > > > > > > > > > > * > > > > > > > > > > > Mechanisms are CDI beans found by their identifier, as > > > > Dimitri > > > > > > > > > proposed. > > > > > > > > > > > * > > > > > > > > > > > The CLOUDFLARE_R2 bean and the R2 vending code will > > follow in > > > > > the > > > > > > > > next > > > > > > > > > PR > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > CI workflow is waiting approval to run. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > > > Date: Tuesday, September 15, 2026 at 20:00 > > > > > > > > > > > To: [email protected] <[email protected]> > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with > scoped > > > > > > credential > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > Having getCloudflareR2Endpoint on > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > > > > > > Yes, this does feel awkward. > > > > > > > > > > > > > > > > > > > > > > From my POV pursuing the option with the > > "credentialIssuers" > > > > > > > property > > > > > > > > > > looks > > > > > > > > > > > promising. This property is fairly generic and fits > > naturally > > > > > > into > > > > > > > > the > > > > > > > > > > > config, given that we already target multiple backends. > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, Dmitri, > > > > > > > > > > > > > > > > > > > > > > > > > Having getCloudflareR2Endpoint on > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > > > > > > > > > That does feel awkward. However, > > > > AwsStorageConfigurationInfo > > > > > > has > > > > > > > > > > > supported > > > > > > > > > > > > S3-compatible backends beyond AWS for a while, so > > there is > > > > > > > > precedent > > > > > > > > > > > (e.g., > > > > > > > > > > > > MinIO, etc.). The underlying mismatch is that we call > > the > > > > > > storage > > > > > > > > > type > > > > > > > > > > > S3, > > > > > > > > > > > > but its configuration mixes shared S3 settings with > > > > > > AWS-specific > > > > > > > > > ones: > > > > > > > > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo → storage type: S3 > > > > > > > > > > > > ├── Common S3 settings > > > > > > > > > > > > │ endpoint, region, pathStyleAccess, > > allowedLocations > > > > > > > > > > > > └── AWS-specific settings > > > > > > > > > > > > roleArn, externalId, STS, KMS > > > > > > > > > > > > > > > > > > > > > > > > I’m open to separating the shared S3 settings from > > > > > > > > provider-specific > > > > > > > > > > > > configuration if that helps, but I don’t think we > need > > to > > > > > > settle > > > > > > > > on a > > > > > > > > > > > > broader refactor now. > > > > > > > > > > > > > > > > > > > > > > > > Let me know when the PR is ready for review. I’d be > > happy > > > > to > > > > > > > take a > > > > > > > > > > > closer > > > > > > > > > > > > look. > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri Bourlatchkov < > > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > +1 to adding "credentialIssues" or similar > (renamed) > > > > > property > > > > > > > to > > > > > > > > S3 > > > > > > > > > > > > config. > > > > > > > > > > > > > In fact, I was looking exactly for something like > > that > > > > last > > > > > > > week > > > > > > > > > but > > > > > > > > > > > the > > > > > > > > > > > > > idea did not quite condense in my mind :) > > > > > > > > > > > > > > > > > > > > > > > > > > I posted some preliminary comments on the draft PR. > > > > > > > > > > > > > > > > > > > > > > > > > > Just for community awareness, I'd like to propose > > here > > > > that > > > > > > we > > > > > > > > use > > > > > > > > > > CDI > > > > > > > > > > > > for > > > > > > > > > > > > > finding actual "issuer" / "mechanism" > > implementations. > > > > The > > > > > > > > codebase > > > > > > > > > > has > > > > > > > > > > > > > many examples of this. The closest match in this > > case is > > > > > > > > > > > > > probably PolarisStorageIntegrationProvider. > > > > > > > > > > > > > > > > > > > > > > > > > > This way the API does not have to enumerate all > > possible > > > > > > > values. > > > > > > > > > > > > Downstream > > > > > > > > > > > > > builds are free to add different implementations > > without > > > > > > having > > > > > > > > to > > > > > > > > > > > change > > > > > > > > > > > > > the API spec. The "allowed" list still makes sense. > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I like the direction this feature is > > taking, > > > > > > > > although I > > > > > > > > > > > > expect > > > > > > > > > > > > > PR reviews may take a few rounds :) > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek < > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Closing the loop since my last note and answering > > the > > > > > > > questions > > > > > > > > > > that > > > > > > > > > > > > have > > > > > > > > > > > > > > come up. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t have > > > > AWS-style > > > > > > IAM > > > > > > > > > roles > > > > > > > > > > > or > > > > > > > > > > > > > > ARNs. Our implementation signs temporary > > credentials > > > > > with a > > > > > > > > > parent > > > > > > > > > > > > > token’s > > > > > > > > > > > > > > secret, scoped to a bucket and a set of prefixes. > > The > > > > > > account > > > > > > > > ID > > > > > > > > > > and > > > > > > > > > > > > > > jurisdiction come from the R2 endpoint. Putting > the > > > > > account > > > > > > > ID > > > > > > > > in > > > > > > > > > > > > roleArn > > > > > > > > > > > > > > or userArn would give those fields a different > > meaning, > > > > > so > > > > > > > I’ve > > > > > > > > > > left > > > > > > > > > > > > them > > > > > > > > > > > > > > absent for CLOUDFLARE_R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I pursued Option B. The S3 configuration gains > one > > new > > > > > > field, > > > > > > > > > > > > > > credentialIssuer, with values STS > > > > > > > > > > > > > > (default) and CLOUDFLARE_R2 and a realm allowlist > > > > > > > > > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults > > > > > > > > > > > > > > to [STS]. I left stsUnavailable untouched > > (Prithvi) so > > > > it > > > > > > > still > > > > > > > > > > means > > > > > > > > > > > > > > "vend nothing". The > > > > > > > > > > > > > > credentialIssuer is the explicit sign. The R2 > > config > > > > > > rejects > > > > > > > > > > > > > > stsUnavailable along with a host of other STS and > > KMS > > > > > > fields. > > > > > > > > I'm > > > > > > > > > > > > running > > > > > > > > > > > > > > this in our non-production environment at the > > moment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Draft PR for stage 1 (config field, API, > allowlist, > > > > > gates): > > > > > > > > > > > > > > https://github.com/apache/polaris/pull/5513. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still see the appeal of a separate R2 > > configuration, > > > > as > > > > > > > > Dmitri > > > > > > > > > > > > > > suggested. My original reason for separating it > > was to > > > > > keep > > > > > > > the > > > > > > > > > > > > > R2-specific > > > > > > > > > > > > > > configuration and validation together. Having > > > > > > > > > > getCloudflareR2Endpoint > > > > > > > > > > > > on > > > > > > > > > > > > > > AwsStorageConfigurationInfo is one part I’d still > > like > > > > > > > feedback > > > > > > > > > on. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wanted to put a working version of Option B in > > front > > > > of > > > > > > > > > everyone > > > > > > > > > > so > > > > > > > > > > > > we > > > > > > > > > > > > > > could discuss the actual changes. In my earlier > > > > > > > implementation, > > > > > > > > > the > > > > > > > > > > > > > > separate config also meant a separate storage > type > > > > > sharing > > > > > > > > > > S3FileIO. > > > > > > > > > > > > That > > > > > > > > > > > > > > was the tradeoff that led me to try Option B. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do the diffs change anyone’s lean here? And are > the > > > > names > > > > > > > > > > > > > credentialIssuer > > > > > > > > > > > > > > / STS / CLOUDFLARE_R2 / > > SUPPORTED_S3_CREDENTIAL_ISSUERS > > > > > > > > > acceptable? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > > > > > > > > > Date: Sunday, September 13, 2026 at 19:50 > > > > > > > > > > > > > > To: [email protected] < > [email protected] > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with > > > > scoped > > > > > > > > > credential > > > > > > > > > > > > > vending > > > > > > > > > > > > > > > > > > > > > > > > > > > > Building on top of Ayush's and JB's suggestion, I > > agree > > > > > > that > > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > > can live inside the existing endpoint config > field. > > > > > > > > > > > > > > > > > > > > > > > > > > > > That leaves only Account ID, so potentially we > > could > > > > > store > > > > > > it > > > > > > > > in > > > > > > > > > > user > > > > > > > > > > > > or > > > > > > > > > > > > > > role ARN. Note that userArn is currently unused > in > > OSS > > > > > > code. > > > > > > > > > > > > > > > > > > > > > > > > > > > > That should allow us to keep existing > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > for > > > > > > > > > > > > all > > > > > > > > > > > > > > S3 storage backend, but will require special > > parsing of > > > > > > > > user/role > > > > > > > > > > ARN > > > > > > > > > > > > > field > > > > > > > > > > > > > > to distinguish R2 from the other systems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm still leaning towards the separate > > > > > > > > > R2StorageConfigurationInfo, > > > > > > > > > > > but > > > > > > > > > > > > > I'd > > > > > > > > > > > > > > be ok with overloading user/role ARN too if other > > > > people > > > > > > > prefer > > > > > > > > > > > that. I > > > > > > > > > > > > > > guess a key question in this context is whether > R2 > > has > > > > > the > > > > > > > > > concept > > > > > > > > > > of > > > > > > > > > > > > > role > > > > > > > > > > > > > > ARN at all.... but TBH I have no idea about > that. I > > > > hope > > > > > > > Austen > > > > > > > > > > might > > > > > > > > > > > > be > > > > > > > > > > > > > > able to clarify that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Welcome, and thanks for starting with a > > [DISCUSS]. > > > > Best > > > > > > way > > > > > > > > to > > > > > > > > > > > make a > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > contribution :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think the thread is talking about two layers, > > and > > > > we > > > > > > can > > > > > > > > keep > > > > > > > > > > > both > > > > > > > > > > > > > > > answers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On the client side this should stay S3: s3:// > > > > > locations, > > > > > > > s3.* > > > > > > > > > > > > > properties, > > > > > > > > > > > > > > > S3FileIO. Your PyIceberg / DuckDB / Iceberg > Java > > > > > results > > > > > > > > > already > > > > > > > > > > > show > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > unmodified clients work, so a dedicated > R2FileIO > > does > > > > > not > > > > > > > > seem > > > > > > > > > > > worth > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > On the server side I lean toward Dmitri: a > > distinct > > > > > > > > > > > > > > > R2StorageConfigurationInfo and storage > > integration. > > > > > > > Issuance > > > > > > > > is > > > > > > > > > > not > > > > > > > > > > > > > STS, > > > > > > > > > > > > > > > and folding Cloudflare fields into > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > mixes > > > > > > > > > > > > > > > unrelated concepts. The middle ground to avoid > > is a > > > > new > > > > > > > type > > > > > > > > > that > > > > > > > > > > > is > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > AWS-shaped underneath. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One concrete caution if the Option B prototype > > stays > > > > on > > > > > > the > > > > > > > > S3 > > > > > > > > > > > > config: > > > > > > > > > > > > > > > please do not overload stsUnavailable. Today > that > > > > flag > > > > > > > skips > > > > > > > > > > > > AssumeRole > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > disables vending. R2 wants the opposite. vend, > > just > > > > not > > > > > > via > > > > > > > > > STS. > > > > > > > > > > so > > > > > > > > > > > > > that > > > > > > > > > > > > > > > flag would change meaning for existing MinIO / > > NetApp > > > > > > > > catalogs. > > > > > > > > > > If > > > > > > > > > > > we > > > > > > > > > > > > > > stay > > > > > > > > > > > > > > > on S3 config, we need an explicit vending > > signal; a > > > > > > > dedicated > > > > > > > > > R2 > > > > > > > > > > > type > > > > > > > > > > > > > > > avoids that trap. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On config surface, jurisdiction can live in > > endpoint > > > > > > > > > (<account>. > > > > > > > > > > > > > > > r2.cloudflarestorage.com vs > <account>.eu.r2...). > > > > > Account > > > > > > > ID > > > > > > > > is > > > > > > > > > > > still > > > > > > > > > > > > > > > needed > > > > > > > > > > > > > > > for the JWT sub, so keeping an explicit > > accountId and > > > > > > > > deriving > > > > > > > > > > > > audience > > > > > > > > > > > > > > > from the endpoint seems enough. Parent tokens > > should > > > > > stay > > > > > > > in > > > > > > > > > > server > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > via storageName. > > > > > > > > > > > > > > > Local signing fits the vending model. > > Fail-closed on > > > > > > > > > cross-bucket > > > > > > > > > > > and > > > > > > > > > > > > > > mixed > > > > > > > > > > > > > > > read/write grants is the right call. Prefix > > claims > > > > > should > > > > > > > > > follow > > > > > > > > > > > the > > > > > > > > > > > > > same > > > > > > > > > > > > > > > path comparison Polaris already uses, so we do > > not > > > > > mint a > > > > > > > > > > > credential > > > > > > > > > > > > > > wider > > > > > > > > > > > > > > > than the grant. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator (#4486) > > > > > separately > > > > > > if > > > > > > > > R2 > > > > > > > > > > > needs > > > > > > > > > > > > > it, > > > > > > > > > > > > > > > then management API + Java config, then > vending, > > > > Python > > > > > > CLI > > > > > > > > on > > > > > > > > > > its > > > > > > > > > > > > own. > > > > > > > > > > > > > > > Sketching MinIO / Backblaze against the same > > fields > > > > is > > > > > a > > > > > > > > useful > > > > > > > > > > > > design > > > > > > > > > > > > > > > check; a generic non-STS SPI can wait for a > > second > > > > > > backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri > > Bourlatchkov < > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for starting this contribution! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > After briefly looking through your PR [1] I > > think > > > > > > > > > > > > > > > > adding R2StorageConfigInfo makes sense. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Even though "aws" in > > AwsStorageConfigurationInfo is > > > > > > > longer > > > > > > > > > > > implies > > > > > > > > > > > > > AWS > > > > > > > > > > > > > > > > services, and MinIO, Rust, Ozone are > supported > > just > > > > > as > > > > > > > > well, > > > > > > > > > R2 > > > > > > > > > > > > > appears > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > be significantly different from STS-based S3 > > > > systems > > > > > > > that a > > > > > > > > > > > > separate > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > is probably the most natural way to support > it > > in > > > > the > > > > > > > > Polaris > > > > > > > > > > > > > codebase. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The alternative, I guess, is to add R2 > > accountId > > > > and > > > > > > > > > > jurisdiction > > > > > > > > > > > > as > > > > > > > > > > > > > > > > optional properties to > > AwsStorageConfigurationInfo > > > > > (or > > > > > > > > > overload > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > properties), but that looks like mixing > > unrelated > > > > > > > concepts > > > > > > > > > > > > together. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Adding a generic property bag to > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > is > > > > > > > > > > > not > > > > > > > > > > > > > > > > convenient because > > > > > > PolarisStorageIntegrationProviderImpl > > > > > > > > will > > > > > > > > > > > > > probably > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > to know how to interpret it in order to > > redirect > > > > > > > credential > > > > > > > > > > > vending > > > > > > > > > > > > > > > > requests to R2-specific code. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wonder whether R2StorageConfigurationInfo > and > > > > > > > > > > > > > > > AwsStorageConfigurationInfo > > > > > > > > > > > > > > > > (and corresponding "storage integration" > > classes) > > > > > could > > > > > > > > > share a > > > > > > > > > > > > > common > > > > > > > > > > > > > > > base > > > > > > > > > > > > > > > > class for common properties (e.g. endpoint). > > Even > > > > > > though > > > > > > > > > > > "endpoint" > > > > > > > > > > > > > > might > > > > > > > > > > > > > > > > not be relevant to R2 in the cloud, the fact > > that > > > > > > > S3FileIO > > > > > > > > > uses > > > > > > > > > > > it, > > > > > > > > > > > > > and > > > > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > is accessed via S3FileIO means we should > > probably > > > > > allow > > > > > > > > users > > > > > > > > > > to > > > > > > > > > > > > > > > configure > > > > > > > > > > > > > > > > the full set of FileIO properties just in > case. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is fine for different > > > > > > > PolarisStorageIntegration > > > > > > > > > > > > > > > implementations > > > > > > > > > > > > > > > > to produce the same kind of > > StorageAccessConfig (S3 > > > > > in > > > > > > > this > > > > > > > > > > case) > > > > > > > > > > > > and > > > > > > > > > > > > > > > thus > > > > > > > > > > > > > > > > cause S3FileIO to be used by clients > (including > > > > > > Polaris' > > > > > > > > own > > > > > > > > > > > > storage > > > > > > > > > > > > > > > access > > > > > > > > > > > > > > > > paths). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] look a > bit > > > > > > concerning > > > > > > > > to > > > > > > > > > > me. > > > > > > > > > > > I > > > > > > > > > > > > > > wonder > > > > > > > > > > > > > > > > if this code could be refactored to avoid > > > > > dependencies > > > > > > on > > > > > > > > > > storage > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > types completely.... but TBH, I did not look > > too > > > > > deeply > > > > > > > > into > > > > > > > > > > this > > > > > > > > > > > > > > today. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anticipating future PRs for this in the > Polaris > > > > repo, > > > > > > I'd > > > > > > > > > like > > > > > > > > > > to > > > > > > > > > > > > ask > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > separate Python code changes from java > changes > > > > since > > > > > > > these > > > > > > > > > > areas > > > > > > > > > > > > > > usually > > > > > > > > > > > > > > > > attract different reviewers :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From my POV, management API and matching java > > > > changes > > > > > > can > > > > > > > > be > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > PR. I'd say docs should come later (to keep > PRs > > > > > small). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > All in all, thank you again and I hope this > > > > > > contribution > > > > > > > > > lands > > > > > > > > > > in > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM Jean-Baptiste > > > > Onofré > > > > > < > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and > welcome > > > > > aboard > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it makes sense to have R2 built > > under the > > > > > > > > existing > > > > > > > > > S3 > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > configuration. It matches how Polaris is > > already > > > > > > > > > structured: > > > > > > > > > > > > > > > > > StorageType is keyed on the URI scheme > > (s3://, > > > > > > s3a://), > > > > > > > > and > > > > > > > > > > > > > > > > > AwsStorageConfigurationInfo already carries > > > > > endpoint, > > > > > > > > > > > > stsEndpoint, > > > > > > > > > > > > > > > > > pathStyleAccess and, importantly, an > > > > stsUnavailable > > > > > > > flag. > > > > > > > > > > > > > > > > > So, what you need is largely there already: > > > > correct > > > > > > me > > > > > > > if > > > > > > > > > I'm > > > > > > > > > > > > > wrong, > > > > > > > > > > > > > > > > > R2 is essentially "S3 compatible store > where > > STS > > > > is > > > > > > > > > > > unavailable, > > > > > > > > > > > > > plus > > > > > > > > > > > > > > > > > a way to actually vend credentials in that > > case". > > > > > > Going > > > > > > > > > this > > > > > > > > > > > > route > > > > > > > > > > > > > > > > > also keeps this work "isolated" from the > > > > > > > > > > > FileIO-per-storage-type > > > > > > > > > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The interesting design question is a > > pluggable > > > > > > > credential > > > > > > > > > > > vending > > > > > > > > > > > > > > path > > > > > > > > > > > > > > > > > for S3 compatible stores when STS is not > > > > available, > > > > > > > with > > > > > > > > > > > > > Cloudflare's > > > > > > > > > > > > > > > > > local signing as the first concrete > > > > implementation, > > > > > > > > rather > > > > > > > > > > than > > > > > > > > > > > > R2 > > > > > > > > > > > > > > > > > specific code. It would be good to sketch > how > > > > MinIO > > > > > > > would > > > > > > > > > > slot > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > the same config fields before the vending > > path > > > > > > hardens > > > > > > > > > around > > > > > > > > > > > R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I believe we have to check if Cloudflare > > account > > > > ID > > > > > > and > > > > > > > > > > > > > jurisdiction > > > > > > > > > > > > > > > > > really need dedicated config fields, or can > > we > > > > use > > > > > > the > > > > > > > > > > endpoint > > > > > > > > > > > > > value > > > > > > > > > > > > > > > > > (with documentation)? Keeping the > > configuration > > > > > > surface > > > > > > > > > > minimal > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > help the design generalize. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would love to discuss two things: > > > > > > > > > > > > > > > > > 1. Polaris shipping a Cloudflare API token > > and > > > > > > locally > > > > > > > > > > minting > > > > > > > > > > > > > scoped > > > > > > > > > > > > > > > > > credentials makes it effectively an STS for > > R2, > > > > > which > > > > > > > > > > increases > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > blast radius of a server compromise. Your > > "reject > > > > > > > rather > > > > > > > > > than > > > > > > > > > > > > > > boarden" > > > > > > > > > > > > > > > > > handling of unrepresentable scopes is the > > right > > > > > call. > > > > > > > > > > > > > > > > > 2. The PyIceberg/DuckDB/Java results are > > > > promising. > > > > > > > Just > > > > > > > > > > > curious > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > Spark or Trino. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I would > > suggest > > > > to > > > > > > > start > > > > > > > > > with > > > > > > > > > > > the > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > > > > config and the management API as first > > shoot, and > > > > > we > > > > > > > can > > > > > > > > > > follow > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > the credential vending implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen Tomek > > > > > > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago Trading > > Company. > > > > We > > > > > > use > > > > > > > > > > Polaris > > > > > > > > > > > > for > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > Iceberg catalogs and we're evaluating > > Cloudflare > > > > R2 > > > > > > as > > > > > > > an > > > > > > > > > > > object > > > > > > > > > > > > > > store > > > > > > > > > > > > > > > > (the > > > > > > > > > > > > > > > > > no egress fees make it a very compelling > > > > product). > > > > > > I’d > > > > > > > > like > > > > > > > > > > to > > > > > > > > > > > > > > discuss > > > > > > > > > > > > > > > > > contributing R2 support and credential > > vending to > > > > > > > Polaris > > > > > > > > > > with > > > > > > > > > > > > the > > > > > > > > > > > > > > goal > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > getting feedback before opening a > > > > ready-for-review > > > > > > > > upstream > > > > > > > > > > PR. > > > > > > > > > > > > > This > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > my > > > > > > > > > > > > > > > > > first contribution, so trying to do this > > right. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it does > > not > > > > > > provide > > > > > > > > AWS > > > > > > > > > > > STS. > > > > > > > > > > > > > > > > > Cloudflare instead supports these > > short-lived, > > > > > scoped > > > > > > > > > > > credentials > > > > > > > > > > > > > > > > generated > > > > > > > > > > > > > > > > > by locally signed JWT with the server > > holding the > > > > > > > parent > > > > > > > > > API > > > > > > > > > > > > token > > > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > This allows us to retain Polaris’s > > authorization > > > > > and > > > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > > model while storing our table data in R2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have a prototype with the following > > design: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > An opt-in R2 storage type, with catalog > > > > > > configuration > > > > > > > > > > > > identifying > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > Cloudflare account and optional > jurisdiction. > > > > > Parent > > > > > > > > > > > credentials > > > > > > > > > > > > > > remain > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > server configuration and can be selected > > through > > > > > > > > > storageName. > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > Local credential generation using > > Cloudflare’s > > > > > > > > documented > > > > > > > > > > > > format. > > > > > > > > > > > > > > > > > Credentials expire, are restricted to one > > bucket, > > > > > and > > > > > > > > carry > > > > > > > > > > > > prefix > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > access scopes derived from Polaris’s > location > > > > > grants. > > > > > > > The > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > uses > > > > > > > > > > > > > > > > > Polaris’s existing credential cache. > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > Credentials returned through the existing > > > > Iceberg > > > > > > > REST > > > > > > > > > > > vending > > > > > > > > > > > > > flow > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > s3.* properties, including the endpoint, > > session > > > > > > token, > > > > > > > > and > > > > > > > > > > > > expiry. > > > > > > > > > > > > > > > > > Compatible clients use their existing S3 > > FileIO. > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > Conservative handling of scopes the > > > > > implementation > > > > > > > > cannot > > > > > > > > > > > > > > represent: > > > > > > > > > > > > > > > > > cross-bucket and mixed read/write grants > are > > > > > rejected > > > > > > > > > rather > > > > > > > > > > > than > > > > > > > > > > > > > > > > > broadened. R2 is excluded from the default > > > > > > > > > > > > supported-storage-types > > > > > > > > > > > > > > > list. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The main design questions I feel this > runs > > > > into: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > Should this be a distinct storage type > or a > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > > mechanism > > > > > > > > > > > > > > > > > selected through the existing S3 > > configuration? > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > I chose a separate type because the > > identity, > > > > > > > > credential > > > > > > > > > > > > > > generation, > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > scoping model differ from AWS STS, as this > > seemed > > > > > > more > > > > > > > > > > > > appropriate > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is a draft preview PR against my > fork > > > > [2], > > > > > > > > > including > > > > > > > > > > > > > > > > > implementation, management API changes, CLI > > > > > support, > > > > > > > > tests, > > > > > > > > > > and > > > > > > > > > > > > > > > > > documentation. It also discloses the AI > > > > assistance > > > > > > used > > > > > > > > > > during > > > > > > > > > > > > > > > > > implementation. I leaned heavily on Fable > > 5.1 for > > > > > > this > > > > > > > > > > > > > implementation > > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > want to be upfront about it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We are running the implementation in a > > > > > > non-production > > > > > > > > > > > > environment > > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > now as a POC. Validation includes live R2 > > > > > > reads/writes, > > > > > > > > > > > multipart > > > > > > > > > > > > > > > > > operations, negative scope-boundary tests, > > our > > > > > > internal > > > > > > > > > > Python > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > regression matrix, and credential > > > > rotation/recovery > > > > > > > > > > exercises. > > > > > > > > > > > > The > > > > > > > > > > > > > > > > upstream > > > > > > > > > > > > > > > > > CI workflow has passed on the fork. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The change also encounters the existing > > > > > assumption > > > > > > > > that a > > > > > > > > > > > > FileIO > > > > > > > > > > > > > > > > > implementation identifies exactly one > storage > > > > type: > > > > > > > both > > > > > > > > S3 > > > > > > > > > > and > > > > > > > > > > > > R2 > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > S3FileIO. This overlaps with #4486 [3]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback on: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > Should R2 have its own storage > > > > > configuration/type, > > > > > > or > > > > > > > > > > should > > > > > > > > > > > > > > non-STS > > > > > > > > > > > > > > > > > vending be modeled within the existing S3 > > > > > > integration? > > > > > > > > > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > Does the local-signing approach fit > > Polaris’s > > > > > > > > > > > > credential-vending > > > > > > > > > > > > > > > model? > > > > > > > > > > > > > > > > > Any additional constraints or validations > > > > > necessary? > > > > > > > > > > > > > > > > > > 3. > > > > > > > > > > > > > > > > > > Would you prefer the shared FileIO > > validation > > > > > > change > > > > > > > to > > > > > > > > > > land > > > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > > > and how should we sequence the management > > API and > > > > > R2 > > > > > > > > > > > integration > > > > > > > > > > > > > > > changes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Happy to adapt the design and split the > > > > > > > contribution > > > > > > > > > into > > > > > > > > > > > > more > > > > > > > > > > > > > > > > focused > > > > > > > > > > > > > > > > > PRs. Wanted to start the discussion > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > > > > > > > > > > > > > > > > > [2] > > > > > > https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > > > [3] > > > > > > https://github.com/apache/polaris/issues/4486 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac< > > > > > > https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any > > attached > > > > > files > > > > > > > > > contain > > > > > > > > > > > > > > > information > > > > > > > > > > > > > > > > > intended for the exclusive use of the > > individual > > > > or > > > > > > > > entity > > > > > > > > > to > > > > > > > > > > > > whom > > > > > > > > > > > > > it > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > addressed and may contain information that > is > > > > > > > > proprietary, > > > > > > > > > > > > > > confidential > > > > > > > > > > > > > > > > > and/or exempt from disclosure under > > applicable > > > > law. > > > > > > If > > > > > > > > you > > > > > > > > > > are > > > > > > > > > > > > not > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > intended recipient, you are hereby notified > > that > > > > > any > > > > > > > > > viewing, > > > > > > > > > > > > > > copying, > > > > > > > > > > > > > > > > > disclosure or distribution of this > > information > > > > may > > > > > be > > > > > > > > > subject > > > > > > > > > > > to > > > > > > > > > > > > > > legal > > > > > > > > > > > > > > > > > restriction or sanction. Please notify the > > > > sender, > > > > > by > > > > > > > > > > > electronic > > > > > > > > > > > > > mail > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > telephone, of any unintended recipients and > > > > delete > > > > > > the > > > > > > > > > > original > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached > files > > > > > contain > > > > > > > > > > > information > > > > > > > > > > > > > > intended for the exclusive use of the individual > or > > > > > entity > > > > > > to > > > > > > > > > whom > > > > > > > > > > it > > > > > > > > > > > > is > > > > > > > > > > > > > > addressed and may contain information that is > > > > > proprietary, > > > > > > > > > > > confidential > > > > > > > > > > > > > > and/or exempt from disclosure under applicable > > law. If > > > > > you > > > > > > > are > > > > > > > > > not > > > > > > > > > > > the > > > > > > > > > > > > > > intended recipient, you are hereby notified that > > any > > > > > > viewing, > > > > > > > > > > > copying, > > > > > > > > > > > > > > disclosure or distribution of this information > may > > be > > > > > > subject > > > > > > > > to > > > > > > > > > > > legal > > > > > > > > > > > > > > restriction or sanction. Please notify the > sender, > > by > > > > > > > > electronic > > > > > > > > > > mail > > > > > > > > > > > > or > > > > > > > > > > > > > > telephone, of any unintended recipients and > delete > > the > > > > > > > original > > > > > > > > > > > message > > > > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > > contain > > > > > > > > information > > > > > > > > > > > intended for the exclusive use of the individual or > > entity to > > > > > > whom > > > > > > > it > > > > > > > > > is > > > > > > > > > > > addressed and may contain information that is > > proprietary, > > > > > > > > confidential > > > > > > > > > > > and/or exempt from disclosure under applicable law. If > > you > > > > are > > > > > > not > > > > > > > > the > > > > > > > > > > > intended recipient, you are hereby notified that any > > viewing, > > > > > > > > copying, > > > > > > > > > > > disclosure or distribution of this information may be > > subject > > > > > to > > > > > > > > legal > > > > > > > > > > > restriction or sanction. Please notify the sender, by > > > > > electronic > > > > > > > mail > > > > > > > > > or > > > > > > > > > > > telephone, of any unintended recipients and delete the > > > > original > > > > > > > > message > > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > contain > > > > > > > information > > > > > > > > > > intended for the exclusive use of the individual or > entity > > to > > > > > whom > > > > > > it > > > > > > > > is > > > > > > > > > > addressed and may contain information that is > proprietary, > > > > > > > confidential > > > > > > > > > > and/or exempt from disclosure under applicable law. If > you > > are > > > > > not > > > > > > > the > > > > > > > > > > intended recipient, you are hereby notified that any > > viewing, > > > > > > > copying, > > > > > > > > > > disclosure or distribution of this information may be > > subject > > > > to > > > > > > > legal > > > > > > > > > > restriction or sanction. Please notify the sender, by > > > > electronic > > > > > > mail > > > > > > > > or > > > > > > > > > > telephone, of any unintended recipients and delete the > > original > > > > > > > message > > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > > > > information > > > > > > > > intended for the exclusive use of the individual or entity to > > whom > > > > it > > > > > > is > > > > > > > > addressed and may contain information that is proprietary, > > > > > confidential > > > > > > > > and/or exempt from disclosure under applicable law. If you > are > > not > > > > > the > > > > > > > > intended recipient, you are hereby notified that any viewing, > > > > > copying, > > > > > > > > disclosure or distribution of this information may be subject > > to > > > > > legal > > > > > > > > restriction or sanction. Please notify the sender, by > > electronic > > > > mail > > > > > > or > > > > > > > > telephone, of any unintended recipients and delete the > original > > > > > message > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies.
