Hey all, Any chance I can get further reviews this week on the PR? I’d like to get moving onto Phase 2 here if there are no further issues to address.
Get Outlook for Mac<https://aka.ms/GetOutlookForMac> From: Dmitri Bourlatchkov <[email protected]> Date: Thursday, September 24, 2026 at 17:33 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential vending Hi Yufei, > The REST spec is a contract between Polaris servers and clients [...] I tend to think that this statement is a bit too strong. The Management API spec defines a protocol for clients and servers to communicate. > [...] This would help avoid a situation where a Polaris client works > only with certain deployments. Since Polaris has been developed as a customizable system, a REST API specification defined by Apache Polaris cannot control the behaviour of all possible downstream environments. The purpose of the spec is to ensure protocol compatibility. Note that the value of `credentialVendingMechanism` has no bearing on any logic clients need to execute. It is an opaque parameter controlled by the Polaris administrator. > The new credentialVendingMechanism field in the REST spec looks > reasonable. My only suggestion is to use an enum instead of a string. [...] The above points are exactly the reason why the Management API cannot enumerate all possible vending mechanisms. The same applies to storage types too. However, we do not have any practical examples of 3rd party storage systems that go beyond S3/GCS/ADLS (yet). Cheers, Dmitri. On Thu, Sep 24, 2026 at 4:38 PM Yufei Gu <[email protected]> wrote: > Thanks Austen. It's fair that endpoints are not reliable given that some > s3-compatible can use arbitrary hostnames. > I also think stsUnavailable is fine to keep as-is. I don’t think we need to > change its scope or repurpose it. > > The new credentialVendingMechanism field in the REST spec looks reasonable. > My only suggestion is to use an enum instead of a string. The REST spec is > a contract between Polaris servers and clients, so I’d expect it to define > the valid values and their meanings rather than leaving them up to each > deployment. This would help avoid a situation where a Polaris client works > only with certain deployments. Deployment-specific mechanisms shouldn’t be > part of the standard REST contract; users who need a custom mechanism can > always extend the server in their own deployment. > > > Either way the operator needs a per-realm switch like > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control which realm can mint > R2 credentials from the operators Cloudflare account. > > Could you clarify why we need a separate per-realm switch? Once the storage > configuration is set at the catalog level to select a specific mechanism, > the catalog will have no choice but to support it. Do we need another layer > of gating? > > Yufei > > > On Wed, Sep 23, 2026 at 5:53 AM Austen Tomek < > [email protected]> wrote: > > > Hey Yufei, > > > > Apologies for missing your question. > > > > I don’t think Endpoint is sufficient. > > > > Endpoint matching only works where the hostname identifies the vendor > like > > R2’s public hostnames that are recognizable, MinIO and other self-hosted > > S3-compatible stores use arbitrary hostnames so there would be nothing to > > match on. On top of that, if a match fails it would just silently fall > back > > to STS. > > > > Right now the endpoint just says how you’ll connect not how the > > credentials are vended. There's endpoint and also endpointInternal for > > server-side access, so it would be unclear which of the two decides. > > > > For your other question, there is no second mechanism for R2 today, but > > for other self-hosted stores the same endpoint could be served by STS or > by > > a different token mechanism chosen per catalog. > > > > stsUnavailable keeps it’s current behavior in the PR (that was Prithvi’s > > point earlier in the thread) > > > > Either way the operator needs a per-realm switch like > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS to control which realm can > mint > > R2 credentials from the operators Cloudflare account. > > > > Thanks, > > Austen > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > From: Yufei Gu <[email protected]> > > Date: Monday, September 21, 2026 at 17:53 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > vending > > > > Hi Austen, > > > > Thanks for the update. After looking at PR #5513, I think we should first > > clarify whether we need a new credentialVendingMechanism field in the > > spec/config. > > > > Austen, you mentioned that the R2 account ID and jurisdiction already > come > > from the endpoint. Could we also use the existing field like `endpoint`, > to > > recognize R2 and select its credential-vending mechanism, while keeping > the > > current stsUnavailable behavior unchanged? > > > > Are there cases where the endpoint isn’t enough, or where users need to > > choose different vending mechanisms for the same endpoint(e.g., R2) per > > catalog? > > > > On Wed, Sep 16, 2026 at 7:24 AM Austen Tomek < > > [email protected]> wrote: > > > > > Hi Yufei, Dimitri, > > > > > > Thanks, both. PR 1 is ready for review: > > > https://github.com/apache/polaris/pull/5513 > > > > > > I pushed a revision last night that takes in Dimitri’s comments. > > > > > > > > > * > > > I changed from issuer to credentialVendingMechanism. It’s also now a > > plain > > > string and the realm allowlist is > > > SUPPORTED_S3_CREDENTIAL_VENDING_MECHANISMS, which still defaults to > [STS] > > > * > > > Mechanisms are CDI beans found by their identifier, as Dimitri > proposed. > > > * > > > The CLOUDFLARE_R2 bean and the R2 vending code will follow in the next > PR > > > > > > > > > CI workflow is waiting approval to run. > > > > > > Thanks, > > > Austen > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > Date: Tuesday, September 15, 2026 at 20:00 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential > > vending > > > > > > Hi All, > > > > > > > Having getCloudflareR2Endpoint on AwsStorageConfigurationInfo > > > > > > Yes, this does feel awkward. > > > > > > From my POV pursuing the option with the "credentialIssuers" property > > looks > > > promising. This property is fairly generic and fits naturally into the > > > config, given that we already target multiple backends. > > > > > > Cheers, > > > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 8:48 PM Yufei Gu <[email protected]> wrote: > > > > > > > Hi Austen, Dmitri, > > > > > > > > > Having getCloudflareR2Endpoint on AwsStorageConfigurationInfo > > > > > > > > That does feel awkward. However, AwsStorageConfigurationInfo has > > > supported > > > > S3-compatible backends beyond AWS for a while, so there is precedent > > > (e.g., > > > > MinIO, etc.). The underlying mismatch is that we call the storage > type > > > S3, > > > > but its configuration mixes shared S3 settings with AWS-specific > ones: > > > > > > > > AwsStorageConfigurationInfo → storage type: S3 > > > > ├── Common S3 settings > > > > │ endpoint, region, pathStyleAccess, allowedLocations > > > > └── AWS-specific settings > > > > roleArn, externalId, STS, KMS > > > > > > > > I’m open to separating the shared S3 settings from provider-specific > > > > configuration if that helps, but I don’t think we need to settle on a > > > > broader refactor now. > > > > > > > > Let me know when the PR is ready for review. I’d be happy to take a > > > closer > > > > look. > > > > > > > > Yufei > > > > > > > > > > > > Yufei > > > > > > > > > > > > On Mon, Sep 14, 2026 at 3:18 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Austen, > > > > > > > > > > +1 to adding "credentialIssues" or similar (renamed) property to S3 > > > > config. > > > > > In fact, I was looking exactly for something like that last week > but > > > the > > > > > idea did not quite condense in my mind :) > > > > > > > > > > I posted some preliminary comments on the draft PR. > > > > > > > > > > Just for community awareness, I'd like to propose here that we use > > CDI > > > > for > > > > > finding actual "issuer" / "mechanism" implementations. The codebase > > has > > > > > many examples of this. The closest match in this case is > > > > > probably PolarisStorageIntegrationProvider. > > > > > > > > > > This way the API does not have to enumerate all possible values. > > > > Downstream > > > > > builds are free to add different implementations without having to > > > change > > > > > the API spec. The "allowed" list still makes sense. > > > > > > > > > > In general, I like the direction this feature is taking, although I > > > > expect > > > > > PR reviews may take a few rounds :) > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek < > > > > > [email protected]> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Closing the loop since my last note and answering the questions > > that > > > > have > > > > > > come up. > > > > > > > > > > > > Dmitri's question on role ARNs: R2 doesn’t have AWS-style IAM > roles > > > or > > > > > > ARNs. Our implementation signs temporary credentials with a > parent > > > > > token’s > > > > > > secret, scoped to a bucket and a set of prefixes. The account ID > > and > > > > > > jurisdiction come from the R2 endpoint. Putting the account ID in > > > > roleArn > > > > > > or userArn would give those fields a different meaning, so I’ve > > left > > > > them > > > > > > absent for CLOUDFLARE_R2. > > > > > > > > > > > > I pursued Option B. The S3 configuration gains one new field, > > > > > > credentialIssuer, with values STS > > > > > > (default) and CLOUDFLARE_R2 and a realm allowlist > > > > > > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults > > > > > > to [STS]. I left stsUnavailable untouched (Prithvi) so it still > > means > > > > > > "vend nothing". The > > > > > > credentialIssuer is the explicit sign. The R2 config rejects > > > > > > stsUnavailable along with a host of other STS and KMS fields. I'm > > > > running > > > > > > this in our non-production environment at the moment. > > > > > > > > > > > > Draft PR for stage 1 (config field, API, allowlist, gates): > > > > > > https://github.com/apache/polaris/pull/5513. > > > > > > > > > > > > I still see the appeal of a separate R2 configuration, as Dmitri > > > > > > suggested. My original reason for separating it was to keep the > > > > > R2-specific > > > > > > configuration and validation together. Having > > getCloudflareR2Endpoint > > > > on > > > > > > AwsStorageConfigurationInfo is one part I’d still like feedback > on. > > > > > > > > > > > > I wanted to put a working version of Option B in front of > everyone > > so > > > > we > > > > > > could discuss the actual changes. In my earlier implementation, > the > > > > > > separate config also meant a separate storage type sharing > > S3FileIO. > > > > That > > > > > > was the tradeoff that led me to try Option B. > > > > > > > > > > > > Do the diffs change anyone’s lean here? And are the names > > > > > credentialIssuer > > > > > > / STS / CLOUDFLARE_R2 / SUPPORTED_S3_CREDENTIAL_ISSUERS > acceptable? > > > > > > > > > > > > Thanks, > > > > > > Austen > > > > > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > > > > Date: Sunday, September 13, 2026 at 19:50 > > > > > > To: [email protected] <[email protected]> > > > > > > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped > credential > > > > > vending > > > > > > > > > > > > Building on top of Ayush's and JB's suggestion, I agree that > > > > jurisdiction > > > > > > can live inside the existing endpoint config field. > > > > > > > > > > > > That leaves only Account ID, so potentially we could store it in > > user > > > > or > > > > > > role ARN. Note that userArn is currently unused in OSS code. > > > > > > > > > > > > That should allow us to keep existing AwsStorageConfigurationInfo > > for > > > > all > > > > > > S3 storage backend, but will require special parsing of user/role > > ARN > > > > > field > > > > > > to distinguish R2 from the other systems. > > > > > > > > > > > > I'm still leaning towards the separate > R2StorageConfigurationInfo, > > > but > > > > > I'd > > > > > > be ok with overloading user/role ARN too if other people prefer > > > that. I > > > > > > guess a key question in this context is whether R2 has the > concept > > of > > > > > role > > > > > > ARN at all.... but TBH I have no idea about that. I hope Austen > > might > > > > be > > > > > > able to clarify that. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > Welcome, and thanks for starting with a [DISCUSS]. Best way to > > > make a > > > > > > first > > > > > > > contribution :) > > > > > > > > > > > > > > I think the thread is talking about two layers, and we can keep > > > both > > > > > > > answers. > > > > > > > > > > > > > > On the client side this should stay S3: s3:// locations, s3.* > > > > > properties, > > > > > > > S3FileIO. Your PyIceberg / DuckDB / Iceberg Java results > already > > > show > > > > > > that > > > > > > > unmodified clients work, so a dedicated R2FileIO does not seem > > > worth > > > > > it. > > > > > > > On the server side I lean toward Dmitri: a distinct > > > > > > > R2StorageConfigurationInfo and storage integration. Issuance is > > not > > > > > STS, > > > > > > > and folding Cloudflare fields into AwsStorageConfigurationInfo > > > mixes > > > > > > > unrelated concepts. The middle ground to avoid is a new type > that > > > is > > > > > > still > > > > > > > AWS-shaped underneath. > > > > > > > > > > > > > > One concrete caution if the Option B prototype stays on the S3 > > > > config: > > > > > > > please do not overload stsUnavailable. Today that flag skips > > > > AssumeRole > > > > > > and > > > > > > > disables vending. R2 wants the opposite. vend, just not via > STS. > > so > > > > > that > > > > > > > flag would change meaning for existing MinIO / NetApp catalogs. > > If > > > we > > > > > > stay > > > > > > > on S3 config, we need an explicit vending signal; a dedicated > R2 > > > type > > > > > > > avoids that trap. > > > > > > > > > > > > > > On config surface, jurisdiction can live in endpoint > (<account>. > > > > > > > r2.cloudflarestorage.com vs <account>.eu.r2...). Account ID is > > > still > > > > > > > needed > > > > > > > for the JWT sub, so keeping an explicit accountId and deriving > > > > audience > > > > > > > from the endpoint seems enough. Parent tokens should stay in > > server > > > > > > config > > > > > > > via storageName. > > > > > > > Local signing fits the vending model. Fail-closed on > cross-bucket > > > and > > > > > > mixed > > > > > > > read/write grants is the right call. Prefix claims should > follow > > > the > > > > > same > > > > > > > path comparison Polaris already uses, so we do not mint a > > > credential > > > > > > wider > > > > > > > than the grant. > > > > > > > > > > > > > > +1 to small PRs: FileIO discriminator (#4486) separately if R2 > > > needs > > > > > it, > > > > > > > then management API + Java config, then vending, Python CLI on > > its > > > > own. > > > > > > > Sketching MinIO / Backblaze against the same fields is a useful > > > > design > > > > > > > check; a generic non-STS SPI can wait for a second backend. > > > > > > > > > > > > > > Regards, > > > > > > > Prithvi S > > > > > > > > > > > > > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri Bourlatchkov < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > Thanks for starting this contribution! > > > > > > > > > > > > > > > > After briefly looking through your PR [1] I think > > > > > > > > adding R2StorageConfigInfo makes sense. > > > > > > > > > > > > > > > > Even though "aws" in AwsStorageConfigurationInfo is longer > > > implies > > > > > AWS > > > > > > > > services, and MinIO, Rust, Ozone are supported just as well, > R2 > > > > > appears > > > > > > > to > > > > > > > > be significantly different from STS-based S3 systems that a > > > > separate > > > > > > > config > > > > > > > > is probably the most natural way to support it in the Polaris > > > > > codebase. > > > > > > > > > > > > > > > > The alternative, I guess, is to add R2 accountId and > > jurisdiction > > > > as > > > > > > > > optional properties to AwsStorageConfigurationInfo (or > overload > > > > > > existing > > > > > > > > properties), but that looks like mixing unrelated concepts > > > > together. > > > > > > > > > > > > > > > > Adding a generic property bag to AwsStorageConfigurationInfo > is > > > not > > > > > > > > convenient because PolarisStorageIntegrationProviderImpl will > > > > > probably > > > > > > > have > > > > > > > > to know how to interpret it in order to redirect credential > > > vending > > > > > > > > requests to R2-specific code. > > > > > > > > > > > > > > > > I wonder whether R2StorageConfigurationInfo and > > > > > > > AwsStorageConfigurationInfo > > > > > > > > (and corresponding "storage integration" classes) could > share a > > > > > common > > > > > > > base > > > > > > > > class for common properties (e.g. endpoint). Even though > > > "endpoint" > > > > > > might > > > > > > > > not be relevant to R2 in the cloud, the fact that S3FileIO > uses > > > it, > > > > > and > > > > > > > R2 > > > > > > > > is accessed via S3FileIO means we should probably allow users > > to > > > > > > > configure > > > > > > > > the full set of FileIO properties just in case. > > > > > > > > > > > > > > > > I think it is fine for different PolarisStorageIntegration > > > > > > > implementations > > > > > > > > to produce the same kind of StorageAccessConfig (S3 in this > > case) > > > > and > > > > > > > thus > > > > > > > > cause S3FileIO to be used by clients (including Polaris' own > > > > storage > > > > > > > access > > > > > > > > paths). > > > > > > > > > > > > > > > > Changes in StorageTypeFileIO in [1] look a bit concerning to > > me. > > > I > > > > > > wonder > > > > > > > > if this code could be refactored to avoid dependencies on > > storage > > > > > > config > > > > > > > > types completely.... but TBH, I did not look too deeply into > > this > > > > > > today. > > > > > > > > > > > > > > > > Anticipating future PRs for this in the Polaris repo, I'd > like > > to > > > > ask > > > > > > to > > > > > > > > separate Python code changes from java changes since these > > areas > > > > > > usually > > > > > > > > attract different reviewers :) > > > > > > > > > > > > > > > > From my POV, management API and matching java changes can be > in > > > the > > > > > > same > > > > > > > > PR. I'd say docs should come later (to keep PRs small). > > > > > > > > > > > > > > > > All in all, thank you again and I hope this contribution > lands > > in > > > > > > Polaris > > > > > > > > :) > > > > > > > > > > > > > > > > [1] https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:52 PM Jean-Baptiste Onofré < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Austen, > > > > > > > > > > > > > > > > > > Nice discussion! Thanks for that, and welcome aboard :) > > > > > > > > > > > > > > > > > > I think it makes sense to have R2 built under the existing > S3 > > > > > storage > > > > > > > > > configuration. It matches how Polaris is already > structured: > > > > > > > > > StorageType is keyed on the URI scheme (s3://, s3a://), and > > > > > > > > > AwsStorageConfigurationInfo already carries endpoint, > > > > stsEndpoint, > > > > > > > > > pathStyleAccess and, importantly, an stsUnavailable flag. > > > > > > > > > So, what you need is largely there already: correct me if > I'm > > > > > wrong, > > > > > > > > > R2 is essentially "S3 compatible store where STS is > > > unavailable, > > > > > plus > > > > > > > > > a way to actually vend credentials in that case". Going > this > > > > route > > > > > > > > > also keeps this work "isolated" from the > > > FileIO-per-storage-type > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > The interesting design question is a pluggable credential > > > vending > > > > > > path > > > > > > > > > for S3 compatible stores when STS is not available, with > > > > > Cloudflare's > > > > > > > > > local signing as the first concrete implementation, rather > > than > > > > R2 > > > > > > > > > specific code. It would be good to sketch how MinIO would > > slot > > > > into > > > > > > > > > the same config fields before the vending path hardens > around > > > R2. > > > > > > > > > > > > > > > > > > I believe we have to check if Cloudflare account ID and > > > > > jurisdiction > > > > > > > > > really need dedicated config fields, or can we use the > > endpoint > > > > > value > > > > > > > > > (with documentation)? Keeping the configuration surface > > minimal > > > > > will > > > > > > > > > help the design generalize. > > > > > > > > > > > > > > > > > > I would love to discuss two things: > > > > > > > > > 1. Polaris shipping a Cloudflare API token and locally > > minting > > > > > scoped > > > > > > > > > credentials makes it effectively an STS for R2, which > > increases > > > > the > > > > > > > > > blast radius of a server compromise. Your "reject rather > than > > > > > > boarden" > > > > > > > > > handling of unrepresentable scopes is the right call. > > > > > > > > > 2. The PyIceberg/DuckDB/Java results are promising. Just > > > curious > > > > > > about > > > > > > > > > Spark or Trino. > > > > > > > > > > > > > > > > > > We love small, well-scoped PRs. I would suggest to start > with > > > the > > > > > S3 > > > > > > > > > config and the management API as first shoot, and we can > > follow > > > > > with > > > > > > > > > the credential vending implementation. > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > JB > > > > > > > > > > > > > > > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen Tomek > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I’m Austen Tomek from Chicago Trading Company. We use > > Polaris > > > > for > > > > > > our > > > > > > > > > Iceberg catalogs and we're evaluating Cloudflare R2 as an > > > object > > > > > > store > > > > > > > > (the > > > > > > > > > no egress fees make it a very compelling product). I’d like > > to > > > > > > discuss > > > > > > > > > contributing R2 support and credential vending to Polaris > > with > > > > the > > > > > > goal > > > > > > > > of > > > > > > > > > getting feedback before opening a ready-for-review upstream > > PR. > > > > > This > > > > > > is > > > > > > > > my > > > > > > > > > first contribution, so trying to do this right. > > > > > > > > > > > > > > > > > > > > R2 exposes S3-compatible APIs but it does not provide AWS > > > STS. > > > > > > > > > Cloudflare instead supports these short-lived, scoped > > > credentials > > > > > > > > generated > > > > > > > > > by locally signed JWT with the server holding the parent > API > > > > token > > > > > > [1]. > > > > > > > > > This allows us to retain Polaris’s authorization and > > > > > > credential-vending > > > > > > > > > model while storing our table data in R2. > > > > > > > > > > > > > > > > > > > > I have a prototype with the following design: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > An opt-in R2 storage type, with catalog configuration > > > > identifying > > > > > > the > > > > > > > > > Cloudflare account and optional jurisdiction. Parent > > > credentials > > > > > > remain > > > > > > > > in > > > > > > > > > server configuration and can be selected through > storageName. > > > > > > > > > > * > > > > > > > > > > Local credential generation using Cloudflare’s documented > > > > format. > > > > > > > > > Credentials expire, are restricted to one bucket, and carry > > > > prefix > > > > > > and > > > > > > > > > access scopes derived from Polaris’s location grants. The > > > > > integration > > > > > > > > uses > > > > > > > > > Polaris’s existing credential cache. > > > > > > > > > > * > > > > > > > > > > Credentials returned through the existing Iceberg REST > > > vending > > > > > flow > > > > > > > as > > > > > > > > > s3.* properties, including the endpoint, session token, and > > > > expiry. > > > > > > > > > Compatible clients use their existing S3 FileIO. > > > > > > > > > > * > > > > > > > > > > Conservative handling of scopes the implementation cannot > > > > > > represent: > > > > > > > > > cross-bucket and mixed read/write grants are rejected > rather > > > than > > > > > > > > > broadened. R2 is excluded from the default > > > > supported-storage-types > > > > > > > list. > > > > > > > > > > > > > > > > > > > > The main design questions I feel this runs into: > > > > > > > > > > > > > > > > > > > > * > > > > > > > > > > Should this be a distinct storage type or a > > > credential-vending > > > > > > > > mechanism > > > > > > > > > selected through the existing S3 configuration? > > > > > > > > > > * > > > > > > > > > > I chose a separate type because the identity, credential > > > > > > generation, > > > > > > > > and > > > > > > > > > scoping model differ from AWS STS, as this seemed more > > > > appropriate > > > > > > > > > > > > > > > > > > > > There is a draft preview PR against my fork [2], > including > > > > > > > > > implementation, management API changes, CLI support, tests, > > and > > > > > > > > > documentation. It also discloses the AI assistance used > > during > > > > > > > > > implementation. I leaned heavily on Fable 5.1 for this > > > > > implementation > > > > > > > so > > > > > > > > I > > > > > > > > > want to be upfront about it. > > > > > > > > > > > > > > > > > > > > We are running the implementation in a non-production > > > > environment > > > > > > > right > > > > > > > > > now as a POC. Validation includes live R2 reads/writes, > > > multipart > > > > > > > > > operations, negative scope-boundary tests, our internal > > Python > > > > > client > > > > > > > > > regression matrix, and credential rotation/recovery > > exercises. > > > > The > > > > > > > > upstream > > > > > > > > > CI workflow has passed on the fork. > > > > > > > > > > > > > > > > > > > > The change also encounters the existing assumption that a > > > > FileIO > > > > > > > > > implementation identifies exactly one storage type: both S3 > > and > > > > R2 > > > > > > use > > > > > > > > > S3FileIO. This overlaps with #4486 [3]. > > > > > > > > > > > > > > > > > > > > I’d particularly appreciate feedback on: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > Should R2 have its own storage configuration/type, or > > should > > > > > > non-STS > > > > > > > > > vending be modeled within the existing S3 integration? > > > > > > > > > > 2. > > > > > > > > > > Does the local-signing approach fit Polaris’s > > > > credential-vending > > > > > > > model? > > > > > > > > > Any additional constraints or validations necessary? > > > > > > > > > > 3. > > > > > > > > > > Would you prefer the shared FileIO validation change to > > land > > > > > > > > separately, > > > > > > > > > and how should we sequence the management API and R2 > > > integration > > > > > > > changes? > > > > > > > > > > > > > > > > > > > > Happy to adapt the design and split the contribution > into > > > > more > > > > > > > > focused > > > > > > > > > PRs. Wanted to start the discussion > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Austen > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > > > > > > > > > [2] https://github.com/deepdishgary/polaris/pull/1 > > > > > > > > > > [3] https://github.com/apache/polaris/issues/4486 > > > > > > > > > > > > > > > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files > contain > > > > > > > information > > > > > > > > > intended for the exclusive use of the individual or entity > to > > > > whom > > > > > it > > > > > > > is > > > > > > > > > addressed and may contain information that is proprietary, > > > > > > confidential > > > > > > > > > and/or exempt from disclosure under applicable law. If you > > are > > > > not > > > > > > the > > > > > > > > > intended recipient, you are hereby notified that any > viewing, > > > > > > copying, > > > > > > > > > disclosure or distribution of this information may be > subject > > > to > > > > > > legal > > > > > > > > > restriction or sanction. Please notify the sender, by > > > electronic > > > > > mail > > > > > > > or > > > > > > > > > telephone, of any unintended recipients and delete the > > original > > > > > > message > > > > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain > > > information > > > > > > intended for the exclusive use of the individual or entity to > whom > > it > > > > is > > > > > > addressed and may contain information that is proprietary, > > > confidential > > > > > > and/or exempt from disclosure under applicable law. If you are > not > > > the > > > > > > intended recipient, you are hereby notified that any viewing, > > > copying, > > > > > > disclosure or distribution of this information may be subject to > > > legal > > > > > > restriction or sanction. Please notify the sender, by electronic > > mail > > > > or > > > > > > telephone, of any unintended recipients and delete the original > > > message > > > > > > without making any copies. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > > intended for the exclusive use of the individual or entity to whom it > is > > > addressed and may contain information that is proprietary, confidential > > > and/or exempt from disclosure under applicable law. If you are not the > > > intended recipient, you are hereby notified that any viewing, copying, > > > disclosure or distribution of this information may be subject to legal > > > restriction or sanction. Please notify the sender, by electronic mail > or > > > telephone, of any unintended recipients and delete the original message > > > without making any copies. > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > intended for the exclusive use of the individual or entity to whom it is > > addressed and may contain information that is proprietary, confidential > > and/or exempt from disclosure under applicable law. If you are not the > > intended recipient, you are hereby notified that any viewing, copying, > > disclosure or distribution of this information may be subject to legal > > restriction or sanction. Please notify the sender, by electronic mail or > > telephone, of any unintended recipients and delete the original message > > without making any copies. > > > This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies.
