Hi Austen, Dmitri,
> Having getCloudflareR2Endpoint on AwsStorageConfigurationInfo
That does feel awkward. However, AwsStorageConfigurationInfo has supported
S3-compatible backends beyond AWS for a while, so there is precedent (e.g.,
MinIO, etc.). The underlying mismatch is that we call the storage type S3,
but its configuration mixes shared S3 settings with AWS-specific ones:
AwsStorageConfigurationInfo → storage type: S3
├── Common S3 settings
│ endpoint, region, pathStyleAccess, allowedLocations
└── AWS-specific settings
roleArn, externalId, STS, KMS
I’m open to separating the shared S3 settings from provider-specific
configuration if that helps, but I don’t think we need to settle on a
broader refactor now.
Let me know when the PR is ready for review. I’d be happy to take a closer
look.
Yufei
Yufei
On Mon, Sep 14, 2026 at 3:18 PM Dmitri Bourlatchkov <[email protected]>
wrote:
> Hi Austen,
>
> +1 to adding "credentialIssues" or similar (renamed) property to S3 config.
> In fact, I was looking exactly for something like that last week but the
> idea did not quite condense in my mind :)
>
> I posted some preliminary comments on the draft PR.
>
> Just for community awareness, I'd like to propose here that we use CDI for
> finding actual "issuer" / "mechanism" implementations. The codebase has
> many examples of this. The closest match in this case is
> probably PolarisStorageIntegrationProvider.
>
> This way the API does not have to enumerate all possible values. Downstream
> builds are free to add different implementations without having to change
> the API spec. The "allowed" list still makes sense.
>
> In general, I like the direction this feature is taking, although I expect
> PR reviews may take a few rounds :)
>
> Cheers,
> Dmitri.
>
> On Mon, Sep 14, 2026 at 3:38 PM Austen Tomek <
> [email protected]> wrote:
>
> > Hi all,
> >
> > Closing the loop since my last note and answering the questions that have
> > come up.
> >
> > Dmitri's question on role ARNs: R2 doesn’t have AWS-style IAM roles or
> > ARNs. Our implementation signs temporary credentials with a parent
> token’s
> > secret, scoped to a bucket and a set of prefixes. The account ID and
> > jurisdiction come from the R2 endpoint. Putting the account ID in roleArn
> > or userArn would give those fields a different meaning, so I’ve left them
> > absent for CLOUDFLARE_R2.
> >
> > I pursued Option B. The S3 configuration gains one new field,
> > credentialIssuer, with values STS
> > (default) and CLOUDFLARE_R2 and a realm allowlist
> > SUPPORTED_S3_CREDENTIAL_ISSUERS that defaults
> > to [STS]. I left stsUnavailable untouched (Prithvi) so it still means
> > "vend nothing". The
> > credentialIssuer is the explicit sign. The R2 config rejects
> > stsUnavailable along with a host of other STS and KMS fields. I'm running
> > this in our non-production environment at the moment.
> >
> > Draft PR for stage 1 (config field, API, allowlist, gates):
> > https://github.com/apache/polaris/pull/5513.
> >
> > I still see the appeal of a separate R2 configuration, as Dmitri
> > suggested. My original reason for separating it was to keep the
> R2-specific
> > configuration and validation together. Having getCloudflareR2Endpoint on
> > AwsStorageConfigurationInfo is one part I’d still like feedback on.
> >
> > I wanted to put a working version of Option B in front of everyone so we
> > could discuss the actual changes. In my earlier implementation, the
> > separate config also meant a separate storage type sharing S3FileIO. That
> > was the tradeoff that led me to try Option B.
> >
> > Do the diffs change anyone’s lean here? And are the names
> credentialIssuer
> > / STS / CLOUDFLARE_R2 / SUPPORTED_S3_CREDENTIAL_ISSUERS acceptable?
> >
> > Thanks,
> > Austen
> >
> >
> >
> > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
> > From: Dmitri Bourlatchkov <[email protected]>
> > Date: Sunday, September 13, 2026 at 19:50
> > To: [email protected] <[email protected]>
> > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential
> vending
> >
> > Building on top of Ayush's and JB's suggestion, I agree that jurisdiction
> > can live inside the existing endpoint config field.
> >
> > That leaves only Account ID, so potentially we could store it in user or
> > role ARN. Note that userArn is currently unused in OSS code.
> >
> > That should allow us to keep existing AwsStorageConfigurationInfo for all
> > S3 storage backend, but will require special parsing of user/role ARN
> field
> > to distinguish R2 from the other systems.
> >
> > I'm still leaning towards the separate R2StorageConfigurationInfo, but
> I'd
> > be ok with overloading user/role ARN too if other people prefer that. I
> > guess a key question in this context is whether R2 has the concept of
> role
> > ARN at all.... but TBH I have no idea about that. I hope Austen might be
> > able to clarify that.
> >
> > Cheers,
> > Dmitri.
> >
> > On Sun, Sep 13, 2026 at 9:18 AM Prithvi S <[email protected]>
> > wrote:
> >
> > > Hi Austen,
> > >
> > > Welcome, and thanks for starting with a [DISCUSS]. Best way to make a
> > first
> > > contribution :)
> > >
> > > I think the thread is talking about two layers, and we can keep both
> > > answers.
> > >
> > > On the client side this should stay S3: s3:// locations, s3.*
> properties,
> > > S3FileIO. Your PyIceberg / DuckDB / Iceberg Java results already show
> > that
> > > unmodified clients work, so a dedicated R2FileIO does not seem worth
> it.
> > > On the server side I lean toward Dmitri: a distinct
> > > R2StorageConfigurationInfo and storage integration. Issuance is not
> STS,
> > > and folding Cloudflare fields into AwsStorageConfigurationInfo mixes
> > > unrelated concepts. The middle ground to avoid is a new type that is
> > still
> > > AWS-shaped underneath.
> > >
> > > One concrete caution if the Option B prototype stays on the S3 config:
> > > please do not overload stsUnavailable. Today that flag skips AssumeRole
> > and
> > > disables vending. R2 wants the opposite. vend, just not via STS. so
> that
> > > flag would change meaning for existing MinIO / NetApp catalogs. If we
> > stay
> > > on S3 config, we need an explicit vending signal; a dedicated R2 type
> > > avoids that trap.
> > >
> > > On config surface, jurisdiction can live in endpoint (<account>.
> > > r2.cloudflarestorage.com vs <account>.eu.r2...). Account ID is still
> > > needed
> > > for the JWT sub, so keeping an explicit accountId and deriving audience
> > > from the endpoint seems enough. Parent tokens should stay in server
> > config
> > > via storageName.
> > > Local signing fits the vending model. Fail-closed on cross-bucket and
> > mixed
> > > read/write grants is the right call. Prefix claims should follow the
> same
> > > path comparison Polaris already uses, so we do not mint a credential
> > wider
> > > than the grant.
> > >
> > > +1 to small PRs: FileIO discriminator (#4486) separately if R2 needs
> it,
> > > then management API + Java config, then vending, Python CLI on its own.
> > > Sketching MinIO / Backblaze against the same fields is a useful design
> > > check; a generic non-STS SPI can wait for a second backend.
> > >
> > > Regards,
> > > Prithvi S
> > >
> > > On Fri, Sep 11, 2026 at 4:31 AM Dmitri Bourlatchkov <[email protected]>
> > > wrote:
> > >
> > > > Hi Austen,
> > > >
> > > > Thanks for starting this contribution!
> > > >
> > > > After briefly looking through your PR [1] I think
> > > > adding R2StorageConfigInfo makes sense.
> > > >
> > > > Even though "aws" in AwsStorageConfigurationInfo is longer implies
> AWS
> > > > services, and MinIO, Rust, Ozone are supported just as well, R2
> appears
> > > to
> > > > be significantly different from STS-based S3 systems that a separate
> > > config
> > > > is probably the most natural way to support it in the Polaris
> codebase.
> > > >
> > > > The alternative, I guess, is to add R2 accountId and jurisdiction as
> > > > optional properties to AwsStorageConfigurationInfo (or overload
> > existing
> > > > properties), but that looks like mixing unrelated concepts together.
> > > >
> > > > Adding a generic property bag to AwsStorageConfigurationInfo is not
> > > > convenient because PolarisStorageIntegrationProviderImpl will
> probably
> > > have
> > > > to know how to interpret it in order to redirect credential vending
> > > > requests to R2-specific code.
> > > >
> > > > I wonder whether R2StorageConfigurationInfo and
> > > AwsStorageConfigurationInfo
> > > > (and corresponding "storage integration" classes) could share a
> common
> > > base
> > > > class for common properties (e.g. endpoint). Even though "endpoint"
> > might
> > > > not be relevant to R2 in the cloud, the fact that S3FileIO uses it,
> and
> > > R2
> > > > is accessed via S3FileIO means we should probably allow users to
> > > configure
> > > > the full set of FileIO properties just in case.
> > > >
> > > > I think it is fine for different PolarisStorageIntegration
> > > implementations
> > > > to produce the same kind of StorageAccessConfig (S3 in this case) and
> > > thus
> > > > cause S3FileIO to be used by clients (including Polaris' own storage
> > > access
> > > > paths).
> > > >
> > > > Changes in StorageTypeFileIO in [1] look a bit concerning to me. I
> > wonder
> > > > if this code could be refactored to avoid dependencies on storage
> > config
> > > > types completely.... but TBH, I did not look too deeply into this
> > today.
> > > >
> > > > Anticipating future PRs for this in the Polaris repo, I'd like to ask
> > to
> > > > separate Python code changes from java changes since these areas
> > usually
> > > > attract different reviewers :)
> > > >
> > > > From my POV, management API and matching java changes can be in the
> > same
> > > > PR. I'd say docs should come later (to keep PRs small).
> > > >
> > > > All in all, thank you again and I hope this contribution lands in
> > Polaris
> > > > :)
> > > >
> > > > [1] https://github.com/deepdishgary/polaris/pull/1
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > >
> > > > On Thu, Sep 10, 2026 at 1:52 PM Jean-Baptiste Onofré <
> [email protected]>
> > > > wrote:
> > > >
> > > > > Hi Austen,
> > > > >
> > > > > Nice discussion! Thanks for that, and welcome aboard :)
> > > > >
> > > > > I think it makes sense to have R2 built under the existing S3
> storage
> > > > > configuration. It matches how Polaris is already structured:
> > > > > StorageType is keyed on the URI scheme (s3://, s3a://), and
> > > > > AwsStorageConfigurationInfo already carries endpoint, stsEndpoint,
> > > > > pathStyleAccess and, importantly, an stsUnavailable flag.
> > > > > So, what you need is largely there already: correct me if I'm
> wrong,
> > > > > R2 is essentially "S3 compatible store where STS is unavailable,
> plus
> > > > > a way to actually vend credentials in that case". Going this route
> > > > > also keeps this work "isolated" from the FileIO-per-storage-type
> > > > > discussion.
> > > > >
> > > > > The interesting design question is a pluggable credential vending
> > path
> > > > > for S3 compatible stores when STS is not available, with
> Cloudflare's
> > > > > local signing as the first concrete implementation, rather than R2
> > > > > specific code. It would be good to sketch how MinIO would slot into
> > > > > the same config fields before the vending path hardens around R2.
> > > > >
> > > > > I believe we have to check if Cloudflare account ID and
> jurisdiction
> > > > > really need dedicated config fields, or can we use the endpoint
> value
> > > > > (with documentation)? Keeping the configuration surface minimal
> will
> > > > > help the design generalize.
> > > > >
> > > > > I would love to discuss two things:
> > > > > 1. Polaris shipping a Cloudflare API token and locally minting
> scoped
> > > > > credentials makes it effectively an STS for R2, which increases the
> > > > > blast radius of a server compromise. Your "reject rather than
> > boarden"
> > > > > handling of unrepresentable scopes is the right call.
> > > > > 2. The PyIceberg/DuckDB/Java results are promising. Just curious
> > about
> > > > > Spark or Trino.
> > > > >
> > > > > We love small, well-scoped PRs. I would suggest to start with the
> S3
> > > > > config and the management API as first shoot, and we can follow
> with
> > > > > the credential vending implementation.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Wed, Sep 9, 2026 at 8:18 PM Austen Tomek
> > > > > <[email protected]> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I’m Austen Tomek from Chicago Trading Company. We use Polaris for
> > our
> > > > > Iceberg catalogs and we're evaluating Cloudflare R2 as an object
> > store
> > > > (the
> > > > > no egress fees make it a very compelling product). I’d like to
> > discuss
> > > > > contributing R2 support and credential vending to Polaris with the
> > goal
> > > > of
> > > > > getting feedback before opening a ready-for-review upstream PR.
> This
> > is
> > > > my
> > > > > first contribution, so trying to do this right.
> > > > > >
> > > > > > R2 exposes S3-compatible APIs but it does not provide AWS STS.
> > > > > Cloudflare instead supports these short-lived, scoped credentials
> > > > generated
> > > > > by locally signed JWT with the server holding the parent API token
> > [1].
> > > > > This allows us to retain Polaris’s authorization and
> > credential-vending
> > > > > model while storing our table data in R2.
> > > > > >
> > > > > > I have a prototype with the following design:
> > > > > >
> > > > > >
> > > > > > *
> > > > > > An opt-in R2 storage type, with catalog configuration identifying
> > the
> > > > > Cloudflare account and optional jurisdiction. Parent credentials
> > remain
> > > > in
> > > > > server configuration and can be selected through storageName.
> > > > > > *
> > > > > > Local credential generation using Cloudflare’s documented format.
> > > > > Credentials expire, are restricted to one bucket, and carry prefix
> > and
> > > > > access scopes derived from Polaris’s location grants. The
> integration
> > > > uses
> > > > > Polaris’s existing credential cache.
> > > > > > *
> > > > > > Credentials returned through the existing Iceberg REST vending
> flow
> > > as
> > > > > s3.* properties, including the endpoint, session token, and expiry.
> > > > > Compatible clients use their existing S3 FileIO.
> > > > > > *
> > > > > > Conservative handling of scopes the implementation cannot
> > represent:
> > > > > cross-bucket and mixed read/write grants are rejected rather than
> > > > > broadened. R2 is excluded from the default supported-storage-types
> > > list.
> > > > > >
> > > > > > The main design questions I feel this runs into:
> > > > > >
> > > > > > *
> > > > > > Should this be a distinct storage type or a credential-vending
> > > > mechanism
> > > > > selected through the existing S3 configuration?
> > > > > > *
> > > > > > I chose a separate type because the identity, credential
> > generation,
> > > > and
> > > > > scoping model differ from AWS STS, as this seemed more appropriate
> > > > > >
> > > > > > There is a draft preview PR against my fork [2], including
> > > > > implementation, management API changes, CLI support, tests, and
> > > > > documentation. It also discloses the AI assistance used during
> > > > > implementation. I leaned heavily on Fable 5.1 for this
> implementation
> > > so
> > > > I
> > > > > want to be upfront about it.
> > > > > >
> > > > > > We are running the implementation in a non-production environment
> > > right
> > > > > now as a POC. Validation includes live R2 reads/writes, multipart
> > > > > operations, negative scope-boundary tests, our internal Python
> client
> > > > > regression matrix, and credential rotation/recovery exercises. The
> > > > upstream
> > > > > CI workflow has passed on the fork.
> > > > > >
> > > > > > The change also encounters the existing assumption that a FileIO
> > > > > implementation identifies exactly one storage type: both S3 and R2
> > use
> > > > > S3FileIO. This overlaps with #4486 [3].
> > > > > >
> > > > > > I’d particularly appreciate feedback on:
> > > > > >
> > > > > >
> > > > > > 1.
> > > > > > Should R2 have its own storage configuration/type, or should
> > non-STS
> > > > > vending be modeled within the existing S3 integration?
> > > > > > 2.
> > > > > > Does the local-signing approach fit Polaris’s credential-vending
> > > model?
> > > > > Any additional constraints or validations necessary?
> > > > > > 3.
> > > > > > Would you prefer the shared FileIO validation change to land
> > > > separately,
> > > > > and how should we sequence the management API and R2 integration
> > > changes?
> > > > > >
> > > > > > Happy to adapt the design and split the contribution into more
> > > > focused
> > > > > PRs. Wanted to start the discussion
> > > > > >
> > > > > > Thanks,
> > > > > > Austen
> > > > > >
> > > > > > [1]
> > > > https://developers.cloudflare.com/r2/api/s3/temporary-credentials/
> > > > > > [2] https://github.com/deepdishgary/polaris/pull/1
> > > > > > [3] https://github.com/apache/polaris/issues/4486
> > > > > >
> > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > This electronic mail message and any attached files contain
> > > information
> > > > > intended for the exclusive use of the individual or entity to whom
> it
> > > is
> > > > > addressed and may contain information that is proprietary,
> > confidential
> > > > > and/or exempt from disclosure under applicable law. If you are not
> > the
> > > > > intended recipient, you are hereby notified that any viewing,
> > copying,
> > > > > disclosure or distribution of this information may be subject to
> > legal
> > > > > restriction or sanction. Please notify the sender, by electronic
> mail
> > > or
> > > > > telephone, of any unintended recipients and delete the original
> > message
> > > > > without making any copies.
> > > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
> >
> > This electronic mail message and any attached files contain information
> > intended for the exclusive use of the individual or entity to whom it is
> > addressed and may contain information that is proprietary, confidential
> > and/or exempt from disclosure under applicable law. If you are not the
> > intended recipient, you are hereby notified that any viewing, copying,
> > disclosure or distribution of this information may be subject to legal
> > restriction or sanction. Please notify the sender, by electronic mail or
> > telephone, of any unintended recipients and delete the original message
> > without making any copies.
> >
>
