Hi Dennis, Let me try to restate my concern from another angle. Apologies that I was not very clear in my previous messages.
Currently the create listing doc says: "Create a listing, which grants an external consumer access to this share". So, this binds authorization decisions about a Share (Listing) to ExternalConsumer. In IRC use cases a PolarisPrincipal is established by authenticating the client request (either via internal JWT or via external IdP), Subsequently, the PolarisPrincipal is passed to PolarisAuthorizer for making authorization decisions. The latter part can be external too (e.g. OPA / Ranger). (Side note: the "external" work in ExternalConsumer is actually quite confusing with respect to prior usage of that word in external IdP use cases.) In my view, the ExternalConsumer and related concepts in current Shares API do not align with the external IdP and Authorizer workflows. IIRC, actual access to shared tables is going to use the IRC API, right? That access path in existing Polaris use cases maps well to external IdP and external authorizers. Why does the Shares API need to define a "Consumer" as a separate entity and narrow authorization decisions to its relationship with Listings? I believe authentication and authorization options should remain as wide as possible because these matters are usually addressed very differently in each deployment. To be clear: I do not mind defining Share "Consumers" in the internal Polaris RBAC domain. It can fit well there, I'm only suggesting keeping the Shares API itself neutral to Auth N/Z concerns. Cheers, Dmitri. On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <[email protected]> wrote: > Ah thanks for clarifying - to be clear, the ConsumerCredential is *not* > strictly necessary for the Listing in terms of spec expressibility, only > the ExternalConsumer is. > > An ExternalConsumer doesn't need to be backed by a ConsumerCredential; the > ExternalConsumer is an abstract Sharing-specific pointer to whatever does > the indirection in the sharing implementation. > > Responded to your github comment - I'll split anything that touches > "ConsumerCredential" into a separate yaml file so it's easier to see the > core auth-agnostic sharing spec in one file, with the more > implementation-specific decorations around ConsumerCredential in an > auxiliary file. > > I suppose another option is to even split it into separate PRs (in addition > to separate yaml files?). Then we can easily see one commit that's just the > pure abstract auth-agnostic API, that in theory private Polaris > implementations could already take and implement on external-IdP > architectures to support the same agnostic spec. > > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Dennis, > > > > While I still support the idea of splitting OpenAPI yaml files and URI > > bases between the Management and Sharing APIs, I do not think it is > tightly > > coupled with my proposal for the authentication-neural Sharing API spec. > > > > To give an example, currently [5446] CreateListingRequest includes a > > "consumerName". This immediately binds the Listing definition to > > an ExternalConsumer and by extension to ConsumerCredential. > > > > ConsumerCredential is naturally part of the authentication process during > > Share access. I believe ConsumerCredential (and related entities) is > worth > > isolating from the Shares API to increase interoperability. > > > > Implementations that choose to rely on ConsumerCredential for Share > access > > can still do so. However, ConsumerCredential will not be a requirement > for > > all possible Share API implementations. > > > > This can work with any OpenAPI yaml and URI paths layout. > > > > I'd like to resolve this aspect before we go back to URI paths and YAML > > file locations. Actually, I do not feel too strongly about path > prefixes. I > > do believe separate prefixes make implementations easier in many aspects > > (including network setup), but interleaving those paths with the > Management > > API is not a blocker, from my POV, only an inconvenience. > > > > Cheers, > > Dmitri. > > > > [5446] https://github.com/apache/polaris/pull/5446 > > > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <[email protected]> wrote: > > > > > Yeah, that's a good topic to discuss; I guess it touches on our core > > > (unresolved?) conventions on how we organize API url-prefixes and yaml > > > files. Reposting my response to the github comment as well: > > > > > > Right, I think we could certainly split the API routes and > > yaml-definitions > > > even further if we want to ensure a well-defined auth-detail-free core, > > but > > > for the standalone-Polaris use case we'd still need to define it > > > *somewhere* > > > under a Sharing-specific surface, right? > > > > > > @dimas-b <https://github.com/dimas-b> Were you thinking of just > > splitting > > > the yaml file but keeping the share-management root, or adding a third > > > sharing prefix like > > > > > > > > > /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials... > > > ? > > > > > > Or just yaml-file split would be keeping > > > /api/shares-management/v1/external-consumers/{name}/credentials... but > > > putting it in a separate yaml file to make it easy for service-runners > to > > > choose not to support that whole yaml file's concrete APIs. > > > > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > Re: the shating API PR [5446], I'm repeating my GH comment here for > > > > visibility. > > > > > > > > I'd like the API to be neutral from the authentication perspective. > > > > > > > > This is how we handle the IRC spec in Polaris. The API declares that > > > > clients need to authenticate, but specific details about principals, > > > > clients and secrets are delegated to the server. > > > > > > > > I think this is a sound approach for the sharing API too. Specific > > > details > > > > related to native Polaris principals (ConsumerCredential, etc.) can > > > > probably remain in the Management API. > > > > > > > > WDYT? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > [5446] https://github.com/apache/polaris/pull/5446 > > > > > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré < > [email protected]> > > > > wrote: > > > > > > > > > Hi > > > > > > > > > > As I said during the community meeting, let's focus on the API PR > > > > > first. We will build from there. > > > > > > > > > > Regards > > > > > JB > > > > > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <[email protected]> wrote: > > > > > > > > > > > > +1, thanks for the discussion everyone! > > > > > > > > > > > > Yes, I meant to flip it to "ready to review" earlier; flipped it > > now. > > > > I'm > > > > > > still working through the existing set of comments and may need > to > > > come > > > > > > back to it in the next few days before getting proper replies, > but > > > feel > > > > > > free to add additional comments. > > > > > > > > > > > > Recapping the community sync discussion re: Dmitri's last > message, > > > I'm > > > > > > fully aligned on building the sharing on top of existing authz > > > > > mechanisms, > > > > > > instead of being a new mechanism, and indeed external IdP is > > intended > > > > to > > > > > be > > > > > > a first-class supported use case. > > > > > > > > > > > > The latest distinction is that the ExternalConsumer isn't > > > fundamentally > > > > > > identified by any internal PolarisPrincipal, but rather that it > > > serves > > > > > as a > > > > > > wrapper that references some identity, whether that identity is > > > defined > > > > > in > > > > > > an external IdP without any latent in-Polaris representation or > if > > > > it's a > > > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use > case). > > > > > > > > > > > > The sharing endpoint via ExternalConsumer is the only layer of > > > > > indirection > > > > > > at the API entry-point, but then once the set of > > activePrincipalRoles > > > > is > > > > > > resolved, the underlying authz engine doesn't behave any > > differently > > > > from > > > > > > normal Polaris authz. > > > > > > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <[email protected]> > > > wrote: > > > > > > > > > > > > > Thanks everyone for the discussion! > > > > > > > > > > > > > > As a next step discussed in today's community sync, we will > start > > > > > reviewing > > > > > > > the spec change. > > > > > > > > > > > > > > Hi Dennis, is this PR ready for review? > > > > > > > https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for this discussion Dennis! > > > > > > > > > > > > > > > > Wanted to weigh in with my thoughts and questions. > > > > > > > > > > > > > > > > > > > > > > > > > > Do you prefer /api/shares-management/v1/ > > > > > > > > > instead of adding to /api/management/v1? > > > > > > > > > > > > > > > > > > > > > > > > While I can see the argument for a share specific management > > > > endpoint > > > > > > > there > > > > > > > > doesn't seem to be a clear obvious choice from what I can see > > - I > > > > > lean > > > > > > > > towards /api/management as being the central place for all > the > > > > > > > management - > > > > > > > > share or not. As long as we're careful not to leak share > types > > or > > > > non > > > > > > > share > > > > > > > > types into the wrong api (which could be helped by having > > > separate > > > > > yamls) > > > > > > > > then it feels reasonable to have all under the management > api. > > > > > > > > > > > > > > > > Also went through the doc and PR and made several comments as > > > well. > > > > > > > > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > Prithvi mentioned ExternalConsumer and principals. I'd like > > to > > > > > > > > > reinforce the need to further discuss "consumers" and > > > principals > > > > > and > > > > > > > how > > > > > > > > > they relate to IdP and Polaris Authentication. > > > > > > > > > > > > > > > > > > From my POV, share access control should not be a new > > mechanism > > > > but > > > > > > > > should > > > > > > > > > build on top of the existing extensible authentication / > > > > > authorization > > > > > > > > > framework. Supporting external IdP for shares would be a > > > valuable > > > > > > > feature > > > > > > > > > IMHO. > > > > > > > > > > > > > > > > > > Side note: I'm posting comments on PR 5446 too. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > Thank you for reviving this, and for the concrete spec in > > PR > > > > > 5446.. > > > > > > > the > > > > > > > > > > journeys doc was especially helpful for seeing the > > end-to-end > > > > > flow. > > > > > > > > > > > > > > > > > > > > I agree this is worth doing, and that v1 is more than a > > > > tutorial > > > > > on > > > > > > > > > today's > > > > > > > > > > RBAC. A partner principal on /api/catalog is already > > > possible; > > > > > what > > > > > > > > seems > > > > > > > > > > valuable is a first-class share (enumerated members, a > > > > restricted > > > > > > > > > consumer, > > > > > > > > > > a listing as the binding and audit unit) plus a > segregated, > > > > > read-only > > > > > > > > > > Iceberg REST surface with credential vending. Stock > Iceberg > > > > REST > > > > > as > > > > > > > the > > > > > > > > > > consumer protocol, and keeping Flight and > > Polaris-to-Polaris > > > > > > > federation > > > > > > > > > out > > > > > > > > > > of v1, all sound right to me. > > > > > > > > > > > > > > > > > > > > I don't have a strong view yet on the URL layout > > (management > > > vs > > > > > > > shares > > > > > > > > > > prefix, nested vs top-level, prefix keyed by share vs > > > listing) > > > > > and am > > > > > > > > > happy > > > > > > > > > > to follow whatever the community prefers there. > > > > > > > > > > > > > > > > > > > > A few things I wanted to check on the spec itself, > several > > of > > > > > these > > > > > > > > echo > > > > > > > > > > points already raised on the list: > > > > > > > > > > > > > > > > > > > > 1. Membership as source of truth. If a hidden catalog > role > > is > > > > > > > > > independently > > > > > > > > > > grantable, a share could be widened without going through > > the > > > > > share > > > > > > > > APIs. > > > > > > > > > > Would it make sense to say the data plane authorizes > > against > > > > > > > > membership + > > > > > > > > > > listing, and that any backing roles are not user-visible > / > > > not > > > > > > > > > > independently grantable? > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap > already > > > > > noted we > > > > > > > > > should > > > > > > > > > > not assume a PrincipalEntity behind every > ExternalConsumer. > > > The > > > > > > > > journeys > > > > > > > > > > still describe granting the consumer a share principal > > role, > > > I > > > > > assume > > > > > > > > > that > > > > > > > > > > is an implementation sketch for v1 client_id/secret, not > > the > > > > > public > > > > > > > > > model? > > > > > > > > > > 3. Listing expiration. JB's original note was "share > these > > > > tables > > > > > > > with > > > > > > > > > this > > > > > > > > > > consumer until this date." Credential expiry is a bit > > > > different. > > > > > > > Would > > > > > > > > a > > > > > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be > in > > > > > scope for > > > > > > > > v1? > > > > > > > > > > 4. Time-travel. As raised in the community sync, sharing > > via > > > > > Iceberg > > > > > > > > REST > > > > > > > > > > means historical snapshots remain readable. Totally fine > > for > > > v1 > > > > > to > > > > > > > > > support > > > > > > > > > > only full history, would an explicit field on the share > > help > > > > so a > > > > > > > later > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise? > > > > > > > > > > 5. Consumer data plane. The current file is control-plane > > > only, > > > > > which > > > > > > > > is > > > > > > > > > a > > > > > > > > > > good first step. It might help to list the Iceberg routes > > > that > > > > > are > > > > > > > > > mounted, > > > > > > > > > > that writes 404, that non-members 404, and that /config > > does > > > > not > > > > > > > return > > > > > > > > > > provider catalog properties, so implementations don't > > > diverge. > > > > > > > > > > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey the > > consumer > > > > > still > > > > > > > > posts > > > > > > > > > to > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not > supposed > > to > > > > > touch > > > > > > > the > > > > > > > > > > internal catalog API, should the default token endpoint > > live > > > on > > > > > the > > > > > > > > > shares > > > > > > > > > > surface (or an external IdP)? And on Dmitri's point about > > > share > > > > > > > > operators > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the > bound > > > > > catalog > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that > work > > > on > > > > > any > > > > > > > URL > > > > > > > > > > prefix. > > > > > > > > > > > > > > > > > > > > Happy to review the next revision, and thanks again to > both > > > of > > > > > you > > > > > > > for > > > > > > > > > > driving this. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > As discussed live, looks like we both might have some > > > drafts > > > > > we can > > > > > > > > > > compare > > > > > > > > > > > and choose/merge from :) > > > > > > > > > > > > > > > > > > > > > > Here's my current draft spec PR; it opts for my > preferred > > > > > syntax > > > > > > > > > choices > > > > > > > > > > > for now (/api/management/ for share-management, > > > /api/shares/ > > > > > for > > > > > > > > > consumer > > > > > > > > > > > data-plane): > https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > > > > > > > > > Since we were talking about preferring mdfiles for > design > > > > > iteration > > > > > > > > > > here's > > > > > > > > > > > a companion design-addendum which just enumerates what > > the > > > > > > > end-to-end > > > > > > > > > > user > > > > > > > > > > > journeys look like syntactically for each combination > of > > > > design > > > > > > > > > choices: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md > > > > > > > > > > > > > > > > > > > > > > Please let me know if anyone prefers a Google Doc > version > > > of > > > > > the > > > > > > > > > > > design-addendum for easier commenting, or if the > > > > > design-addendum > > > > > > > > itself > > > > > > > > > > > should also be modeled as a PR on some free-floating > > branch > > > > > just to > > > > > > > > > > capture > > > > > > > > > > > comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > I am resuming work on this proposal. > > > > > > > > > > > > > > > > > > > > > > > > Regarding our recent discussion, I will move forward > > with > > > > > > > opening a > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to help > > > illustrate > > > > > the > > > > > > > > > > > > design. > > > > > > > > > > > > > > > > > > > > > > > > I will update this thread once the PR is ready. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Dennis, > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the recap! > > > > > > > > > > > > > > > > > > > > > > > > > > I was thinking that share management is probably > > > distinct > > > > > from > > > > > > > > > > catalog > > > > > > > > > > > > > management. A user who already has a realm+catalog > > may > > > > > have an > > > > > > > > > option > > > > > > > > > > > to > > > > > > > > > > > > > manage shares, but not the catalog itself. Would it > > > make > > > > > sense > > > > > > > to > > > > > > > > > > > > separate > > > > > > > > > > > > > the share management API from the general Polaris > > > > > management > > > > > > > > API? I > > > > > > > > > > > mean > > > > > > > > > > > > > using a different prefix for URIs. > > > > > > > > > > > > > > > > > > > > > > > > > > For example: /api/shares/v1/ > > > > > > > > > > > > > > > > > > > > > > > > > > Obviously clients should not make any assumptions > > about > > > > the > > > > > > > base > > > > > > > > > path > > > > > > > > > > > > > (/api/shares), which could be different in > different > > > > > > > deployments > > > > > > > > > > (e.g. > > > > > > > > > > > > > going through a gateway). Clients must be able to > > take > > > > the > > > > > base > > > > > > > > URI > > > > > > > > > > as > > > > > > > > > > > a > > > > > > > > > > > > > config option. Path segments after /v1/ will be > > > specified > > > > > by > > > > > > > > > Polaris > > > > > > > > > > > > (Open > > > > > > > > > > > > > API yaml). > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Just to recap some of the discussion from today's > > > > > community > > > > > > > > sync, > > > > > > > > > > > > Dmitri, > > > > > > > > > > > > > > Robert, and Alex helped raise a few > considerations > > to > > > > > roll > > > > > > > into > > > > > > > > > the > > > > > > > > > > > > next > > > > > > > > > > > > > > design/prototyping phase: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Given the way sharing is packaged as a > separate > > > > > thing, it > > > > > > > > may > > > > > > > > > > not > > > > > > > > > > > be > > > > > > > > > > > > > > obvious to users that historical snapshots are > > > equally > > > > > > > > available > > > > > > > > > to > > > > > > > > > > > > > > consumers as the latest snapshot (by virtue of > > > sharing > > > > > via > > > > > > > the > > > > > > > > > > > > standard IRC > > > > > > > > > > > > > > protocol) - we should make sure to document that > > fact > > > > > well, > > > > > > > > > > including > > > > > > > > > > > > the > > > > > > > > > > > > > > pitfall that deleting rows by updating a table > > > doesn't > > > > > > > > > necessarily > > > > > > > > > > > mean > > > > > > > > > > > > > > those rows become inaccessible to consumers > > > > > > > > > > > > > > 2. Make sure not to hard-code assumptions about > the > > > > > existence > > > > > > > > of > > > > > > > > > an > > > > > > > > > > > > > > internal PrincipalEntity backing an > > ExternalConsumer > > > - > > > > > even > > > > > > > if > > > > > > > > > the > > > > > > > > > > > > first > > > > > > > > > > > > > > MVP implementation uses a nested/hidden Principal > > of > > > > some > > > > > > > type > > > > > > > > > that > > > > > > > > > > > > holds a > > > > > > > > > > > > > > Polaris-managed client_id/client_secret, the > > > 3rd-party > > > > > IdP > > > > > > > use > > > > > > > > > > cases > > > > > > > > > > > > > > require "implicit" Principals resolved via the > > oauth > > > > > token. > > > > > > > In > > > > > > > > > such > > > > > > > > > > > > cases > > > > > > > > > > > > > > the ExternalConsumer contains the metadata > > > coordinating > > > > > the > > > > > > > > > > three-way > > > > > > > > > > > > > > handshake between the data-provider, > data-consumer, > > > and > > > > > IdP, > > > > > > > > but > > > > > > > > > > > there > > > > > > > > > > > > is > > > > > > > > > > > > > > no PrincipalEntity underlying it. > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle > > "on-behalf-of" > > > > > > > semantics > > > > > > > > - > > > > > > > > > > > > initially, > > > > > > > > > > > > > > the way an ExternalConsumer works is that the > > > > > finer-grained > > > > > > > > > > consumer > > > > > > > > > > > > users > > > > > > > > > > > > > > share a single ExternalConsumer identity, just > like > > > > how a > > > > > > > > > FEDERATED > > > > > > > > > > > > catalog > > > > > > > > > > > > > > shares a single service principal in its > > > > > ConnectionConfig. > > > > > > > But > > > > > > > > in > > > > > > > > > > > some > > > > > > > > > > > > > > cases the consumer may want some end-to-end > > > attribution > > > > > of > > > > > > > > > > > fine-grained > > > > > > > > > > > > > > users, like "on-behalf-of: bob @ > > consumercompany.com > > > " > > > > > when > > > > > > > the > > > > > > > > > > > > > > ExternalConsumer implicit principal is something > > like > > > > > > > > > > > > > > "externalconsumer$consumercompany" > > > > > > > > > > > > > > 4. We can iterate more rapidly on a draft spec PR > > and > > > > > also > > > > > > > > expand > > > > > > > > > > on > > > > > > > > > > > > the > > > > > > > > > > > > > > detailed technical design > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo < > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks JB for reviving this thread, and belated > > > > thanks > > > > > to > > > > > > > all > > > > > > > > > > > > reviewers > > > > > > > > > > > > > > > who have left feedback in the doc or this > thread > > so > > > > > far! > > > > > > > And > > > > > > > > > > > > apologies > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > the silence on this, it's still been on my > radar > > > but > > > > I > > > > > got > > > > > > > > > > > > distracted by > > > > > > > > > > > > > > > some other work for awhile. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed > that > > > > we're > > > > > > > > mostly > > > > > > > > > > just > > > > > > > > > > > > > > > reusing existing Authz/RBAC machinery, and the > > > > > translation > > > > > > > > > layer > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > sharing concepts sits *on top of* the existing > > > authz > > > > > engine > > > > > > > > as > > > > > > > > > > much > > > > > > > > > > > > as > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So even if we're adding layers of indirection > on > > > > > > > > > > > > ExternalConsumer/Listing > > > > > > > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, > > the > > > > > parts > > > > > > > that > > > > > > > > > end > > > > > > > > > > > up > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > the underlying authz just reuse existing > > Authorizer > > > > > logic. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Based on some other offline discussion with > > > Dmitri, I > > > > > > > > realized > > > > > > > > > > why > > > > > > > > > > > > the > > > > > > > > > > > > > > > discussion around "requirements" may have been > > > > > confusing: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently > > cut > > > > out > > > > > > > some > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > > more > > > > > > > > > > > > > > > "abstract requirements" formulations > > > > > > > > > > > > > > > 2. The project in itself is somewhat more of a > > > "form > > > > > *as* > > > > > > > > > > function" > > > > > > > > > > > > > > > feature than *strictly* functional in nature -- > > for > > > > > > > example, > > > > > > > > > many > > > > > > > > > > > > > > correctly > > > > > > > > > > > > > > > pointed out that "sharing" via the existing > > > > > RBAC/Principals > > > > > > > > > > syntax > > > > > > > > > > > is > > > > > > > > > > > > > > > already *possible* just by letting a cross-org > > > > > "consumer" > > > > > > > be > > > > > > > > a > > > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > principal and access your own company's main > > > Polaris > > > > > > > catalog > > > > > > > > > > > > endpoints. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For (1) I added this to the Requirements > section > > in > > > > > the doc > > > > > > > > to > > > > > > > > > > > better > > > > > > > > > > > > > > > clarify the abstract form of the data model > > > element: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A. Sharing Container*: Container concept for > > > holding > > > > > > > things > > > > > > > > > you > > > > > > > > > > > > want to > > > > > > > > > > > > > > > share - “ShareEntity” > > > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of > > > > > principal-like > > > > > > > > entity > > > > > > > > > > > > > > > representing a consumer relationship - > > > > > “ExternalConsumer” > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way to > > specify > > > > > letting > > > > > > > a > > > > > > > > > > given > > > > > > > > > > > > > > > consumer access a given sharing container - > > > “Listing” > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API > > > access-point > > > > > for > > > > > > > > > > > consumers - > > > > > > > > > > > > > > > “EndpointConfig” > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to specify > > which > > > > > > > in-catalog > > > > > > > > > > > objects > > > > > > > > > > > > > > > belong in the big sharing container - > > > > “ShareMembership” > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This still bakes in a bit of implementation > > detail > > > > > since it > > > > > > > > > begs > > > > > > > > > > > the > > > > > > > > > > > > > > > question why not just let core RBAC/grants > handle > > > (C) > > > > > and > > > > > > > > (E). > > > > > > > > > To > > > > > > > > > > > > that > > > > > > > > > > > > > > end, > > > > > > > > > > > > > > > and addressing problem (2) of where the "form > > *is* > > > > > > > > function", I > > > > > > > > > > > > added a > > > > > > > > > > > > > > > "Concept Mapping" tab: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think the details of more advanced features > can > > > > > still be > > > > > > > > > hashed > > > > > > > > > > > out > > > > > > > > > > > > > > > incrementally without blocking work on an MVP > > (and > > > > > > > certainly > > > > > > > > > the > > > > > > > > > > > MVP > > > > > > > > > > > > > > scope > > > > > > > > > > > > > > > could be trimmed so we can launch-and-iterate), > > so > > > as > > > > > long > > > > > > > as > > > > > > > > > > > there's > > > > > > > > > > > > > > > community alignment on the general direction I > > > > propose > > > > > we > > > > > > > > could > > > > > > > > > > > move > > > > > > > > > > > > > > > forward on basic building blocks. Or if > building > > > > blocks > > > > > > > > > > themselves > > > > > > > > > > > > are > > > > > > > > > > > > > > > controversial we could still proceed with some > > > > > prototyping > > > > > > > > > across > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > options. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste > > > > Onofré < > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi everyone, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> I would like to resume the discussion about > Data > > > > > Sharing > > > > > > > in > > > > > > > > > > > Polaris > > > > > > > > > > > > > > >> and propose focusing on three major items to > > help > > > us > > > > > move > > > > > > > > > > forward. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> 1. What is data sharing? > > > > > > > > > > > > > > >> In a nutshell, it is an open protocol > > proposal > > > > > > > allowing > > > > > > > > > > > > > > >> organizations to share live data without > copying > > > or > > > > > > > > > replication. > > > > > > > > > > > The > > > > > > > > > > > > > > >> main goal is zero-copy sharing achieved > through > > > > shared > > > > > > > > > metadata > > > > > > > > > > > and > > > > > > > > > > > > > > >> authorization. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> On Polaris, this would mean: > > > > > > > > > > > > > > >> a. The consumer is authenticated as usual. > > > > > > > > > > > > > > >> b. The Polaris server provides the consumer > with > > > > > direct > > > > > > > > access > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > >> data files (e.g., Parquet, as specified in the > > > > > > > > metadata.json). > > > > > > > > > > > > > > >> c. The consumer reads the data files directly > > > using > > > > > their > > > > > > > > > > > preferred > > > > > > > > > > > > > > >> tool (e.g., a query engine). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> While Polaris already has most of the > necessary > > > > > underlying > > > > > > > > > > > > > > >> capabilities, we need clear branding/packaging > > > > around > > > > > this > > > > > > > > use > > > > > > > > > > > case. > > > > > > > > > > > > > > >> Specifically, this means: > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in the > Polaris > > > > > server, > > > > > > > or > > > > > > > > a > > > > > > > > > > > > catalog > > > > > > > > > > > > > > >> role scoped to specific namespaces. > > > > > > > > > > > > > > >> b. A recipient is represented by a principal > > and a > > > > > > > principal > > > > > > > > > > role. > > > > > > > > > > > > > > >> c. A consumer profile consists of the catalog > > URI > > > > and > > > > > the > > > > > > > > > OAuth2 > > > > > > > > > > > > > > >> client_id/client_secret. > > > > > > > > > > > > > > >> d. We use credential vending (temporary STS > > > > > credentials) > > > > > > > for > > > > > > > > > > > > consumer > > > > > > > > > > > > > > >> access to the data files. > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to expose the > > > > > metadata and > > > > > > > > the > > > > > > > > > > > file > > > > > > > > > > > > > > >> locations/access. > > > > > > > > > > > > > > >> f. We can also support Polaris-to-Polaris > > sharing > > > > via > > > > > > > > > > Federation. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Concretely: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> - On the provider side, we use the standard > > > > Polaris > > > > > RBAC > > > > > > > > > model > > > > > > > > > > > > > > >> (assigning grants to catalog roles, and > > assigning > > > > > those > > > > > > > > > catalog > > > > > > > > > > > > roles > > > > > > > > > > > > > > >> to principal roles): > > > > > > > > > > > > > > >> - Create a dedicated principal with its > > own > > > > > > > > > > > > > > client_id/client_secret. > > > > > > > > > > > > > > >> - Create a catalog role with > > > TABLE_READ_DATA, > > > > > > > > > TABLE_LIST, > > > > > > > > > > > and > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be > > > shared > > > > > (no > > > > > > > > write > > > > > > > > > > > > > > >> grants). > > > > > > > > > > > > > > >> - Create a principal role per > destination, > > > > > attach > > > > > > > the > > > > > > > > > > > catalog > > > > > > > > > > > > > > >> role, and assign the principal. > > > > > > > > > > > > > > >> - The destination engine uses loadTable > > with > > > > the > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation: > > vended-credentials" > > > > > header. > > > > > > > > > > Polaris > > > > > > > > > > > > then > > > > > > > > > > > > > > >> emits temporary, read-only storage > credentials, > > > > > allowing > > > > > > > the > > > > > > > > > > > engine > > > > > > > > > > > > to > > > > > > > > > > > > > > >> read the data files directly from the owner's > > > bucket > > > > > > > without > > > > > > > > > > > > accessing > > > > > > > > > > > > > > >> the rest of the bucket. This achieves > zero-copy > > > > > sharing. > > > > > > > > > > > > > > >> - On the recipient/consumer side, no special > > > > > > > configuration > > > > > > > > > is > > > > > > > > > > > > > > >> needed; any IRC client can use it. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> 2. What is data sharing not? > > > > > > > > > > > > > > >> I previously considered adding an Arrow > > Flight > > > > > > > endpoint > > > > > > > > to > > > > > > > > > > > > > > >> Polaris, but I no longer think it is a good > idea > > > for > > > > > the > > > > > > > > > initial > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would serve > the > > > data > > > > > > > > directly, > > > > > > > > > > > which > > > > > > > > > > > > > > >> would require Polaris to read the data and > > expose > > > > it. > > > > > I am > > > > > > > > > > against > > > > > > > > > > > > > > >> embedding a query engine in Polaris, as this > > tight > > > > > > > coupling > > > > > > > > > > feels > > > > > > > > > > > > like > > > > > > > > > > > > > > >> an anti-pattern. While we could use a > delegation > > > > > service > > > > > > > to > > > > > > > > > > serve > > > > > > > > > > > > the > > > > > > > > > > > > > > >> Flight endpoint, I think it should be > considered > > > > for a > > > > > > > later > > > > > > > > > > > phase, > > > > > > > > > > > > if > > > > > > > > > > > > > > >> at all, since a Flight endpoint might be > better > > > > > suited as > > > > > > > a > > > > > > > > > > > consumer > > > > > > > > > > > > > > >> responsibility and could encourage data > copying. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> The first phase should focus on packaging and > > > > > branding the > > > > > > > > > > > > > > >> capabilities we already have. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> 3. What could the Data Sharing "package" look > > > like > > > > in > > > > > > > > > Polaris? > > > > > > > > > > > > > > >> To make Data Sharing obvious and easy to > > use, > > > we > > > > > > > should > > > > > > > > > > > > introduce > > > > > > > > > > > > > > >> the missing concept of a "share"—a named > > > definition > > > > > that > > > > > > > > users > > > > > > > > > > can > > > > > > > > > > > > > > >> list, discover, and define. The goal is to > > > > > materialize the > > > > > > > > > > user's > > > > > > > > > > > > > > >> intent ("I want to share these tables with > this > > > > > consumer > > > > > > > > until > > > > > > > > > > > this > > > > > > > > > > > > > > >> date") into standard Polaris entities > > (principal, > > > > > > > principal > > > > > > > > > > role, > > > > > > > > > > > > > > >> catalog role, and grants). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Therefore, I propose adding a /shares endpoint > > to > > > > the > > > > > > > > Polaris > > > > > > > > > > > > > > >> Management API to package and materialize > these > > > > share > > > > > > > > > > definitions > > > > > > > > > > > > into > > > > > > > > > > > > > > >> Polaris entities. This would provide a usable, > > > > > > > first-version > > > > > > > > > > > package > > > > > > > > > > > > > > >> for Polaris Data Sharing. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> I hope this addresses Dmitri's questions and > > > > provides > > > > > a > > > > > > > more > > > > > > > > > > > > concrete > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Thoughts? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Regards, > > > > > > > > > > > > > > >> JB > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Hi All, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > One big emerging enterprise use case coming > up > > > as > > > > > more > > > > > > > > > people > > > > > > > > > > > > > > >> consolidate > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is something > > > commonly > > > > > known > > > > > > > > as > > > > > > > > > > > "Data > > > > > > > > > > > > > > >> Sharing", > > > > > > > > > > > > > > >> > an more specifically over the course of > > adoption > > > > of > > > > > open > > > > > > > > > table > > > > > > > > > > > > formats > > > > > > > > > > > > > > >> > "Open Data Sharing". > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Examples of existing managed service > > providers' > > > > Data > > > > > > > > Sharing > > > > > > > > > > > > features: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > https://www.databricks.com/product/delta-sharing > > > > > > > > > > > > > > >> > > > > > > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > The basic idea is that when you share data > > > between > > > > > > > > different > > > > > > > > > > > > > > companies, > > > > > > > > > > > > > > >> you > > > > > > > > > > > > > > >> > need a first-class governance/management > layer > > > and > > > > > extra > > > > > > > > > > > > > > >> bells-and-whistles > > > > > > > > > > > > > > >> > that are distinct from just the basic > > > capabilities > > > > > of > > > > > > > RBAC > > > > > > > > > or > > > > > > > > > > > > > > >> generalized > > > > > > > > > > > > > > >> > access-control (i.e. if you're sharing > across > > > > > > > > > > > partially-untrusted > > > > > > > > > > > > org > > > > > > > > > > > > > > >> > boundaries, you don't just let the consumer > > > > > organization > > > > > > > > log > > > > > > > > > > > into > > > > > > > > > > > > your > > > > > > > > > > > > > > >> > datalake like one of your own employees). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > JB and I put together this high-level > proposal > > > for > > > > > > > > > supporting > > > > > > > > > > > Open > > > > > > > > > > > > > > >> Sharing > > > > > > > > > > > > > > >> > in Polaris: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5 logical data > > > model > > > > > > > > > constructs, > > > > > > > > > > > > some of > > > > > > > > > > > > > > >> > which may be a first-class PolarisEntity > type, > > > > > others > > > > > > > > > subtypes > > > > > > > > > > > of > > > > > > > > > > > > > > >> existing > > > > > > > > > > > > > > >> > entities, and others just a nested > construct: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > - ShareEntity (would behave similarly to > a > > > > > Catalog) > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits from > > > > > Principal) > > > > > > > > > > > > > > >> > - Listing (Similar to a "role grant" but > > has > > > > > > > different > > > > > > > > > > > > metadata) > > > > > > > > > > > > > > >> > - EndpointConfig (nested config under > > > Listing) > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a > "securable > > > > > grant" but > > > > > > > > > > > different > > > > > > > > > > > > > > >> metadata) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also bring > it > > up > > > > for > > > > > > > live > > > > > > > > > > > > discussion > > > > > > > > > > > > > > if > > > > > > > > > > > > > > >> > there's time in the community sync. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Cheers, > > > > > > > > > > > > > > >> > Dennis > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
