Makes sense, thanks! I've gone ahead with the new split in the PR: 1. polaris-shares-api.yaml - Shares and ShareMembership for adding securables to the Share 2. polaris-shares-consumer-management-api.yaml - Consumer/Listing/Credentials management
We can make sure the SPIs allow the core Shares part to operate independently from the consumer-management side. On Thu, Oct 22, 2026 at 4:27 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Dennis, > > Thanks for the detailed path forward. > > The stages / compartmentalized approach make sense to me. > > 1) The Shares API defines "what to share". It does not depend on anything > else. > > 2) The Authorizer SPI changes provide a way to express Share access > requests. > > 3) Internal RBAC implements (2) on top of the internal Principals and > Roles. > > I suppose ExternalConsumer will fit here. > > 4) External Authorizer implementations adapt to (2) and allow users to > configure access rules in their respective configuration styles. > > Does that make sense? > > Thanks, > Dmitri. > > On Thu, Oct 22, 2026 at 6:58 PM Dennis Huo <[email protected]> wrote: > > > Right, sorry, to clarify, my stated bullets in the last email were my > > hypothetical portrayal of what your proposed core set of APIs would be, > not > > what my current proposal says (so that you can confirm or deny whether > > those three bullets convey your ideal state). > > > > It sounds like (2) and (3) match your intent, and (1) is exactly where > the > > current spec PR differs, since it contains consumers and listings. > > > > For the particular use cases that went into the scoping for the Polaris > > Shares proposal, the ExternalConsumer is fairly fundamental to extracting > > as a first-class object. > > > > But you're right that there could be a trimmed scoping that allows > > implementors to fully abstract away the ExternalConsumer/Listing > semantics > > instead. In the same way, there's even a fully-trimmed formulation where > we > > don't implement *any* management APIs at all: Where "Sharing" is just > data > > providers providing dedicated lightweight Polaris servers, or adding a > very > > simple config-file-based mapping for what data to make accessible to > > dedicated "sharing" Principals or PrincipalRoles. > > > > We could reconcile this by also moving the ExternalConsumer/Listings APIs > > to the same "optional" yaml file I moved the other > > ConsumerCredential-related endpoints into and keeping the "core Shares > API" > > limited to what you wanted. Broadly, the two pillars of the proposal are: > > > > 1. *What* to share > > 2. *With whom* to share > > > > The core Shares API would only cover the first, in line with your > proposal: > > CRUD on Shares and the securables contained in them (listShares, > > createShare, getShare, updateShare, deleteShare, listShareMembers, > > updateShareMembers). > > > > The optional file would cover the second: ExternalConsumers, Listings, > the > > endpoint configuration, and credentials for the polaris-standalone case. > > > > Polaris service-runners could then decide if they just want sharing > > consumers to live in the exact same identity domain as their other native > > Polaris readers, or segregate them by enabling the "With whom" half of > the > > API. > > > > One thing to discuss for the "core-only" case - today the management API > > doesn't syntactically support granting a share to a PrincipalRole, so > we'd > > need to extend grant types or expose some of the internals of a share's > > backing catalog role. I guess we can iteratively refine any necessary API > > changes as we discover these in the core-only implementation. > > > > Note that the "with whom" half is necessary for the full "segregation" > > reqiurements stated in the original proposal doc, such as preventing > > assignment to other roles or accessing the "normal" catalog endpoints, > > which is why it can't be *fully* subsumed by only the native Polaris > > management API. > > > > It also doesn't conflict with external IdPs: Compare to Databricks' > > RECIPIENT for example, which can wrap a Databricks identity, a > > provider-issued token, or the recipients own IdP through an OIDC > federatoin > > policy, and shares are granted uniformly to it in all three cases: > > > https://docs.databricks.com/aws/en/delta-sharing/create-recipient-oidc-fed > > > > Even in the full API model my PR proposes, we could let the Listing be > > something the authorizer looks at alongside other authz conext, so the > > pluggable authorizers/OPA/Ranger/etc can integrate as the authoritative > > layer. > > > > > > On Thu, Oct 22, 2026 at 2:00 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Dennis, > > > > > > > - Shares API defines only shares and the objects placed in shares, > with > > > no > > > > Consumer or Listing > > > > > > POST /shares/{shareName}/listings taking a CreateListingRequest with a > > > `consumerName` is still part of `spec/polaris-shares-api.yaml`. > > > > > > I'm kind of confused about the scope of "shares" API vs. other APIs :) > > > Where are the boundaries? > > > > > > Would could be the minimal isoalated API sufficient for managing > Shares? > > > > > > > - Controlling who can read shares is only ordinary Polaris authz, > e.g. > > > > assignPrincipalRoletoPrincipal or > > > addGrantOnSecurableToPrincipalRole(share, > > > > principalRole) > > > > > > Sounds good. > > > > > > > - The partner organization consuming the share is just a principal, > > which > > > > could either be Polaris-managed or from an external IdP > > > > > > This also sounds good to me. > > > > > > I guess the only concern / confusion area is around Listing / Consumer > > > (first point above). > > > > > > Cheers, > > > Dmitri. > > > > > > > > > On Thu, Oct 22, 2026 at 2:07 PM Dennis Huo <[email protected]> wrote: > > > > > > > Thanks Dmitri, that does help highlight the the concern. Just to make > > > sure > > > > I understand what you have in mind: > > > > > > > > - Shares API defines only shares and the objects placed in shares, > with > > > no > > > > Consumer or Listing > > > > - Controlling who can read shares is only ordinary Polaris authz, > e.g. > > > > assignPrincipalRoletoPrincipal or > > > addGrantOnSecurableToPrincipalRole(share, > > > > principalRole) > > > > - The partner organization consuming the share is just a principal, > > which > > > > could either be Polaris-managed or from an external IdP > > > > > > > > Is that correct? > > > > > > > > On Wed, Oct 21, 2026 at 4:34 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Dennis, > > > > > > > > > > Let me try to restate my concern from another angle. Apologies > that I > > > was > > > > > not very clear in my previous messages. > > > > > > > > > > Currently the create listing doc says: "Create a listing, which > > grants > > > an > > > > > external consumer access to this share". > > > > > > > > > > So, this binds authorization decisions about a Share (Listing) to > > > > > ExternalConsumer. > > > > > > > > > > In IRC use cases a PolarisPrincipal is established by > authenticating > > > the > > > > > client request (either via internal JWT or via external IdP), > > > > Subsequently, > > > > > the PolarisPrincipal is passed to PolarisAuthorizer for making > > > > > authorization decisions. The latter part can be external too (e.g. > > OPA > > > / > > > > > Ranger). > > > > > > > > > > (Side note: the "external" work in ExternalConsumer is actually > quite > > > > > confusing with respect to prior usage of that word in external IdP > > use > > > > > cases.) > > > > > > > > > > In my view, the ExternalConsumer and related concepts in current > > Shares > > > > API > > > > > do not align with the external IdP and Authorizer workflows. > > > > > > > > > > IIRC, actual access to shared tables is going to use the IRC API, > > > right? > > > > > That access path in existing Polaris use cases maps well to > external > > > IdP > > > > > and external authorizers. > > > > > > > > > > Why does the Shares API need to define a "Consumer" as a separate > > > entity > > > > > and narrow authorization decisions to its relationship with > Listings? > > > > > > > > > > I believe authentication and authorization options should remain as > > > wide > > > > as > > > > > possible because these matters are usually addressed very > differently > > > in > > > > > each deployment. > > > > > > > > > > To be clear: I do not mind defining Share "Consumers" in the > internal > > > > > Polaris RBAC domain. It can fit well there, I'm only suggesting > > keeping > > > > the > > > > > Shares API itself neutral to Auth N/Z concerns. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <[email protected]> > wrote: > > > > > > > > > > > Ah thanks for clarifying - to be clear, the ConsumerCredential is > > > *not* > > > > > > strictly necessary for the Listing in terms of spec > expressibility, > > > > only > > > > > > the ExternalConsumer is. > > > > > > > > > > > > An ExternalConsumer doesn't need to be backed by a > > > ConsumerCredential; > > > > > the > > > > > > ExternalConsumer is an abstract Sharing-specific pointer to > > whatever > > > > does > > > > > > the indirection in the sharing implementation. > > > > > > > > > > > > Responded to your github comment - I'll split anything that > touches > > > > > > "ConsumerCredential" into a separate yaml file so it's easier to > > see > > > > the > > > > > > core auth-agnostic sharing spec in one file, with the more > > > > > > implementation-specific decorations around ConsumerCredential in > an > > > > > > auxiliary file. > > > > > > > > > > > > I suppose another option is to even split it into separate PRs > (in > > > > > addition > > > > > > to separate yaml files?). Then we can easily see one commit > that's > > > just > > > > > the > > > > > > pure abstract auth-agnostic API, that in theory private Polaris > > > > > > implementations could already take and implement on external-IdP > > > > > > architectures to support the same agnostic spec. > > > > > > > > > > > > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Dennis, > > > > > > > > > > > > > > While I still support the idea of splitting OpenAPI yaml files > > and > > > > URI > > > > > > > bases between the Management and Sharing APIs, I do not think > it > > is > > > > > > tightly > > > > > > > coupled with my proposal for the authentication-neural Sharing > > API > > > > > spec. > > > > > > > > > > > > > > To give an example, currently [5446] CreateListingRequest > > includes > > > a > > > > > > > "consumerName". This immediately binds the Listing definition > to > > > > > > > an ExternalConsumer and by extension to ConsumerCredential. > > > > > > > > > > > > > > ConsumerCredential is naturally part of the authentication > > process > > > > > during > > > > > > > Share access. I believe ConsumerCredential (and related > entities) > > > is > > > > > > worth > > > > > > > isolating from the Shares API to increase interoperability. > > > > > > > > > > > > > > Implementations that choose to rely on ConsumerCredential for > > Share > > > > > > access > > > > > > > can still do so. However, ConsumerCredential will not be a > > > > requirement > > > > > > for > > > > > > > all possible Share API implementations. > > > > > > > > > > > > > > This can work with any OpenAPI yaml and URI paths layout. > > > > > > > > > > > > > > I'd like to resolve this aspect before we go back to URI paths > > and > > > > YAML > > > > > > > file locations. Actually, I do not feel too strongly about path > > > > > > prefixes. I > > > > > > > do believe separate prefixes make implementations easier in > many > > > > > aspects > > > > > > > (including network setup), but interleaving those paths with > the > > > > > > Management > > > > > > > API is not a blocker, from my POV, only an inconvenience. > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > [5446] https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <[email protected]> > > > wrote: > > > > > > > > > > > > > > > Yeah, that's a good topic to discuss; I guess it touches on > our > > > > core > > > > > > > > (unresolved?) conventions on how we organize API url-prefixes > > and > > > > > yaml > > > > > > > > files. Reposting my response to the github comment as well: > > > > > > > > > > > > > > > > Right, I think we could certainly split the API routes and > > > > > > > yaml-definitions > > > > > > > > even further if we want to ensure a well-defined > > auth-detail-free > > > > > core, > > > > > > > but > > > > > > > > for the standalone-Polaris use case we'd still need to define > > it > > > > > > > > *somewhere* > > > > > > > > under a Sharing-specific surface, right? > > > > > > > > > > > > > > > > @dimas-b <https://github.com/dimas-b> Were you thinking of > > just > > > > > > > splitting > > > > > > > > the yaml file but keeping the share-management root, or > adding > > a > > > > > third > > > > > > > > sharing prefix like > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials... > > > > > > > > ? > > > > > > > > > > > > > > > > Or just yaml-file split would be keeping > > > > > > > > > > > /api/shares-management/v1/external-consumers/{name}/credentials... > > > > > but > > > > > > > > putting it in a separate yaml file to make it easy for > > > > > service-runners > > > > > > to > > > > > > > > choose not to support that whole yaml file's concrete APIs. > > > > > > > > > > > > > > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > Re: the shating API PR [5446], I'm repeating my GH comment > > here > > > > for > > > > > > > > > visibility. > > > > > > > > > > > > > > > > > > I'd like the API to be neutral from the authentication > > > > perspective. > > > > > > > > > > > > > > > > > > This is how we handle the IRC spec in Polaris. The API > > declares > > > > > that > > > > > > > > > clients need to authenticate, but specific details about > > > > > principals, > > > > > > > > > clients and secrets are delegated to the server. > > > > > > > > > > > > > > > > > > I think this is a sound approach for the sharing API too. > > > > Specific > > > > > > > > details > > > > > > > > > related to native Polaris principals (ConsumerCredential, > > etc.) > > > > can > > > > > > > > > probably remain in the Management API. > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > [5446] https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > > > > > As I said during the community meeting, let's focus on > the > > > API > > > > PR > > > > > > > > > > first. We will build from there. > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > +1, thanks for the discussion everyone! > > > > > > > > > > > > > > > > > > > > > > Yes, I meant to flip it to "ready to review" earlier; > > > flipped > > > > > it > > > > > > > now. > > > > > > > > > I'm > > > > > > > > > > > still working through the existing set of comments and > > may > > > > need > > > > > > to > > > > > > > > come > > > > > > > > > > > back to it in the next few days before getting proper > > > > replies, > > > > > > but > > > > > > > > feel > > > > > > > > > > > free to add additional comments. > > > > > > > > > > > > > > > > > > > > > > Recapping the community sync discussion re: Dmitri's > last > > > > > > message, > > > > > > > > I'm > > > > > > > > > > > fully aligned on building the sharing on top of > existing > > > > authz > > > > > > > > > > mechanisms, > > > > > > > > > > > instead of being a new mechanism, and indeed external > IdP > > > is > > > > > > > intended > > > > > > > > > to > > > > > > > > > > be > > > > > > > > > > > a first-class supported use case. > > > > > > > > > > > > > > > > > > > > > > The latest distinction is that the ExternalConsumer > isn't > > > > > > > > fundamentally > > > > > > > > > > > identified by any internal PolarisPrincipal, but rather > > > that > > > > it > > > > > > > > serves > > > > > > > > > > as a > > > > > > > > > > > wrapper that references some identity, whether that > > > identity > > > > is > > > > > > > > defined > > > > > > > > > > in > > > > > > > > > > > an external IdP without any latent in-Polaris > > > representation > > > > or > > > > > > if > > > > > > > > > it's a > > > > > > > > > > > hidden PolarisPrincipal (for the Polaris-standalone > RBAC > > > use > > > > > > case). > > > > > > > > > > > > > > > > > > > > > > The sharing endpoint via ExternalConsumer is the only > > layer > > > > of > > > > > > > > > > indirection > > > > > > > > > > > at the API entry-point, but then once the set of > > > > > > > activePrincipalRoles > > > > > > > > > is > > > > > > > > > > > resolved, the underlying authz engine doesn't behave > any > > > > > > > differently > > > > > > > > > from > > > > > > > > > > > normal Polaris authz. > > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the discussion! > > > > > > > > > > > > > > > > > > > > > > > > As a next step discussed in today's community sync, > we > > > will > > > > > > start > > > > > > > > > > reviewing > > > > > > > > > > > > the spec change. > > > > > > > > > > > > > > > > > > > > > > > > Hi Dennis, is this PR ready for review? > > > > > > > > > > > > https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen < > > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for this discussion Dennis! > > > > > > > > > > > > > > > > > > > > > > > > > > Wanted to weigh in with my thoughts and questions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you prefer /api/shares-management/v1/ > > > > > > > > > > > > > > instead of adding to /api/management/v1? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > While I can see the argument for a share specific > > > > > management > > > > > > > > > endpoint > > > > > > > > > > > > there > > > > > > > > > > > > > doesn't seem to be a clear obvious choice from > what I > > > can > > > > > see > > > > > > > - I > > > > > > > > > > lean > > > > > > > > > > > > > towards /api/management as being the central place > > for > > > > all > > > > > > the > > > > > > > > > > > > management - > > > > > > > > > > > > > share or not. As long as we're careful not to leak > > > share > > > > > > types > > > > > > > or > > > > > > > > > non > > > > > > > > > > > > share > > > > > > > > > > > > > types into the wrong api (which could be helped by > > > having > > > > > > > > separate > > > > > > > > > > yamls) > > > > > > > > > > > > > then it feels reasonable to have all under the > > > management > > > > > > api. > > > > > > > > > > > > > > > > > > > > > > > > > > Also went through the doc and PR and made several > > > > comments > > > > > as > > > > > > > > well. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri > Bourlatchkov < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Prithvi mentioned ExternalConsumer and > principals. > > > I'd > > > > > like > > > > > > > to > > > > > > > > > > > > > > reinforce the need to further discuss "consumers" > > and > > > > > > > > principals > > > > > > > > > > and > > > > > > > > > > > > how > > > > > > > > > > > > > > they relate to IdP and Polaris Authentication. > > > > > > > > > > > > > > > > > > > > > > > > > > > > From my POV, share access control should not be a > > new > > > > > > > mechanism > > > > > > > > > but > > > > > > > > > > > > > should > > > > > > > > > > > > > > build on top of the existing extensible > > > authentication > > > > / > > > > > > > > > > authorization > > > > > > > > > > > > > > framework. Supporting external IdP for shares > would > > > be > > > > a > > > > > > > > valuable > > > > > > > > > > > > feature > > > > > > > > > > > > > > IMHO. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Side note: I'm posting comments on PR 5446 too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for reviving this, and for the > concrete > > > > spec > > > > > in > > > > > > > PR > > > > > > > > > > 5446.. > > > > > > > > > > > > the > > > > > > > > > > > > > > > journeys doc was especially helpful for seeing > > the > > > > > > > end-to-end > > > > > > > > > > flow. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree this is worth doing, and that v1 is > more > > > > than a > > > > > > > > > tutorial > > > > > > > > > > on > > > > > > > > > > > > > > today's > > > > > > > > > > > > > > > RBAC. A partner principal on /api/catalog is > > > already > > > > > > > > possible; > > > > > > > > > > what > > > > > > > > > > > > > seems > > > > > > > > > > > > > > > valuable is a first-class share (enumerated > > > members, > > > > a > > > > > > > > > restricted > > > > > > > > > > > > > > consumer, > > > > > > > > > > > > > > > a listing as the binding and audit unit) plus a > > > > > > segregated, > > > > > > > > > > read-only > > > > > > > > > > > > > > > Iceberg REST surface with credential vending. > > Stock > > > > > > Iceberg > > > > > > > > > REST > > > > > > > > > > as > > > > > > > > > > > > the > > > > > > > > > > > > > > > consumer protocol, and keeping Flight and > > > > > > > Polaris-to-Polaris > > > > > > > > > > > > federation > > > > > > > > > > > > > > out > > > > > > > > > > > > > > > of v1, all sound right to me. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't have a strong view yet on the URL > layout > > > > > > > (management > > > > > > > > vs > > > > > > > > > > > > shares > > > > > > > > > > > > > > > prefix, nested vs top-level, prefix keyed by > > share > > > vs > > > > > > > > listing) > > > > > > > > > > and am > > > > > > > > > > > > > > happy > > > > > > > > > > > > > > > to follow whatever the community prefers there. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > A few things I wanted to check on the spec > > itself, > > > > > > several > > > > > > > of > > > > > > > > > > these > > > > > > > > > > > > > echo > > > > > > > > > > > > > > > points already raised on the list: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Membership as source of truth. If a hidden > > > catalog > > > > > > role > > > > > > > is > > > > > > > > > > > > > > independently > > > > > > > > > > > > > > > grantable, a share could be widened without > going > > > > > through > > > > > > > the > > > > > > > > > > share > > > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > Would it make sense to say the data plane > > > authorizes > > > > > > > against > > > > > > > > > > > > > membership + > > > > > > > > > > > > > > > listing, and that any backing roles are not > > > > > user-visible > > > > > > / > > > > > > > > not > > > > > > > > > > > > > > > independently grantable? > > > > > > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 > > recap > > > > > > already > > > > > > > > > > noted we > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > not assume a PrincipalEntity behind every > > > > > > ExternalConsumer. > > > > > > > > The > > > > > > > > > > > > > journeys > > > > > > > > > > > > > > > still describe granting the consumer a share > > > > principal > > > > > > > role, > > > > > > > > I > > > > > > > > > > assume > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > is an implementation sketch for v1 > > > client_id/secret, > > > > > not > > > > > > > the > > > > > > > > > > public > > > > > > > > > > > > > > model? > > > > > > > > > > > > > > > 3. Listing expiration. JB's original note was > > > "share > > > > > > these > > > > > > > > > tables > > > > > > > > > > > > with > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > consumer until this date." Credential expiry > is a > > > bit > > > > > > > > > different. > > > > > > > > > > > > Would > > > > > > > > > > > > > a > > > > > > > > > > > > > > > listing-level expiration (and maybe > > > ACTIVE/SUSPENDED) > > > > > be > > > > > > in > > > > > > > > > > scope for > > > > > > > > > > > > > v1? > > > > > > > > > > > > > > > 4. Time-travel. As raised in the community > sync, > > > > > sharing > > > > > > > via > > > > > > > > > > Iceberg > > > > > > > > > > > > > REST > > > > > > > > > > > > > > > means historical snapshots remain readable. > > Totally > > > > > fine > > > > > > > for > > > > > > > > v1 > > > > > > > > > > to > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > only full history, would an explicit field on > the > > > > share > > > > > > > help > > > > > > > > > so a > > > > > > > > > > > > later > > > > > > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise? > > > > > > > > > > > > > > > 5. Consumer data plane. The current file is > > > > > control-plane > > > > > > > > only, > > > > > > > > > > which > > > > > > > > > > > > > is > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > good first step. It might help to list the > > Iceberg > > > > > routes > > > > > > > > that > > > > > > > > > > are > > > > > > > > > > > > > > mounted, > > > > > > > > > > > > > > > that writes 404, that non-members 404, and that > > > > /config > > > > > > > does > > > > > > > > > not > > > > > > > > > > > > return > > > > > > > > > > > > > > > provider catalog properties, so implementations > > > don't > > > > > > > > diverge. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey > > the > > > > > > > consumer > > > > > > > > > > still > > > > > > > > > > > > > posts > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are > not > > > > > > supposed > > > > > > > to > > > > > > > > > > touch > > > > > > > > > > > > the > > > > > > > > > > > > > > > internal catalog API, should the default token > > > > endpoint > > > > > > > live > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > > > > > shares > > > > > > > > > > > > > > > surface (or an external IdP)? And on Dmitri's > > point > > > > > about > > > > > > > > share > > > > > > > > > > > > > operators > > > > > > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege > on > > > the > > > > > > bound > > > > > > > > > > catalog > > > > > > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would > make > > > > that > > > > > > work > > > > > > > > on > > > > > > > > > > any > > > > > > > > > > > > URL > > > > > > > > > > > > > > > prefix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Happy to review the next revision, and thanks > > again > > > > to > > > > > > both > > > > > > > > of > > > > > > > > > > you > > > > > > > > > > > > for > > > > > > > > > > > > > > > driving this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As discussed live, looks like we both might > > have > > > > some > > > > > > > > drafts > > > > > > > > > > we can > > > > > > > > > > > > > > > compare > > > > > > > > > > > > > > > > and choose/merge from :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here's my current draft spec PR; it opts for > my > > > > > > preferred > > > > > > > > > > syntax > > > > > > > > > > > > > > choices > > > > > > > > > > > > > > > > for now (/api/management/ for > share-management, > > > > > > > > /api/shares/ > > > > > > > > > > for > > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > > data-plane): > > > > > > https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Since we were talking about preferring > mdfiles > > > for > > > > > > design > > > > > > > > > > iteration > > > > > > > > > > > > > > > here's > > > > > > > > > > > > > > > > a companion design-addendum which just > > enumerates > > > > > what > > > > > > > the > > > > > > > > > > > > end-to-end > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > journeys look like syntactically for each > > > > combination > > > > > > of > > > > > > > > > design > > > > > > > > > > > > > > choices: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please let me know if anyone prefers a Google > > Doc > > > > > > version > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > > > > > > design-addendum for easier commenting, or if > > the > > > > > > > > > > design-addendum > > > > > > > > > > > > > itself > > > > > > > > > > > > > > > > should also be modeled as a PR on some > > > > free-floating > > > > > > > branch > > > > > > > > > > just to > > > > > > > > > > > > > > > capture > > > > > > > > > > > > > > > > comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste > > > > Onofré < > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am resuming work on this proposal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding our recent discussion, I will > move > > > > > forward > > > > > > > with > > > > > > > > > > > > opening a > > > > > > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to > > > help > > > > > > > > illustrate > > > > > > > > > > the > > > > > > > > > > > > > > > > > design. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I will update this thread once the PR is > > ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri > > > > Bourlatchkov > > > > > < > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Dennis, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the recap! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I was thinking that share management is > > > > probably > > > > > > > > distinct > > > > > > > > > > from > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > management. A user who already has a > > > > > realm+catalog > > > > > > > may > > > > > > > > > > have an > > > > > > > > > > > > > > option > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > manage shares, but not the catalog > itself. > > > > Would > > > > > it > > > > > > > > make > > > > > > > > > > sense > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > separate > > > > > > > > > > > > > > > > > > the share management API from the general > > > > Polaris > > > > > > > > > > management > > > > > > > > > > > > > API? I > > > > > > > > > > > > > > > > mean > > > > > > > > > > > > > > > > > > using a different prefix for URIs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example: /api/shares/v1/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Obviously clients should not make any > > > > assumptions > > > > > > > about > > > > > > > > > the > > > > > > > > > > > > base > > > > > > > > > > > > > > path > > > > > > > > > > > > > > > > > > (/api/shares), which could be different > in > > > > > > different > > > > > > > > > > > > deployments > > > > > > > > > > > > > > > (e.g. > > > > > > > > > > > > > > > > > > going through a gateway). Clients must be > > > able > > > > to > > > > > > > take > > > > > > > > > the > > > > > > > > > > base > > > > > > > > > > > > > URI > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > config option. Path segments after /v1/ > > will > > > be > > > > > > > > specified > > > > > > > > > > by > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > (Open > > > > > > > > > > > > > > > > > > API yaml). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis > Huo < > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just to recap some of the discussion > from > > > > > today's > > > > > > > > > > community > > > > > > > > > > > > > sync, > > > > > > > > > > > > > > > > > Dmitri, > > > > > > > > > > > > > > > > > > > Robert, and Alex helped raise a few > > > > > > considerations > > > > > > > to > > > > > > > > > > roll > > > > > > > > > > > > into > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > next > > > > > > > > > > > > > > > > > > > design/prototyping phase: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Given the way sharing is packaged > as a > > > > > > separate > > > > > > > > > > thing, it > > > > > > > > > > > > > may > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > obvious to users that historical > > snapshots > > > > are > > > > > > > > equally > > > > > > > > > > > > > available > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > consumers as the latest snapshot (by > > virtue > > > > of > > > > > > > > sharing > > > > > > > > > > via > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > standard IRC > > > > > > > > > > > > > > > > > > > protocol) - we should make sure to > > document > > > > > that > > > > > > > fact > > > > > > > > > > well, > > > > > > > > > > > > > > > including > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > pitfall that deleting rows by updating > a > > > > table > > > > > > > > doesn't > > > > > > > > > > > > > > necessarily > > > > > > > > > > > > > > > > mean > > > > > > > > > > > > > > > > > > > those rows become inaccessible to > > consumers > > > > > > > > > > > > > > > > > > > 2. Make sure not to hard-code > assumptions > > > > about > > > > > > the > > > > > > > > > > existence > > > > > > > > > > > > > of > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > internal PrincipalEntity backing an > > > > > > > ExternalConsumer > > > > > > > > - > > > > > > > > > > even > > > > > > > > > > > > if > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > > > MVP implementation uses a nested/hidden > > > > > Principal > > > > > > > of > > > > > > > > > some > > > > > > > > > > > > type > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > holds a > > > > > > > > > > > > > > > > > > > Polaris-managed > client_id/client_secret, > > > the > > > > > > > > 3rd-party > > > > > > > > > > IdP > > > > > > > > > > > > use > > > > > > > > > > > > > > > cases > > > > > > > > > > > > > > > > > > > require "implicit" Principals resolved > > via > > > > the > > > > > > > oauth > > > > > > > > > > token. > > > > > > > > > > > > In > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > cases > > > > > > > > > > > > > > > > > > > the ExternalConsumer contains the > > metadata > > > > > > > > coordinating > > > > > > > > > > the > > > > > > > > > > > > > > > three-way > > > > > > > > > > > > > > > > > > > handshake between the data-provider, > > > > > > data-consumer, > > > > > > > > and > > > > > > > > > > IdP, > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > no PrincipalEntity underlying it. > > > > > > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle > > > > > > > "on-behalf-of" > > > > > > > > > > > > semantics > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > > initially, > > > > > > > > > > > > > > > > > > > the way an ExternalConsumer works is > that > > > the > > > > > > > > > > finer-grained > > > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > > > share a single ExternalConsumer > identity, > > > > just > > > > > > like > > > > > > > > > how a > > > > > > > > > > > > > > FEDERATED > > > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > > shares a single service principal in > its > > > > > > > > > > ConnectionConfig. > > > > > > > > > > > > But > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > cases the consumer may want some > > end-to-end > > > > > > > > attribution > > > > > > > > > > of > > > > > > > > > > > > > > > > fine-grained > > > > > > > > > > > > > > > > > > > users, like "on-behalf-of: bob @ > > > > > > > consumercompany.com > > > > > > > > " > > > > > > > > > > when > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > ExternalConsumer implicit principal is > > > > > something > > > > > > > like > > > > > > > > > > > > > > > > > > > "externalconsumer$consumercompany" > > > > > > > > > > > > > > > > > > > 4. We can iterate more rapidly on a > draft > > > > spec > > > > > PR > > > > > > > and > > > > > > > > > > also > > > > > > > > > > > > > expand > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > detailed technical design > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis > > Huo > > > < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks JB for reviving this thread, > and > > > > > belated > > > > > > > > > thanks > > > > > > > > > > to > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > reviewers > > > > > > > > > > > > > > > > > > > > who have left feedback in the doc or > > this > > > > > > thread > > > > > > > so > > > > > > > > > > far! > > > > > > > > > > > > And > > > > > > > > > > > > > > > > > apologies > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > the silence on this, it's still been > on > > > my > > > > > > radar > > > > > > > > but > > > > > > > > > I > > > > > > > > > > got > > > > > > > > > > > > > > > > > distracted by > > > > > > > > > > > > > > > > > > > > some other work for awhile. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is > > > > indeed > > > > > > that > > > > > > > > > we're > > > > > > > > > > > > > mostly > > > > > > > > > > > > > > > just > > > > > > > > > > > > > > > > > > > > reusing existing Authz/RBAC > machinery, > > > and > > > > > the > > > > > > > > > > translation > > > > > > > > > > > > > > layer > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > sharing concepts sits *on top of* the > > > > > existing > > > > > > > > authz > > > > > > > > > > engine > > > > > > > > > > > > > as > > > > > > > > > > > > > > > much > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So even if we're adding layers of > > > > indirection > > > > > > on > > > > > > > > > > > > > > > > > ExternalConsumer/Listing > > > > > > > > > > > > > > > > > > > > -> > > > > > Principal/PrincipalRole/CatalogRole+grants, > > > > > > > the > > > > > > > > > > parts > > > > > > > > > > > > that > > > > > > > > > > > > > > end > > > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > > > > > > the underlying authz just reuse > > existing > > > > > > > Authorizer > > > > > > > > > > logic. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Based on some other offline > discussion > > > with > > > > > > > > Dmitri, I > > > > > > > > > > > > > realized > > > > > > > > > > > > > > > why > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > discussion around "requirements" may > > have > > > > > been > > > > > > > > > > confusing: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc > > > > > inadvertently > > > > > > > cut > > > > > > > > > out > > > > > > > > > > > > some > > > > > > > > > > > > > of > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > "abstract requirements" formulations > > > > > > > > > > > > > > > > > > > > 2. The project in itself is somewhat > > more > > > > of > > > > > a > > > > > > > > "form > > > > > > > > > > *as* > > > > > > > > > > > > > > > function" > > > > > > > > > > > > > > > > > > > > feature than *strictly* functional in > > > > nature > > > > > -- > > > > > > > for > > > > > > > > > > > > example, > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > correctly > > > > > > > > > > > > > > > > > > > > pointed out that "sharing" via the > > > existing > > > > > > > > > > RBAC/Principals > > > > > > > > > > > > > > > syntax > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > already *possible* just by letting a > > > > > cross-org > > > > > > > > > > "consumer" > > > > > > > > > > > > be > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > > > > > > principal and access your own > company's > > > > main > > > > > > > > Polaris > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > endpoints. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For (1) I added this to the > > Requirements > > > > > > section > > > > > > > in > > > > > > > > > > the doc > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > better > > > > > > > > > > > > > > > > > > > > clarify the abstract form of the data > > > model > > > > > > > > element: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A. Sharing Container*: Container > > concept > > > > for > > > > > > > > holding > > > > > > > > > > > > things > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > > want to > > > > > > > > > > > > > > > > > > > > share - “ShareEntity” > > > > > > > > > > > > > > > > > > > > *B. Consumer Identity*: Special > flavor > > of > > > > > > > > > > principal-like > > > > > > > > > > > > > entity > > > > > > > > > > > > > > > > > > > > representing a consumer relationship > - > > > > > > > > > > “ExternalConsumer” > > > > > > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A > way > > > to > > > > > > > specify > > > > > > > > > > letting > > > > > > > > > > > > a > > > > > > > > > > > > > > > given > > > > > > > > > > > > > > > > > > > > consumer access a given sharing > > > container - > > > > > > > > “Listing” > > > > > > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated > > API > > > > > > > > access-point > > > > > > > > > > for > > > > > > > > > > > > > > > > consumers - > > > > > > > > > > > > > > > > > > > > “EndpointConfig” > > > > > > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to > > > > specify > > > > > > > which > > > > > > > > > > > > in-catalog > > > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > > > belong in the big sharing container - > > > > > > > > > “ShareMembership” > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This still bakes in a bit of > > > implementation > > > > > > > detail > > > > > > > > > > since it > > > > > > > > > > > > > > begs > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > question why not just let core > > > RBAC/grants > > > > > > handle > > > > > > > > (C) > > > > > > > > > > and > > > > > > > > > > > > > (E). > > > > > > > > > > > > > > To > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > end, > > > > > > > > > > > > > > > > > > > > and addressing problem (2) of where > the > > > > "form > > > > > > > *is* > > > > > > > > > > > > > function", I > > > > > > > > > > > > > > > > > added a > > > > > > > > > > > > > > > > > > > > "Concept Mapping" tab: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think the details of more advanced > > > > features > > > > > > can > > > > > > > > > > still be > > > > > > > > > > > > > > hashed > > > > > > > > > > > > > > > > out > > > > > > > > > > > > > > > > > > > > incrementally without blocking work > on > > an > > > > MVP > > > > > > > (and > > > > > > > > > > > > certainly > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > MVP > > > > > > > > > > > > > > > > > > > scope > > > > > > > > > > > > > > > > > > > > could be trimmed so we can > > > > > launch-and-iterate), > > > > > > > so > > > > > > > > as > > > > > > > > > > long > > > > > > > > > > > > as > > > > > > > > > > > > > > > > there's > > > > > > > > > > > > > > > > > > > > community alignment on the general > > > > direction > > > > > I > > > > > > > > > propose > > > > > > > > > > we > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > move > > > > > > > > > > > > > > > > > > > > forward on basic building blocks. Or > if > > > > > > building > > > > > > > > > blocks > > > > > > > > > > > > > > > themselves > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > controversial we could still proceed > > with > > > > > some > > > > > > > > > > prototyping > > > > > > > > > > > > > > across > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > > > options. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM > > > > > Jean-Baptiste > > > > > > > > > Onofré < > > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi everyone, > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> I would like to resume the > discussion > > > > about > > > > > > Data > > > > > > > > > > Sharing > > > > > > > > > > > > in > > > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > > > >> and propose focusing on three major > > > items > > > > to > > > > > > > help > > > > > > > > us > > > > > > > > > > move > > > > > > > > > > > > > > > forward. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> 1. What is data sharing? > > > > > > > > > > > > > > > > > > > >> In a nutshell, it is an open > > > protocol > > > > > > > proposal > > > > > > > > > > > > allowing > > > > > > > > > > > > > > > > > > > >> organizations to share live data > > without > > > > > > copying > > > > > > > > or > > > > > > > > > > > > > > replication. > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > >> main goal is zero-copy sharing > > achieved > > > > > > through > > > > > > > > > shared > > > > > > > > > > > > > > metadata > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > >> authorization. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> On Polaris, this would mean: > > > > > > > > > > > > > > > > > > > >> a. The consumer is authenticated as > > > usual. > > > > > > > > > > > > > > > > > > > >> b. The Polaris server provides the > > > > consumer > > > > > > with > > > > > > > > > > direct > > > > > > > > > > > > > access > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > >> data files (e.g., Parquet, as > > specified > > > in > > > > > the > > > > > > > > > > > > > metadata.json). > > > > > > > > > > > > > > > > > > > >> c. The consumer reads the data files > > > > > directly > > > > > > > > using > > > > > > > > > > their > > > > > > > > > > > > > > > > preferred > > > > > > > > > > > > > > > > > > > >> tool (e.g., a query engine). > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> While Polaris already has most of > the > > > > > > necessary > > > > > > > > > > underlying > > > > > > > > > > > > > > > > > > > >> capabilities, we need clear > > > > > branding/packaging > > > > > > > > > around > > > > > > > > > > this > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > case. > > > > > > > > > > > > > > > > > > > >> Specifically, this means: > > > > > > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in > > the > > > > > > Polaris > > > > > > > > > > server, > > > > > > > > > > > > or > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > > >> role scoped to specific namespaces. > > > > > > > > > > > > > > > > > > > >> b. A recipient is represented by a > > > > principal > > > > > > > and a > > > > > > > > > > > > principal > > > > > > > > > > > > > > > role. > > > > > > > > > > > > > > > > > > > >> c. A consumer profile consists of > the > > > > > catalog > > > > > > > URI > > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > > > > > OAuth2 > > > > > > > > > > > > > > > > > > > >> client_id/client_secret. > > > > > > > > > > > > > > > > > > > >> d. We use credential vending > > (temporary > > > > STS > > > > > > > > > > credentials) > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > > > > > >> access to the data files. > > > > > > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to > > > expose > > > > > the > > > > > > > > > > metadata and > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > file > > > > > > > > > > > > > > > > > > > >> locations/access. > > > > > > > > > > > > > > > > > > > >> f. We can also support > > > Polaris-to-Polaris > > > > > > > sharing > > > > > > > > > via > > > > > > > > > > > > > > > Federation. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> Concretely: > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> - On the provider side, we use the > > > > > standard > > > > > > > > > Polaris > > > > > > > > > > RBAC > > > > > > > > > > > > > > model > > > > > > > > > > > > > > > > > > > >> (assigning grants to catalog roles, > > and > > > > > > > assigning > > > > > > > > > > those > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > roles > > > > > > > > > > > > > > > > > > > >> to principal roles): > > > > > > > > > > > > > > > > > > > >> - Create a dedicated principal > > > with > > > > > its > > > > > > > own > > > > > > > > > > > > > > > > > > > client_id/client_secret. > > > > > > > > > > > > > > > > > > > >> - Create a catalog role with > > > > > > > > TABLE_READ_DATA, > > > > > > > > > > > > > > TABLE_LIST, > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the > > namespaces > > > to > > > > > be > > > > > > > > shared > > > > > > > > > > (no > > > > > > > > > > > > > write > > > > > > > > > > > > > > > > > > > >> grants). > > > > > > > > > > > > > > > > > > > >> - Create a principal role per > > > > > > destination, > > > > > > > > > > attach > > > > > > > > > > > > the > > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > > >> role, and assign the principal. > > > > > > > > > > > > > > > > > > > >> - The destination engine uses > > > > > loadTable > > > > > > > with > > > > > > > > > the > > > > > > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation: > > > > > > > vended-credentials" > > > > > > > > > > header. > > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > then > > > > > > > > > > > > > > > > > > > >> emits temporary, read-only storage > > > > > > credentials, > > > > > > > > > > allowing > > > > > > > > > > > > the > > > > > > > > > > > > > > > > engine > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > >> read the data files directly from > the > > > > > owner's > > > > > > > > bucket > > > > > > > > > > > > without > > > > > > > > > > > > > > > > > accessing > > > > > > > > > > > > > > > > > > > >> the rest of the bucket. This > achieves > > > > > > zero-copy > > > > > > > > > > sharing. > > > > > > > > > > > > > > > > > > > >> - On the recipient/consumer side, > no > > > > > special > > > > > > > > > > > > configuration > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > >> needed; any IRC client can use it. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> 2. What is data sharing not? > > > > > > > > > > > > > > > > > > > >> I previously considered adding > an > > > > Arrow > > > > > > > Flight > > > > > > > > > > > > endpoint > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > >> Polaris, but I no longer think it > is a > > > > good > > > > > > idea > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > > > > > initial > > > > > > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint > would > > > > serve > > > > > > the > > > > > > > > data > > > > > > > > > > > > > directly, > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > >> would require Polaris to read the > data > > > and > > > > > > > expose > > > > > > > > > it. > > > > > > > > > > I am > > > > > > > > > > > > > > > against > > > > > > > > > > > > > > > > > > > >> embedding a query engine in Polaris, > > as > > > > this > > > > > > > tight > > > > > > > > > > > > coupling > > > > > > > > > > > > > > > feels > > > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > > >> an anti-pattern. While we could use > a > > > > > > delegation > > > > > > > > > > service > > > > > > > > > > > > to > > > > > > > > > > > > > > > serve > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > >> Flight endpoint, I think it should > be > > > > > > considered > > > > > > > > > for a > > > > > > > > > > > > later > > > > > > > > > > > > > > > > phase, > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > >> at all, since a Flight endpoint > might > > be > > > > > > better > > > > > > > > > > suited as > > > > > > > > > > > > a > > > > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > > > > > >> responsibility and could encourage > > data > > > > > > copying. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> The first phase should focus on > > > packaging > > > > > and > > > > > > > > > > branding the > > > > > > > > > > > > > > > > > > > >> capabilities we already have. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> 3. What could the Data Sharing > > > "package" > > > > > look > > > > > > > > like > > > > > > > > > in > > > > > > > > > > > > > > Polaris? > > > > > > > > > > > > > > > > > > > >> To make Data Sharing obvious and > > > easy > > > > to > > > > > > > use, > > > > > > > > we > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > introduce > > > > > > > > > > > > > > > > > > > >> the missing concept of a "share"—a > > named > > > > > > > > definition > > > > > > > > > > that > > > > > > > > > > > > > users > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > >> list, discover, and define. The goal > > is > > > to > > > > > > > > > > materialize the > > > > > > > > > > > > > > > user's > > > > > > > > > > > > > > > > > > > >> intent ("I want to share these > tables > > > with > > > > > > this > > > > > > > > > > consumer > > > > > > > > > > > > > until > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > >> date") into standard Polaris > entities > > > > > > > (principal, > > > > > > > > > > > > principal > > > > > > > > > > > > > > > role, > > > > > > > > > > > > > > > > > > > >> catalog role, and grants). > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> Therefore, I propose adding a > /shares > > > > > endpoint > > > > > > > to > > > > > > > > > the > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > > > >> Management API to package and > > > materialize > > > > > > these > > > > > > > > > share > > > > > > > > > > > > > > > definitions > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > >> Polaris entities. This would > provide a > > > > > usable, > > > > > > > > > > > > first-version > > > > > > > > > > > > > > > > package > > > > > > > > > > > > > > > > > > > >> for Polaris Data Sharing. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> I hope this addresses Dmitri's > > questions > > > > and > > > > > > > > > provides > > > > > > > > > > a > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > concrete > > > > > > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> Thoughts? > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> Regards, > > > > > > > > > > > > > > > > > > > >> JB > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM > Dennis > > > > Huo < > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > Hi All, > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > One big emerging enterprise use > case > > > > > coming > > > > > > up > > > > > > > > as > > > > > > > > > > more > > > > > > > > > > > > > > people > > > > > > > > > > > > > > > > > > > >> consolidate > > > > > > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is > > > > something > > > > > > > > commonly > > > > > > > > > > known > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > "Data > > > > > > > > > > > > > > > > > > > >> Sharing", > > > > > > > > > > > > > > > > > > > >> > an more specifically over the > course > > > of > > > > > > > adoption > > > > > > > > > of > > > > > > > > > > open > > > > > > > > > > > > > > table > > > > > > > > > > > > > > > > > formats > > > > > > > > > > > > > > > > > > > >> > "Open Data Sharing". > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > Examples of existing managed > service > > > > > > > providers' > > > > > > > > > Data > > > > > > > > > > > > > Sharing > > > > > > > > > > > > > > > > > features: > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > https://www.databricks.com/product/delta-sharing > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > The basic idea is that when you > > share > > > > data > > > > > > > > between > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > > companies, > > > > > > > > > > > > > > > > > > > >> you > > > > > > > > > > > > > > > > > > > >> > need a first-class > > > governance/management > > > > > > layer > > > > > > > > and > > > > > > > > > > extra > > > > > > > > > > > > > > > > > > > >> bells-and-whistles > > > > > > > > > > > > > > > > > > > >> > that are distinct from just the > > basic > > > > > > > > capabilities > > > > > > > > > > of > > > > > > > > > > > > RBAC > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > >> generalized > > > > > > > > > > > > > > > > > > > >> > access-control (i.e. if you're > > sharing > > > > > > across > > > > > > > > > > > > > > > > partially-untrusted > > > > > > > > > > > > > > > > > org > > > > > > > > > > > > > > > > > > > >> > boundaries, you don't just let the > > > > > consumer > > > > > > > > > > organization > > > > > > > > > > > > > log > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > your > > > > > > > > > > > > > > > > > > > >> > datalake like one of your own > > > > employees). > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > JB and I put together this > > high-level > > > > > > proposal > > > > > > > > for > > > > > > > > > > > > > > supporting > > > > > > > > > > > > > > > > Open > > > > > > > > > > > > > > > > > > > >> Sharing > > > > > > > > > > > > > > > > > > > >> > in Polaris: > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5 > > > logical > > > > > data > > > > > > > > model > > > > > > > > > > > > > > constructs, > > > > > > > > > > > > > > > > > some of > > > > > > > > > > > > > > > > > > > >> > which may be a first-class > > > PolarisEntity > > > > > > type, > > > > > > > > > > others > > > > > > > > > > > > > > subtypes > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > >> existing > > > > > > > > > > > > > > > > > > > >> > entities, and others just a nested > > > > > > construct: > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > - ShareEntity (would behave > > > similarly > > > > > to > > > > > > a > > > > > > > > > > Catalog) > > > > > > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly > > inherits > > > > > from > > > > > > > > > > Principal) > > > > > > > > > > > > > > > > > > > >> > - Listing (Similar to a "role > > > grant" > > > > > but > > > > > > > has > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > metadata) > > > > > > > > > > > > > > > > > > > >> > - EndpointConfig (nested config > > > under > > > > > > > > Listing) > > > > > > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a > > > > > > "securable > > > > > > > > > > grant" but > > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > > >> metadata) > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll > also > > > > bring > > > > > > it > > > > > > > up > > > > > > > > > for > > > > > > > > > > > > live > > > > > > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > >> > there's time in the community > sync. > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > Cheers, > > > > > > > > > > > > > > > > > > > >> > Dennis > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
