Thanks for this discussion Dennis! Wanted to weigh in with my thoughts and questions.
> > Do you prefer /api/shares-management/v1/ > instead of adding to /api/management/v1? While I can see the argument for a share specific management endpoint there doesn't seem to be a clear obvious choice from what I can see - I lean towards /api/management as being the central place for all the management - share or not. As long as we're careful not to leak share types or non share types into the wrong api (which could be helped by having separate yamls) then it feels reasonable to have all under the management api. Also went through the doc and PR and made several comments as well. On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > Prithvi mentioned ExternalConsumer and principals. I'd like to > reinforce the need to further discuss "consumers" and principals and how > they relate to IdP and Polaris Authentication. > > From my POV, share access control should not be a new mechanism but should > build on top of the existing extensible authentication / authorization > framework. Supporting external IdP for shares would be a valuable feature > IMHO. > > Side note: I'm posting comments on PR 5446 too. > > Cheers, > Dmitri. > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S <[email protected]> > wrote: > > > Hi everyone, > > > > Thank you for reviving this, and for the concrete spec in PR 5446.. the > > journeys doc was especially helpful for seeing the end-to-end flow. > > > > I agree this is worth doing, and that v1 is more than a tutorial on > today's > > RBAC. A partner principal on /api/catalog is already possible; what seems > > valuable is a first-class share (enumerated members, a restricted > consumer, > > a listing as the binding and audit unit) plus a segregated, read-only > > Iceberg REST surface with credential vending. Stock Iceberg REST as the > > consumer protocol, and keeping Flight and Polaris-to-Polaris federation > out > > of v1, all sound right to me. > > > > I don't have a strong view yet on the URL layout (management vs shares > > prefix, nested vs top-level, prefix keyed by share vs listing) and am > happy > > to follow whatever the community prefers there. > > > > A few things I wanted to check on the spec itself, several of these echo > > points already raised on the list: > > > > 1. Membership as source of truth. If a hidden catalog role is > independently > > grantable, a share could be widened without going through the share APIs. > > Would it make sense to say the data plane authorizes against membership + > > listing, and that any backing roles are not user-visible / not > > independently grantable? > > 2. ExternalConsumer and principals. The Aug 7 recap already noted we > should > > not assume a PrincipalEntity behind every ExternalConsumer. The journeys > > still describe granting the consumer a share principal role, I assume > that > > is an implementation sketch for v1 client_id/secret, not the public > model? > > 3. Listing expiration. JB's original note was "share these tables with > this > > consumer until this date." Credential expiry is a bit different. Would a > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for v1? > > 4. Time-travel. As raised in the community sync, sharing via Iceberg REST > > means historical snapshots remain readable. Totally fine for v1 to > support > > only full history, would an explicit field on the share help so a later > > CURRENT_ONLY option is not a breaking surprise? > > 5. Consumer data plane. The current file is control-plane only, which is > a > > good first step. It might help to list the Iceberg routes that are > mounted, > > that writes 404, that non-members 404, and that /config does not return > > provider catalog properties, so implementations don't diverge. > > > > Two smaller notes, if useful: in the J1 journey the consumer still posts > to > > /api/catalog/v1/oauth/tokens, if partners are not supposed to touch the > > internal catalog API, should the default token endpoint live on the > shares > > surface (or an external IdP)? And on Dmitri's point about share operators > > vs catalog operators, a SHARE_MANAGE privilege on the bound catalog > > (independent of CATALOG_MANAGE_ACCESS) would make that work on any URL > > prefix. > > > > Happy to review the next revision, and thanks again to both of you for > > driving this. > > > > Regards, > > Prithvi S > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <[email protected]> wrote: > > > > > As discussed live, looks like we both might have some drafts we can > > compare > > > and choose/merge from :) > > > > > > Here's my current draft spec PR; it opts for my preferred syntax > choices > > > for now (/api/management/ for share-management, /api/shares/ for > consumer > > > data-plane): https://github.com/apache/polaris/pull/5446 > > > > > > Since we were talking about preferring mdfiles for design iteration > > here's > > > a companion design-addendum which just enumerates what the end-to-end > > user > > > journeys look like syntactically for each combination of design > choices: > > > > > > > > > > > > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md > > > > > > Please let me know if anyone prefers a Google Doc version of the > > > design-addendum for easier commenting, or if the design-addendum itself > > > should also be modeled as a PR on some free-floating branch just to > > capture > > > comments. > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > Hi everyone, > > > > > > > > I am resuming work on this proposal. > > > > > > > > Regarding our recent discussion, I will move forward with opening a > > > > draft PR for the /api/shares/v1 endpoint to help illustrate the > > > > design. > > > > > > > > I will update this thread once the PR is ready. > > > > > > > > Thanks, > > > > JB > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <[email protected] > > > > > > wrote: > > > > > > > > > > Hi Dennis, > > > > > > > > > > Thanks for the recap! > > > > > > > > > > I was thinking that share management is probably distinct from > > catalog > > > > > management. A user who already has a realm+catalog may have an > option > > > to > > > > > manage shares, but not the catalog itself. Would it make sense to > > > > separate > > > > > the share management API from the general Polaris management API? I > > > mean > > > > > using a different prefix for URIs. > > > > > > > > > > For example: /api/shares/v1/ > > > > > > > > > > Obviously clients should not make any assumptions about the base > path > > > > > (/api/shares), which could be different in different deployments > > (e.g. > > > > > going through a gateway). Clients must be able to take the base URI > > as > > > a > > > > > config option. Path segments after /v1/ will be specified by > Polaris > > > > (Open > > > > > API yaml). > > > > > > > > > > WDYT? > > > > > > > > > > Thanks, > > > > > Dmitri. > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <[email protected]> wrote: > > > > > > > > > > > Just to recap some of the discussion from today's community sync, > > > > Dmitri, > > > > > > Robert, and Alex helped raise a few considerations to roll into > the > > > > next > > > > > > design/prototyping phase: > > > > > > > > > > > > 1. Given the way sharing is packaged as a separate thing, it may > > not > > > be > > > > > > obvious to users that historical snapshots are equally available > to > > > > > > consumers as the latest snapshot (by virtue of sharing via the > > > > standard IRC > > > > > > protocol) - we should make sure to document that fact well, > > including > > > > the > > > > > > pitfall that deleting rows by updating a table doesn't > necessarily > > > mean > > > > > > those rows become inaccessible to consumers > > > > > > 2. Make sure not to hard-code assumptions about the existence of > an > > > > > > internal PrincipalEntity backing an ExternalConsumer - even if > the > > > > first > > > > > > MVP implementation uses a nested/hidden Principal of some type > that > > > > holds a > > > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use > > cases > > > > > > require "implicit" Principals resolved via the oauth token. In > such > > > > cases > > > > > > the ExternalConsumer contains the metadata coordinating the > > three-way > > > > > > handshake between the data-provider, data-consumer, and IdP, but > > > there > > > > is > > > > > > no PrincipalEntity underlying it. > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics - > > > > initially, > > > > > > the way an ExternalConsumer works is that the finer-grained > > consumer > > > > users > > > > > > share a single ExternalConsumer identity, just like how a > FEDERATED > > > > catalog > > > > > > shares a single service principal in its ConnectionConfig. But in > > > some > > > > > > cases the consumer may want some end-to-end attribution of > > > fine-grained > > > > > > users, like "on-behalf-of: bob @ consumercompany.com" when the > > > > > > ExternalConsumer implicit principal is something like > > > > > > "externalconsumer$consumercompany" > > > > > > 4. We can iterate more rapidly on a draft spec PR and also expand > > on > > > > the > > > > > > detailed technical design > > > > > > > > > > > > Cheers, > > > > > > Dennis > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <[email protected]> > > wrote: > > > > > > > > > > > > > Thanks JB for reviving this thread, and belated thanks to all > > > > reviewers > > > > > > > who have left feedback in the doc or this thread so far! And > > > > apologies > > > > > > for > > > > > > > the silence on this, it's still been on my radar but I got > > > > distracted by > > > > > > > some other work for awhile. > > > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed that we're mostly > > just > > > > > > > reusing existing Authz/RBAC machinery, and the translation > layer > > to > > > > the > > > > > > new > > > > > > > sharing concepts sits *on top of* the existing authz engine as > > much > > > > as > > > > > > > possible. > > > > > > > > > > > > > > So even if we're adding layers of indirection on > > > > ExternalConsumer/Listing > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that > end > > > up > > > > > > doing > > > > > > > the underlying authz just reuse existing Authorizer logic. > > > > > > > > > > > > > > Based on some other offline discussion with Dmitri, I realized > > why > > > > the > > > > > > > discussion around "requirements" may have been confusing: > > > > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut out some of > > the > > > > more > > > > > > > "abstract requirements" formulations > > > > > > > 2. The project in itself is somewhat more of a "form *as* > > function" > > > > > > > feature than *strictly* functional in nature -- for example, > many > > > > > > correctly > > > > > > > pointed out that "sharing" via the existing RBAC/Principals > > syntax > > > is > > > > > > > already *possible* just by letting a cross-org "consumer" be a > > > > > > first-class > > > > > > > principal and access your own company's main Polaris catalog > > > > endpoints. > > > > > > > > > > > > > > For (1) I added this to the Requirements section in the doc to > > > better > > > > > > > clarify the abstract form of the data model element: > > > > > > > > > > > > > > *A. Sharing Container*: Container concept for holding things > you > > > > want to > > > > > > > share - “ShareEntity” > > > > > > > *B. Consumer Identity*: Special flavor of principal-like entity > > > > > > > representing a consumer relationship - “ExternalConsumer” > > > > > > > *C. Map Containers to Consumers*: A way to specify letting a > > given > > > > > > > consumer access a given sharing container - “Listing” > > > > > > > *D. Consumer Endpoint*: A segregated API access-point for > > > consumers - > > > > > > > “EndpointConfig” > > > > > > > *E. Objects in Containers*: A way to specify which in-catalog > > > objects > > > > > > > belong in the big sharing container - “ShareMembership” > > > > > > > > > > > > > > This still bakes in a bit of implementation detail since it > begs > > > the > > > > > > > question why not just let core RBAC/grants handle (C) and (E). > To > > > > that > > > > > > end, > > > > > > > and addressing problem (2) of where the "form *is* function", I > > > > added a > > > > > > > "Concept Mapping" tab: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > > > > > > > > > I think the details of more advanced features can still be > hashed > > > out > > > > > > > incrementally without blocking work on an MVP (and certainly > the > > > MVP > > > > > > scope > > > > > > > could be trimmed so we can launch-and-iterate), so as long as > > > there's > > > > > > > community alignment on the general direction I propose we could > > > move > > > > > > > forward on basic building blocks. Or if building blocks > > themselves > > > > are > > > > > > > controversial we could still proceed with some prototyping > across > > > > > > different > > > > > > > options. > > > > > > > > > > > > > > Cheers, > > > > > > > Dennis > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > >> Hi everyone, > > > > > > >> > > > > > > >> I would like to resume the discussion about Data Sharing in > > > Polaris > > > > > > >> and propose focusing on three major items to help us move > > forward. > > > > > > >> > > > > > > >> 1. What is data sharing? > > > > > > >> In a nutshell, it is an open protocol proposal allowing > > > > > > >> organizations to share live data without copying or > replication. > > > The > > > > > > >> main goal is zero-copy sharing achieved through shared > metadata > > > and > > > > > > >> authorization. > > > > > > >> > > > > > > >> On Polaris, this would mean: > > > > > > >> a. The consumer is authenticated as usual. > > > > > > >> b. The Polaris server provides the consumer with direct access > > to > > > > the > > > > > > >> data files (e.g., Parquet, as specified in the metadata.json). > > > > > > >> c. The consumer reads the data files directly using their > > > preferred > > > > > > >> tool (e.g., a query engine). > > > > > > >> > > > > > > >> While Polaris already has most of the necessary underlying > > > > > > >> capabilities, we need clear branding/packaging around this use > > > case. > > > > > > >> Specifically, this means: > > > > > > >> a. A share is a dedicated catalog in the Polaris server, or a > > > > catalog > > > > > > >> role scoped to specific namespaces. > > > > > > >> b. A recipient is represented by a principal and a principal > > role. > > > > > > >> c. A consumer profile consists of the catalog URI and the > OAuth2 > > > > > > >> client_id/client_secret. > > > > > > >> d. We use credential vending (temporary STS credentials) for > > > > consumer > > > > > > >> access to the data files. > > > > > > >> e. We use the Polaris IRC layer to expose the metadata and the > > > file > > > > > > >> locations/access. > > > > > > >> f. We can also support Polaris-to-Polaris sharing via > > Federation. > > > > > > >> > > > > > > >> Concretely: > > > > > > >> > > > > > > >> - On the provider side, we use the standard Polaris RBAC > model > > > > > > >> (assigning grants to catalog roles, and assigning those > catalog > > > > roles > > > > > > >> to principal roles): > > > > > > >> - Create a dedicated principal with its own > > > > > > client_id/client_secret. > > > > > > >> - Create a catalog role with TABLE_READ_DATA, > TABLE_LIST, > > > and > > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write > > > > > > >> grants). > > > > > > >> - Create a principal role per destination, attach the > > > catalog > > > > > > >> role, and assign the principal. > > > > > > >> - The destination engine uses loadTable with the > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header. > > Polaris > > > > then > > > > > > >> emits temporary, read-only storage credentials, allowing the > > > engine > > > > to > > > > > > >> read the data files directly from the owner's bucket without > > > > accessing > > > > > > >> the rest of the bucket. This achieves zero-copy sharing. > > > > > > >> - On the recipient/consumer side, no special configuration > is > > > > > > >> needed; any IRC client can use it. > > > > > > >> > > > > > > >> 2. What is data sharing not? > > > > > > >> I previously considered adding an Arrow Flight endpoint to > > > > > > >> Polaris, but I no longer think it is a good idea for the > initial > > > > > > >> phase. An Arrow Flight endpoint would serve the data directly, > > > which > > > > > > >> would require Polaris to read the data and expose it. I am > > against > > > > > > >> embedding a query engine in Polaris, as this tight coupling > > feels > > > > like > > > > > > >> an anti-pattern. While we could use a delegation service to > > serve > > > > the > > > > > > >> Flight endpoint, I think it should be considered for a later > > > phase, > > > > if > > > > > > >> at all, since a Flight endpoint might be better suited as a > > > consumer > > > > > > >> responsibility and could encourage data copying. > > > > > > >> > > > > > > >> The first phase should focus on packaging and branding the > > > > > > >> capabilities we already have. > > > > > > >> > > > > > > >> 3. What could the Data Sharing "package" look like in > Polaris? > > > > > > >> To make Data Sharing obvious and easy to use, we should > > > > introduce > > > > > > >> the missing concept of a "share"—a named definition that users > > can > > > > > > >> list, discover, and define. The goal is to materialize the > > user's > > > > > > >> intent ("I want to share these tables with this consumer until > > > this > > > > > > >> date") into standard Polaris entities (principal, principal > > role, > > > > > > >> catalog role, and grants). > > > > > > >> > > > > > > >> Therefore, I propose adding a /shares endpoint to the Polaris > > > > > > >> Management API to package and materialize these share > > definitions > > > > into > > > > > > >> Polaris entities. This would provide a usable, first-version > > > package > > > > > > >> for Polaris Data Sharing. > > > > > > >> > > > > > > >> I hope this addresses Dmitri's questions and provides a more > > > > concrete > > > > > > >> scenario for Polaris Data Sharing. > > > > > > >> > > > > > > >> Thoughts? > > > > > > >> > > > > > > >> Regards, > > > > > > >> JB > > > > > > >> > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <[email protected]> > > > wrote: > > > > > > >> > > > > > > > >> > Hi All, > > > > > > >> > > > > > > > >> > One big emerging enterprise use case coming up as more > people > > > > > > >> consolidate > > > > > > >> > Data Lakehouses and Catalogs is something commonly known as > > > "Data > > > > > > >> Sharing", > > > > > > >> > an more specifically over the course of adoption of open > table > > > > formats > > > > > > >> > "Open Data Sharing". > > > > > > >> > > > > > > > >> > Examples of existing managed service providers' Data Sharing > > > > features: > > > > > > >> > > > > > > > >> > https://www.databricks.com/product/delta-sharing > > > > > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > > > > > >> > > > > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > > > > > >> > > > > > > > > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > > > >> > > > > > > > >> > The basic idea is that when you share data between different > > > > > > companies, > > > > > > >> you > > > > > > >> > need a first-class governance/management layer and extra > > > > > > >> bells-and-whistles > > > > > > >> > that are distinct from just the basic capabilities of RBAC > or > > > > > > >> generalized > > > > > > >> > access-control (i.e. if you're sharing across > > > partially-untrusted > > > > org > > > > > > >> > boundaries, you don't just let the consumer organization log > > > into > > > > your > > > > > > >> > datalake like one of your own employees). > > > > > > >> > > > > > > > >> > JB and I put together this high-level proposal for > supporting > > > Open > > > > > > >> Sharing > > > > > > >> > in Polaris: > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > > > >> > > > > > > > >> > Tentatively, it means adding ~5 logical data model > constructs, > > > > some of > > > > > > >> > which may be a first-class PolarisEntity type, others > subtypes > > > of > > > > > > >> existing > > > > > > >> > entities, and others just a nested construct: > > > > > > >> > > > > > > > >> > - ShareEntity (would behave similarly to a Catalog) > > > > > > >> > - ExternalConsumer (mostly inherits from Principal) > > > > > > >> > - Listing (Similar to a "role grant" but has different > > > > metadata) > > > > > > >> > - EndpointConfig (nested config under Listing) > > > > > > >> > - ShareMembership (Similar to a "securable grant" but > > > different > > > > > > >> metadata) > > > > > > >> > > > > > > > >> > Feedback/comments welcome! I'll also bring it up for live > > > > discussion > > > > > > if > > > > > > >> > there's time in the community sync. > > > > > > >> > > > > > > > >> > Cheers, > > > > > > >> > Dennis > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
