Hi everyone, Thank you for reviving this, and for the concrete spec in PR 5446.. the journeys doc was especially helpful for seeing the end-to-end flow.
I agree this is worth doing, and that v1 is more than a tutorial on today's RBAC. A partner principal on /api/catalog is already possible; what seems valuable is a first-class share (enumerated members, a restricted consumer, a listing as the binding and audit unit) plus a segregated, read-only Iceberg REST surface with credential vending. Stock Iceberg REST as the consumer protocol, and keeping Flight and Polaris-to-Polaris federation out of v1, all sound right to me. I don't have a strong view yet on the URL layout (management vs shares prefix, nested vs top-level, prefix keyed by share vs listing) and am happy to follow whatever the community prefers there. A few things I wanted to check on the spec itself, several of these echo points already raised on the list: 1. Membership as source of truth. If a hidden catalog role is independently grantable, a share could be widened without going through the share APIs. Would it make sense to say the data plane authorizes against membership + listing, and that any backing roles are not user-visible / not independently grantable? 2. ExternalConsumer and principals. The Aug 7 recap already noted we should not assume a PrincipalEntity behind every ExternalConsumer. The journeys still describe granting the consumer a share principal role, I assume that is an implementation sketch for v1 client_id/secret, not the public model? 3. Listing expiration. JB's original note was "share these tables with this consumer until this date." Credential expiry is a bit different. Would a listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for v1? 4. Time-travel. As raised in the community sync, sharing via Iceberg REST means historical snapshots remain readable. Totally fine for v1 to support only full history, would an explicit field on the share help so a later CURRENT_ONLY option is not a breaking surprise? 5. Consumer data plane. The current file is control-plane only, which is a good first step. It might help to list the Iceberg routes that are mounted, that writes 404, that non-members 404, and that /config does not return provider catalog properties, so implementations don't diverge. Two smaller notes, if useful: in the J1 journey the consumer still posts to /api/catalog/v1/oauth/tokens, if partners are not supposed to touch the internal catalog API, should the default token endpoint live on the shares surface (or an external IdP)? And on Dmitri's point about share operators vs catalog operators, a SHARE_MANAGE privilege on the bound catalog (independent of CATALOG_MANAGE_ACCESS) would make that work on any URL prefix. Happy to review the next revision, and thanks again to both of you for driving this. Regards, Prithvi S On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <[email protected]> wrote: > As discussed live, looks like we both might have some drafts we can compare > and choose/merge from :) > > Here's my current draft spec PR; it opts for my preferred syntax choices > for now (/api/management/ for share-management, /api/shares/ for consumer > data-plane): https://github.com/apache/polaris/pull/5446 > > Since we were talking about preferring mdfiles for design iteration here's > a companion design-addendum which just enumerates what the end-to-end user > journeys look like syntactically for each combination of design choices: > > > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md > > Please let me know if anyone prefers a Google Doc version of the > design-addendum for easier commenting, or if the design-addendum itself > should also be modeled as a PR on some free-floating branch just to capture > comments. > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <[email protected]> > wrote: > > > Hi everyone, > > > > I am resuming work on this proposal. > > > > Regarding our recent discussion, I will move forward with opening a > > draft PR for the /api/shares/v1 endpoint to help illustrate the > > design. > > > > I will update this thread once the PR is ready. > > > > Thanks, > > JB > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > > Hi Dennis, > > > > > > Thanks for the recap! > > > > > > I was thinking that share management is probably distinct from catalog > > > management. A user who already has a realm+catalog may have an option > to > > > manage shares, but not the catalog itself. Would it make sense to > > separate > > > the share management API from the general Polaris management API? I > mean > > > using a different prefix for URIs. > > > > > > For example: /api/shares/v1/ > > > > > > Obviously clients should not make any assumptions about the base path > > > (/api/shares), which could be different in different deployments (e.g. > > > going through a gateway). Clients must be able to take the base URI as > a > > > config option. Path segments after /v1/ will be specified by Polaris > > (Open > > > API yaml). > > > > > > WDYT? > > > > > > Thanks, > > > Dmitri. > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <[email protected]> wrote: > > > > > > > Just to recap some of the discussion from today's community sync, > > Dmitri, > > > > Robert, and Alex helped raise a few considerations to roll into the > > next > > > > design/prototyping phase: > > > > > > > > 1. Given the way sharing is packaged as a separate thing, it may not > be > > > > obvious to users that historical snapshots are equally available to > > > > consumers as the latest snapshot (by virtue of sharing via the > > standard IRC > > > > protocol) - we should make sure to document that fact well, including > > the > > > > pitfall that deleting rows by updating a table doesn't necessarily > mean > > > > those rows become inaccessible to consumers > > > > 2. Make sure not to hard-code assumptions about the existence of an > > > > internal PrincipalEntity backing an ExternalConsumer - even if the > > first > > > > MVP implementation uses a nested/hidden Principal of some type that > > holds a > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use cases > > > > require "implicit" Principals resolved via the oauth token. In such > > cases > > > > the ExternalConsumer contains the metadata coordinating the three-way > > > > handshake between the data-provider, data-consumer, and IdP, but > there > > is > > > > no PrincipalEntity underlying it. > > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics - > > initially, > > > > the way an ExternalConsumer works is that the finer-grained consumer > > users > > > > share a single ExternalConsumer identity, just like how a FEDERATED > > catalog > > > > shares a single service principal in its ConnectionConfig. But in > some > > > > cases the consumer may want some end-to-end attribution of > fine-grained > > > > users, like "on-behalf-of: bob @ consumercompany.com" when the > > > > ExternalConsumer implicit principal is something like > > > > "externalconsumer$consumercompany" > > > > 4. We can iterate more rapidly on a draft spec PR and also expand on > > the > > > > detailed technical design > > > > > > > > Cheers, > > > > Dennis > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <[email protected]> wrote: > > > > > > > > > Thanks JB for reviving this thread, and belated thanks to all > > reviewers > > > > > who have left feedback in the doc or this thread so far! And > > apologies > > > > for > > > > > the silence on this, it's still been on my radar but I got > > distracted by > > > > > some other work for awhile. > > > > > > > > > > Echoing JB's summary, the proposal is indeed that we're mostly just > > > > > reusing existing Authz/RBAC machinery, and the translation layer to > > the > > > > new > > > > > sharing concepts sits *on top of* the existing authz engine as much > > as > > > > > possible. > > > > > > > > > > So even if we're adding layers of indirection on > > ExternalConsumer/Listing > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end > up > > > > doing > > > > > the underlying authz just reuse existing Authorizer logic. > > > > > > > > > > Based on some other offline discussion with Dmitri, I realized why > > the > > > > > discussion around "requirements" may have been confusing: > > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut out some of the > > more > > > > > "abstract requirements" formulations > > > > > 2. The project in itself is somewhat more of a "form *as* function" > > > > > feature than *strictly* functional in nature -- for example, many > > > > correctly > > > > > pointed out that "sharing" via the existing RBAC/Principals syntax > is > > > > > already *possible* just by letting a cross-org "consumer" be a > > > > first-class > > > > > principal and access your own company's main Polaris catalog > > endpoints. > > > > > > > > > > For (1) I added this to the Requirements section in the doc to > better > > > > > clarify the abstract form of the data model element: > > > > > > > > > > *A. Sharing Container*: Container concept for holding things you > > want to > > > > > share - “ShareEntity” > > > > > *B. Consumer Identity*: Special flavor of principal-like entity > > > > > representing a consumer relationship - “ExternalConsumer” > > > > > *C. Map Containers to Consumers*: A way to specify letting a given > > > > > consumer access a given sharing container - “Listing” > > > > > *D. Consumer Endpoint*: A segregated API access-point for > consumers - > > > > > “EndpointConfig” > > > > > *E. Objects in Containers*: A way to specify which in-catalog > objects > > > > > belong in the big sharing container - “ShareMembership” > > > > > > > > > > This still bakes in a bit of implementation detail since it begs > the > > > > > question why not just let core RBAC/grants handle (C) and (E). To > > that > > > > end, > > > > > and addressing problem (2) of where the "form *is* function", I > > added a > > > > > "Concept Mapping" tab: > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > > > > > I think the details of more advanced features can still be hashed > out > > > > > incrementally without blocking work on an MVP (and certainly the > MVP > > > > scope > > > > > could be trimmed so we can launch-and-iterate), so as long as > there's > > > > > community alignment on the general direction I propose we could > move > > > > > forward on basic building blocks. Or if building blocks themselves > > are > > > > > controversial we could still proceed with some prototyping across > > > > different > > > > > options. > > > > > > > > > > Cheers, > > > > > Dennis > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré < > > [email protected]> > > > > > wrote: > > > > > > > > > >> Hi everyone, > > > > >> > > > > >> I would like to resume the discussion about Data Sharing in > Polaris > > > > >> and propose focusing on three major items to help us move forward. > > > > >> > > > > >> 1. What is data sharing? > > > > >> In a nutshell, it is an open protocol proposal allowing > > > > >> organizations to share live data without copying or replication. > The > > > > >> main goal is zero-copy sharing achieved through shared metadata > and > > > > >> authorization. > > > > >> > > > > >> On Polaris, this would mean: > > > > >> a. The consumer is authenticated as usual. > > > > >> b. The Polaris server provides the consumer with direct access to > > the > > > > >> data files (e.g., Parquet, as specified in the metadata.json). > > > > >> c. The consumer reads the data files directly using their > preferred > > > > >> tool (e.g., a query engine). > > > > >> > > > > >> While Polaris already has most of the necessary underlying > > > > >> capabilities, we need clear branding/packaging around this use > case. > > > > >> Specifically, this means: > > > > >> a. A share is a dedicated catalog in the Polaris server, or a > > catalog > > > > >> role scoped to specific namespaces. > > > > >> b. A recipient is represented by a principal and a principal role. > > > > >> c. A consumer profile consists of the catalog URI and the OAuth2 > > > > >> client_id/client_secret. > > > > >> d. We use credential vending (temporary STS credentials) for > > consumer > > > > >> access to the data files. > > > > >> e. We use the Polaris IRC layer to expose the metadata and the > file > > > > >> locations/access. > > > > >> f. We can also support Polaris-to-Polaris sharing via Federation. > > > > >> > > > > >> Concretely: > > > > >> > > > > >> - On the provider side, we use the standard Polaris RBAC model > > > > >> (assigning grants to catalog roles, and assigning those catalog > > roles > > > > >> to principal roles): > > > > >> - Create a dedicated principal with its own > > > > client_id/client_secret. > > > > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, > and > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write > > > > >> grants). > > > > >> - Create a principal role per destination, attach the > catalog > > > > >> role, and assign the principal. > > > > >> - The destination engine uses loadTable with the > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris > > then > > > > >> emits temporary, read-only storage credentials, allowing the > engine > > to > > > > >> read the data files directly from the owner's bucket without > > accessing > > > > >> the rest of the bucket. This achieves zero-copy sharing. > > > > >> - On the recipient/consumer side, no special configuration is > > > > >> needed; any IRC client can use it. > > > > >> > > > > >> 2. What is data sharing not? > > > > >> I previously considered adding an Arrow Flight endpoint to > > > > >> Polaris, but I no longer think it is a good idea for the initial > > > > >> phase. An Arrow Flight endpoint would serve the data directly, > which > > > > >> would require Polaris to read the data and expose it. I am against > > > > >> embedding a query engine in Polaris, as this tight coupling feels > > like > > > > >> an anti-pattern. While we could use a delegation service to serve > > the > > > > >> Flight endpoint, I think it should be considered for a later > phase, > > if > > > > >> at all, since a Flight endpoint might be better suited as a > consumer > > > > >> responsibility and could encourage data copying. > > > > >> > > > > >> The first phase should focus on packaging and branding the > > > > >> capabilities we already have. > > > > >> > > > > >> 3. What could the Data Sharing "package" look like in Polaris? > > > > >> To make Data Sharing obvious and easy to use, we should > > introduce > > > > >> the missing concept of a "share"—a named definition that users can > > > > >> list, discover, and define. The goal is to materialize the user's > > > > >> intent ("I want to share these tables with this consumer until > this > > > > >> date") into standard Polaris entities (principal, principal role, > > > > >> catalog role, and grants). > > > > >> > > > > >> Therefore, I propose adding a /shares endpoint to the Polaris > > > > >> Management API to package and materialize these share definitions > > into > > > > >> Polaris entities. This would provide a usable, first-version > package > > > > >> for Polaris Data Sharing. > > > > >> > > > > >> I hope this addresses Dmitri's questions and provides a more > > concrete > > > > >> scenario for Polaris Data Sharing. > > > > >> > > > > >> Thoughts? > > > > >> > > > > >> Regards, > > > > >> JB > > > > >> > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <[email protected]> > wrote: > > > > >> > > > > > >> > Hi All, > > > > >> > > > > > >> > One big emerging enterprise use case coming up as more people > > > > >> consolidate > > > > >> > Data Lakehouses and Catalogs is something commonly known as > "Data > > > > >> Sharing", > > > > >> > an more specifically over the course of adoption of open table > > formats > > > > >> > "Open Data Sharing". > > > > >> > > > > > >> > Examples of existing managed service providers' Data Sharing > > features: > > > > >> > > > > > >> > https://www.databricks.com/product/delta-sharing > > > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > > > >> > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > > > >> > > > > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > > >> > > > > > >> > > > > > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > >> > > > > > >> > The basic idea is that when you share data between different > > > > companies, > > > > >> you > > > > >> > need a first-class governance/management layer and extra > > > > >> bells-and-whistles > > > > >> > that are distinct from just the basic capabilities of RBAC or > > > > >> generalized > > > > >> > access-control (i.e. if you're sharing across > partially-untrusted > > org > > > > >> > boundaries, you don't just let the consumer organization log > into > > your > > > > >> > datalake like one of your own employees). > > > > >> > > > > > >> > JB and I put together this high-level proposal for supporting > Open > > > > >> Sharing > > > > >> > in Polaris: > > > > >> > > > > > >> > > > > > >> > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > >> > > > > > >> > Tentatively, it means adding ~5 logical data model constructs, > > some of > > > > >> > which may be a first-class PolarisEntity type, others subtypes > of > > > > >> existing > > > > >> > entities, and others just a nested construct: > > > > >> > > > > > >> > - ShareEntity (would behave similarly to a Catalog) > > > > >> > - ExternalConsumer (mostly inherits from Principal) > > > > >> > - Listing (Similar to a "role grant" but has different > > metadata) > > > > >> > - EndpointConfig (nested config under Listing) > > > > >> > - ShareMembership (Similar to a "securable grant" but > different > > > > >> metadata) > > > > >> > > > > > >> > Feedback/comments welcome! I'll also bring it up for live > > discussion > > > > if > > > > >> > there's time in the community sync. > > > > >> > > > > > >> > Cheers, > > > > >> > Dennis > > > > >> > > > > > > > > > > > >
