Hi everyone, I am resuming work on this proposal.
Regarding our recent discussion, I will move forward with opening a draft PR for the /api/shares/v1 endpoint to help illustrate the design. I will update this thread once the PR is ready. Thanks, JB On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <[email protected]> wrote: > > Hi Dennis, > > Thanks for the recap! > > I was thinking that share management is probably distinct from catalog > management. A user who already has a realm+catalog may have an option to > manage shares, but not the catalog itself. Would it make sense to separate > the share management API from the general Polaris management API? I mean > using a different prefix for URIs. > > For example: /api/shares/v1/ > > Obviously clients should not make any assumptions about the base path > (/api/shares), which could be different in different deployments (e.g. > going through a gateway). Clients must be able to take the base URI as a > config option. Path segments after /v1/ will be specified by Polaris (Open > API yaml). > > WDYT? > > Thanks, > Dmitri. > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <[email protected]> wrote: > > > Just to recap some of the discussion from today's community sync, Dmitri, > > Robert, and Alex helped raise a few considerations to roll into the next > > design/prototyping phase: > > > > 1. Given the way sharing is packaged as a separate thing, it may not be > > obvious to users that historical snapshots are equally available to > > consumers as the latest snapshot (by virtue of sharing via the standard IRC > > protocol) - we should make sure to document that fact well, including the > > pitfall that deleting rows by updating a table doesn't necessarily mean > > those rows become inaccessible to consumers > > 2. Make sure not to hard-code assumptions about the existence of an > > internal PrincipalEntity backing an ExternalConsumer - even if the first > > MVP implementation uses a nested/hidden Principal of some type that holds a > > Polaris-managed client_id/client_secret, the 3rd-party IdP use cases > > require "implicit" Principals resolved via the oauth token. In such cases > > the ExternalConsumer contains the metadata coordinating the three-way > > handshake between the data-provider, data-consumer, and IdP, but there is > > no PrincipalEntity underlying it. > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics - initially, > > the way an ExternalConsumer works is that the finer-grained consumer users > > share a single ExternalConsumer identity, just like how a FEDERATED catalog > > shares a single service principal in its ConnectionConfig. But in some > > cases the consumer may want some end-to-end attribution of fine-grained > > users, like "on-behalf-of: bob @ consumercompany.com" when the > > ExternalConsumer implicit principal is something like > > "externalconsumer$consumercompany" > > 4. We can iterate more rapidly on a draft spec PR and also expand on the > > detailed technical design > > > > Cheers, > > Dennis > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <[email protected]> wrote: > > > > > Thanks JB for reviving this thread, and belated thanks to all reviewers > > > who have left feedback in the doc or this thread so far! And apologies > > for > > > the silence on this, it's still been on my radar but I got distracted by > > > some other work for awhile. > > > > > > Echoing JB's summary, the proposal is indeed that we're mostly just > > > reusing existing Authz/RBAC machinery, and the translation layer to the > > new > > > sharing concepts sits *on top of* the existing authz engine as much as > > > possible. > > > > > > So even if we're adding layers of indirection on ExternalConsumer/Listing > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end up > > doing > > > the underlying authz just reuse existing Authorizer logic. > > > > > > Based on some other offline discussion with Dmitri, I realized why the > > > discussion around "requirements" may have been confusing: > > > > > > 1. My attempts to shorten the doc inadvertently cut out some of the more > > > "abstract requirements" formulations > > > 2. The project in itself is somewhat more of a "form *as* function" > > > feature than *strictly* functional in nature -- for example, many > > correctly > > > pointed out that "sharing" via the existing RBAC/Principals syntax is > > > already *possible* just by letting a cross-org "consumer" be a > > first-class > > > principal and access your own company's main Polaris catalog endpoints. > > > > > > For (1) I added this to the Requirements section in the doc to better > > > clarify the abstract form of the data model element: > > > > > > *A. Sharing Container*: Container concept for holding things you want to > > > share - “ShareEntity” > > > *B. Consumer Identity*: Special flavor of principal-like entity > > > representing a consumer relationship - “ExternalConsumer” > > > *C. Map Containers to Consumers*: A way to specify letting a given > > > consumer access a given sharing container - “Listing” > > > *D. Consumer Endpoint*: A segregated API access-point for consumers - > > > “EndpointConfig” > > > *E. Objects in Containers*: A way to specify which in-catalog objects > > > belong in the big sharing container - “ShareMembership” > > > > > > This still bakes in a bit of implementation detail since it begs the > > > question why not just let core RBAC/grants handle (C) and (E). To that > > end, > > > and addressing problem (2) of where the "form *is* function", I added a > > > "Concept Mapping" tab: > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > I think the details of more advanced features can still be hashed out > > > incrementally without blocking work on an MVP (and certainly the MVP > > scope > > > could be trimmed so we can launch-and-iterate), so as long as there's > > > community alignment on the general direction I propose we could move > > > forward on basic building blocks. Or if building blocks themselves are > > > controversial we could still proceed with some prototyping across > > different > > > options. > > > > > > Cheers, > > > Dennis > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > >> Hi everyone, > > >> > > >> I would like to resume the discussion about Data Sharing in Polaris > > >> and propose focusing on three major items to help us move forward. > > >> > > >> 1. What is data sharing? > > >> In a nutshell, it is an open protocol proposal allowing > > >> organizations to share live data without copying or replication. The > > >> main goal is zero-copy sharing achieved through shared metadata and > > >> authorization. > > >> > > >> On Polaris, this would mean: > > >> a. The consumer is authenticated as usual. > > >> b. The Polaris server provides the consumer with direct access to the > > >> data files (e.g., Parquet, as specified in the metadata.json). > > >> c. The consumer reads the data files directly using their preferred > > >> tool (e.g., a query engine). > > >> > > >> While Polaris already has most of the necessary underlying > > >> capabilities, we need clear branding/packaging around this use case. > > >> Specifically, this means: > > >> a. A share is a dedicated catalog in the Polaris server, or a catalog > > >> role scoped to specific namespaces. > > >> b. A recipient is represented by a principal and a principal role. > > >> c. A consumer profile consists of the catalog URI and the OAuth2 > > >> client_id/client_secret. > > >> d. We use credential vending (temporary STS credentials) for consumer > > >> access to the data files. > > >> e. We use the Polaris IRC layer to expose the metadata and the file > > >> locations/access. > > >> f. We can also support Polaris-to-Polaris sharing via Federation. > > >> > > >> Concretely: > > >> > > >> - On the provider side, we use the standard Polaris RBAC model > > >> (assigning grants to catalog roles, and assigning those catalog roles > > >> to principal roles): > > >> - Create a dedicated principal with its own > > client_id/client_secret. > > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write > > >> grants). > > >> - Create a principal role per destination, attach the catalog > > >> role, and assign the principal. > > >> - The destination engine uses loadTable with the > > >> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then > > >> emits temporary, read-only storage credentials, allowing the engine to > > >> read the data files directly from the owner's bucket without accessing > > >> the rest of the bucket. This achieves zero-copy sharing. > > >> - On the recipient/consumer side, no special configuration is > > >> needed; any IRC client can use it. > > >> > > >> 2. What is data sharing not? > > >> I previously considered adding an Arrow Flight endpoint to > > >> Polaris, but I no longer think it is a good idea for the initial > > >> phase. An Arrow Flight endpoint would serve the data directly, which > > >> would require Polaris to read the data and expose it. I am against > > >> embedding a query engine in Polaris, as this tight coupling feels like > > >> an anti-pattern. While we could use a delegation service to serve the > > >> Flight endpoint, I think it should be considered for a later phase, if > > >> at all, since a Flight endpoint might be better suited as a consumer > > >> responsibility and could encourage data copying. > > >> > > >> The first phase should focus on packaging and branding the > > >> capabilities we already have. > > >> > > >> 3. What could the Data Sharing "package" look like in Polaris? > > >> To make Data Sharing obvious and easy to use, we should introduce > > >> the missing concept of a "share"—a named definition that users can > > >> list, discover, and define. The goal is to materialize the user's > > >> intent ("I want to share these tables with this consumer until this > > >> date") into standard Polaris entities (principal, principal role, > > >> catalog role, and grants). > > >> > > >> Therefore, I propose adding a /shares endpoint to the Polaris > > >> Management API to package and materialize these share definitions into > > >> Polaris entities. This would provide a usable, first-version package > > >> for Polaris Data Sharing. > > >> > > >> I hope this addresses Dmitri's questions and provides a more concrete > > >> scenario for Polaris Data Sharing. > > >> > > >> Thoughts? > > >> > > >> Regards, > > >> JB > > >> > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <[email protected]> wrote: > > >> > > > >> > Hi All, > > >> > > > >> > One big emerging enterprise use case coming up as more people > > >> consolidate > > >> > Data Lakehouses and Catalogs is something commonly known as "Data > > >> Sharing", > > >> > an more specifically over the course of adoption of open table formats > > >> > "Open Data Sharing". > > >> > > > >> > Examples of existing managed service providers' Data Sharing features: > > >> > > > >> > https://www.databricks.com/product/delta-sharing > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > >> > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > >> > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > >> > > > >> > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > >> > > > >> > The basic idea is that when you share data between different > > companies, > > >> you > > >> > need a first-class governance/management layer and extra > > >> bells-and-whistles > > >> > that are distinct from just the basic capabilities of RBAC or > > >> generalized > > >> > access-control (i.e. if you're sharing across partially-untrusted org > > >> > boundaries, you don't just let the consumer organization log into your > > >> > datalake like one of your own employees). > > >> > > > >> > JB and I put together this high-level proposal for supporting Open > > >> Sharing > > >> > in Polaris: > > >> > > > >> > > > >> > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > >> > > > >> > Tentatively, it means adding ~5 logical data model constructs, some of > > >> > which may be a first-class PolarisEntity type, others subtypes of > > >> existing > > >> > entities, and others just a nested construct: > > >> > > > >> > - ShareEntity (would behave similarly to a Catalog) > > >> > - ExternalConsumer (mostly inherits from Principal) > > >> > - Listing (Similar to a "role grant" but has different metadata) > > >> > - EndpointConfig (nested config under Listing) > > >> > - ShareMembership (Similar to a "securable grant" but different > > >> metadata) > > >> > > > >> > Feedback/comments welcome! I'll also bring it up for live discussion > > if > > >> > there's time in the community sync. > > >> > > > >> > Cheers, > > >> > Dennis > > >> > > > > >
