Just to recap some of the discussion from today's community sync, Dmitri, Robert, and Alex helped raise a few considerations to roll into the next design/prototyping phase:
1. Given the way sharing is packaged as a separate thing, it may not be obvious to users that historical snapshots are equally available to consumers as the latest snapshot (by virtue of sharing via the standard IRC protocol) - we should make sure to document that fact well, including the pitfall that deleting rows by updating a table doesn't necessarily mean those rows become inaccessible to consumers 2. Make sure not to hard-code assumptions about the existence of an internal PrincipalEntity backing an ExternalConsumer - even if the first MVP implementation uses a nested/hidden Principal of some type that holds a Polaris-managed client_id/client_secret, the 3rd-party IdP use cases require "implicit" Principals resolved via the oauth token. In such cases the ExternalConsumer contains the metadata coordinating the three-way handshake between the data-provider, data-consumer, and IdP, but there is no PrincipalEntity underlying it. 3. Longer-term, we may want to tackle "on-behalf-of" semantics - initially, the way an ExternalConsumer works is that the finer-grained consumer users share a single ExternalConsumer identity, just like how a FEDERATED catalog shares a single service principal in its ConnectionConfig. But in some cases the consumer may want some end-to-end attribution of fine-grained users, like "on-behalf-of: bob @ consumercompany.com" when the ExternalConsumer implicit principal is something like "externalconsumer$consumercompany" 4. We can iterate more rapidly on a draft spec PR and also expand on the detailed technical design Cheers, Dennis On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <[email protected]> wrote: > Thanks JB for reviving this thread, and belated thanks to all reviewers > who have left feedback in the doc or this thread so far! And apologies for > the silence on this, it's still been on my radar but I got distracted by > some other work for awhile. > > Echoing JB's summary, the proposal is indeed that we're mostly just > reusing existing Authz/RBAC machinery, and the translation layer to the new > sharing concepts sits *on top of* the existing authz engine as much as > possible. > > So even if we're adding layers of indirection on ExternalConsumer/Listing > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end up doing > the underlying authz just reuse existing Authorizer logic. > > Based on some other offline discussion with Dmitri, I realized why the > discussion around "requirements" may have been confusing: > > 1. My attempts to shorten the doc inadvertently cut out some of the more > "abstract requirements" formulations > 2. The project in itself is somewhat more of a "form *as* function" > feature than *strictly* functional in nature -- for example, many correctly > pointed out that "sharing" via the existing RBAC/Principals syntax is > already *possible* just by letting a cross-org "consumer" be a first-class > principal and access your own company's main Polaris catalog endpoints. > > For (1) I added this to the Requirements section in the doc to better > clarify the abstract form of the data model element: > > *A. Sharing Container*: Container concept for holding things you want to > share - “ShareEntity” > *B. Consumer Identity*: Special flavor of principal-like entity > representing a consumer relationship - “ExternalConsumer” > *C. Map Containers to Consumers*: A way to specify letting a given > consumer access a given sharing container - “Listing” > *D. Consumer Endpoint*: A segregated API access-point for consumers - > “EndpointConfig” > *E. Objects in Containers*: A way to specify which in-catalog objects > belong in the big sharing container - “ShareMembership” > > This still bakes in a bit of implementation detail since it begs the > question why not just let core RBAC/grants handle (C) and (E). To that end, > and addressing problem (2) of where the "form *is* function", I added a > "Concept Mapping" tab: > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > I think the details of more advanced features can still be hashed out > incrementally without blocking work on an MVP (and certainly the MVP scope > could be trimmed so we can launch-and-iterate), so as long as there's > community alignment on the general direction I propose we could move > forward on basic building blocks. Or if building blocks themselves are > controversial we could still proceed with some prototyping across different > options. > > Cheers, > Dennis > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi everyone, >> >> I would like to resume the discussion about Data Sharing in Polaris >> and propose focusing on three major items to help us move forward. >> >> 1. What is data sharing? >> In a nutshell, it is an open protocol proposal allowing >> organizations to share live data without copying or replication. The >> main goal is zero-copy sharing achieved through shared metadata and >> authorization. >> >> On Polaris, this would mean: >> a. The consumer is authenticated as usual. >> b. The Polaris server provides the consumer with direct access to the >> data files (e.g., Parquet, as specified in the metadata.json). >> c. The consumer reads the data files directly using their preferred >> tool (e.g., a query engine). >> >> While Polaris already has most of the necessary underlying >> capabilities, we need clear branding/packaging around this use case. >> Specifically, this means: >> a. A share is a dedicated catalog in the Polaris server, or a catalog >> role scoped to specific namespaces. >> b. A recipient is represented by a principal and a principal role. >> c. A consumer profile consists of the catalog URI and the OAuth2 >> client_id/client_secret. >> d. We use credential vending (temporary STS credentials) for consumer >> access to the data files. >> e. We use the Polaris IRC layer to expose the metadata and the file >> locations/access. >> f. We can also support Polaris-to-Polaris sharing via Federation. >> >> Concretely: >> >> - On the provider side, we use the standard Polaris RBAC model >> (assigning grants to catalog roles, and assigning those catalog roles >> to principal roles): >> - Create a dedicated principal with its own client_id/client_secret. >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and >> NAMESPACE_LIST grants on the namespaces to be shared (no write >> grants). >> - Create a principal role per destination, attach the catalog >> role, and assign the principal. >> - The destination engine uses loadTable with the >> "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then >> emits temporary, read-only storage credentials, allowing the engine to >> read the data files directly from the owner's bucket without accessing >> the rest of the bucket. This achieves zero-copy sharing. >> - On the recipient/consumer side, no special configuration is >> needed; any IRC client can use it. >> >> 2. What is data sharing not? >> I previously considered adding an Arrow Flight endpoint to >> Polaris, but I no longer think it is a good idea for the initial >> phase. An Arrow Flight endpoint would serve the data directly, which >> would require Polaris to read the data and expose it. I am against >> embedding a query engine in Polaris, as this tight coupling feels like >> an anti-pattern. While we could use a delegation service to serve the >> Flight endpoint, I think it should be considered for a later phase, if >> at all, since a Flight endpoint might be better suited as a consumer >> responsibility and could encourage data copying. >> >> The first phase should focus on packaging and branding the >> capabilities we already have. >> >> 3. What could the Data Sharing "package" look like in Polaris? >> To make Data Sharing obvious and easy to use, we should introduce >> the missing concept of a "share"—a named definition that users can >> list, discover, and define. The goal is to materialize the user's >> intent ("I want to share these tables with this consumer until this >> date") into standard Polaris entities (principal, principal role, >> catalog role, and grants). >> >> Therefore, I propose adding a /shares endpoint to the Polaris >> Management API to package and materialize these share definitions into >> Polaris entities. This would provide a usable, first-version package >> for Polaris Data Sharing. >> >> I hope this addresses Dmitri's questions and provides a more concrete >> scenario for Polaris Data Sharing. >> >> Thoughts? >> >> Regards, >> JB >> >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <[email protected]> wrote: >> > >> > Hi All, >> > >> > One big emerging enterprise use case coming up as more people >> consolidate >> > Data Lakehouses and Catalogs is something commonly known as "Data >> Sharing", >> > an more specifically over the course of adoption of open table formats >> > "Open Data Sharing". >> > >> > Examples of existing managed service providers' Data Sharing features: >> > >> > https://www.databricks.com/product/delta-sharing >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro >> > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html >> > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction >> > >> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview >> > >> > The basic idea is that when you share data between different companies, >> you >> > need a first-class governance/management layer and extra >> bells-and-whistles >> > that are distinct from just the basic capabilities of RBAC or >> generalized >> > access-control (i.e. if you're sharing across partially-untrusted org >> > boundaries, you don't just let the consumer organization log into your >> > datalake like one of your own employees). >> > >> > JB and I put together this high-level proposal for supporting Open >> Sharing >> > in Polaris: >> > >> > >> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing >> > >> > Tentatively, it means adding ~5 logical data model constructs, some of >> > which may be a first-class PolarisEntity type, others subtypes of >> existing >> > entities, and others just a nested construct: >> > >> > - ShareEntity (would behave similarly to a Catalog) >> > - ExternalConsumer (mostly inherits from Principal) >> > - Listing (Similar to a "role grant" but has different metadata) >> > - EndpointConfig (nested config under Listing) >> > - ShareMembership (Similar to a "securable grant" but different >> metadata) >> > >> > Feedback/comments welcome! I'll also bring it up for live discussion if >> > there's time in the community sync. >> > >> > Cheers, >> > Dennis >> >
