Thanks for the review Prithvi! All great questions. Also, I realize I forgot to address Dmitri's comment:
Would it make sense to separate > the share management API from the general Polaris management API? I mean > using a different prefix for URIs. > > For example: /api/shares/v1/ Right now IIRC the mental model is that "everything considered data plane" ends up under /api/catalog/ and "everything considered admin plane" ends up under /api/management/, and basically spans all the RBAC *and* catalogs. I agree the *privileges* themselves should be separated between catalog stuff, principal/role stuff, and sharing stuff. Some pointers for posterity since it's a bit hard to keep track of how the basePath that causes /api/catalog/ is kind of hidden in https://github.com/apache/polaris/blob/7866ce5c009d64e4c1454b47bca3df9873c6c0b2/api/iceberg-service/build.gradle.kts#L86 : serverVariables.put("basePath", "api/catalog") while the /api/management is hard-coded into the yaml: https://github.com/apache/polaris/blob/b42f931a461f6b96512f7de6955c7c09b5e413fa/spec/polaris-management-service.yml#L26 servers: - url: "{scheme}://{host}/api/management/v1" Using "/api/shares/" adds an orthogonal carveout. At the very least I think we'd want to keep share-management segregated from share-data-plane. For the data plane, remapping that under "/api/shares/v1" makes sense since we want the entire subtree of share-related *data plane* operations to match the layout of a catalog, so we're basically saying a consumer client can just get a catalogBasePath of /api/shares/v1/ instead of /api/catalog/v1/ and everything under it is the same. If we did split management out, I guess we'd put it under "/api/shares-management/v1/" instead or similar. This is plausible, though the argument in favor of keeping it in /api/management/v1/ is that therre are referential relatinships between the shares and catalogs, and along with the way Listings/ExternalConsumers behave like other management-layer counterparts, these all belong to the same "family" of API surfaces. The yaml files themselves can still be separate to help manageability/modularity of the codebase, the same way we have various /api/catalog/v1/ extensions split into different yamls. I don't feel too strongly though as long as we segregate the prefix for control-plane vs data-plane. WDYT? Do you prefer /api/shares-management/v1/ instead of addint to /api/management/v1? Other responses to Prithvi's questions inline: 1. Membership as source of truth. If a hidden catalog role is independently > grantable, a share could be widened without going through the share APIs. > Would it make sense to say the data plane authorizes against membership + > listing, and that any backing roles are not user-visible / not > independently grantable? > Yes, good point; if we use the grants as a hidden implementation detail we need to make sure they're only modified by the share-management system. The precedent for this is the "system$" reserved-prefix that was briefly discussed most recently here https://github.com/apache/polaris/pull/4629#discussion_r3372018934 I think we don't yet have ingrained usage of it beyond customized provider-specific versions of managed Polaris, so we'll need to be careful that the "system protection" around these entities holds up, but it's at least possible to do. > 2. ExternalConsumer and principals. The Aug 7 recap already noted we should > not assume a PrincipalEntity behind every ExternalConsumer. The journeys > still describe granting the consumer a share principal role, I assume that > is an implementation sketch for v1 client_id/secret, not the public model? > Good nuance, I admittedly gloss over representation of federated identities as ExternalConsumers in the current journeys doc, and I need to make sure the federated-identity case is called out as one of the caveats for the PrincipalRole approach (compared to the options where a share-entity is literally defined as a *child* of a catatalog, and only a CatalogRole needs to participate). Note that the client_id/secret approach is still part of the "public" model, but I'l assume you specifically mean the *federated* public model. For the federated case, it should still be possible to use the system$ type of hidden PrincipalRole, we just need the resolution of the share-attached PrincipalRole *not* to rely on the GrantPrinciplRoleToPrincipal (where the principal would be an ExternalConsumer's hidden principal in the v1 case), but allow the *Listing* itself to be the surrogate resolution path from a virtual-Principal (the federated identity) to the same hidden share PrincipalRole. This is a good design detail to track though - whether the ExternalConsumer<->(Hidden) PrincipalRole is statically resolved at Listing-creation time or by dynamically evaluating the Listing at consumer-query time. We'd need a way to resolve the listing from the federated auth flow. > 3. Listing expiration. JB's original note was "share these tables with this > consumer until this date." Credential expiry is a bit different. Would a > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in scope for v1? > Generally I'd say v1 scope should be just focused on the minimal set of functionality needed, so by default I definitely wouldn't say it's a "blocker" for v1, but I guess we can still debate relative prioritization against other bells-and-whistles. If the nature of the sharing use cases makes this a high-demand feature there's no reason we can't try to add it to earlier scope. > 4. Time-travel. As raised in the community sync, sharing via Iceberg REST > means historical snapshots remain readable. Totally fine for v1 to support > only full history, would an explicit field on the share help so a later > CURRENT_ONLY option is not a breaking surprise? > Yeah, I think the most important thing is to have a line-of-sight up front for how we *could* implement narrower privileges. Offhand it seems like in this case adding a field later should be easy to do in a non-breaking way, but I don't feel too strongly. We could make it an enum with only one available option for now, ALL_HISTORICAL_SNAPSHOTS. Prerequisites would probably be full ScanPlan API support and HTTP presigned URL support in Iceberg so that we could vend file-granularity credentials. > 5. Consumer data plane. The current file is control-plane only, which is a > good first step. It might help to list the Iceberg routes that are mounted, > that writes 404, that non-members 404, and that /config does not return > provider catalog properties, so implementations don't diverge. > Good catch, I was actually just working on some prototyping for whether to autogen vs copy/paste parts of the data-plane api definitions. I'll post a followup shortly. Looks like there's also a related fix even for the basic non-sharing-related GetCatalog case where we need to tighten the semantics of whether /config returnns catalog properties: https://github.com/apache/polaris/pull/5246 This will be a good category of design pitfalls to keep track of where we need to be careful to limit intra-entity segregation of intended access/exposure through a share (for example, whether table-default.* properties set on a Catalog or Namespace should ever be served up in a LoadTableResponse when accessed through a Share) > > Two smaller notes, if useful: in the J1 journey the consumer still posts to > /api/catalog/v1/oauth/tokens, if partners are not supposed to touch the > internal catalog API, should the default token endpoint live on the shares > surface (or an external IdP)? And on Dmitri's point about share operators > vs catalog operators, a SHARE_MANAGE privilege on the bound catalog > (independent of CATALOG_MANAGE_ACCESS) would make that work on any URL > prefix. > > Long term probably want everything to converge more on external IdP. But you're right that as long as we *do* support the Polaris-independent OAuth cases (which I think we will for the foreseeable future), mounting under /api/shares/v1/oauth/tokens is probably best. In concept the in-catalog OAuth deprecation in the Iceberg REST spec doesn't necessarily preclude provider self-owned OAuth, just that the Iceberg layer doesn't mandate it; so the "clean" formulation would be that the catalog provider who *chooses* to also be the oauth provider (in this case, a self-sufficient Polaris instance) could choose to put their token URL at a different basePath entirely like /api/iam/v1/oauth/tokens, and one could argue whether catalog-access and shares-access should share that one. But for now while we still support the version of the IRC spec that *does* have the oauth endpoint defined, I guess the best of all worlds is indeed to have /api/catalog/v1/oauth/tokens be segregated from /api/shares/v1/oauth/tokens, with the coincidental benefit of better structural segregation for the sharing consumers anyways. Good point about the privileges, I agree that having the right target/secondary privileges for share-related AuthorizableOperations should enable segregating the personas. > Happy to review the next revision, and thanks again to both of you for > driving this. > > Regards, > Prithvi S > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <[email protected]> wrote: > > > As discussed live, looks like we both might have some drafts we can > compare > > and choose/merge from :) > > > > Here's my current draft spec PR; it opts for my preferred syntax choices > > for now (/api/management/ for share-management, /api/shares/ for consumer > > data-plane): https://github.com/apache/polaris/pull/5446 > > > > Since we were talking about preferring mdfiles for design iteration > here's > > a companion design-addendum which just enumerates what the end-to-end > user > > journeys look like syntactically for each combination of design choices: > > > > > > > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md > > > > Please let me know if anyone prefers a Google Doc version of the > > design-addendum for easier commenting, or if the design-addendum itself > > should also be modeled as a PR on some free-floating branch just to > capture > > comments. > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi everyone, > > > > > > I am resuming work on this proposal. > > > > > > Regarding our recent discussion, I will move forward with opening a > > > draft PR for the /api/shares/v1 endpoint to help illustrate the > > > design. > > > > > > I will update this thread once the PR is ready. > > > > > > Thanks, > > > JB > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > > Hi Dennis, > > > > > > > > Thanks for the recap! > > > > > > > > I was thinking that share management is probably distinct from > catalog > > > > management. A user who already has a realm+catalog may have an option > > to > > > > manage shares, but not the catalog itself. Would it make sense to > > > separate > > > > the share management API from the general Polaris management API? I > > mean > > > > using a different prefix for URIs. > > > > > > > > For example: /api/shares/v1/ > > > > > > > > Obviously clients should not make any assumptions about the base path > > > > (/api/shares), which could be different in different deployments > (e.g. > > > > going through a gateway). Clients must be able to take the base URI > as > > a > > > > config option. Path segments after /v1/ will be specified by Polaris > > > (Open > > > > API yaml). > > > > > > > > WDYT? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo <[email protected]> wrote: > > > > > > > > > Just to recap some of the discussion from today's community sync, > > > Dmitri, > > > > > Robert, and Alex helped raise a few considerations to roll into the > > > next > > > > > design/prototyping phase: > > > > > > > > > > 1. Given the way sharing is packaged as a separate thing, it may > not > > be > > > > > obvious to users that historical snapshots are equally available to > > > > > consumers as the latest snapshot (by virtue of sharing via the > > > standard IRC > > > > > protocol) - we should make sure to document that fact well, > including > > > the > > > > > pitfall that deleting rows by updating a table doesn't necessarily > > mean > > > > > those rows become inaccessible to consumers > > > > > 2. Make sure not to hard-code assumptions about the existence of an > > > > > internal PrincipalEntity backing an ExternalConsumer - even if the > > > first > > > > > MVP implementation uses a nested/hidden Principal of some type that > > > holds a > > > > > Polaris-managed client_id/client_secret, the 3rd-party IdP use > cases > > > > > require "implicit" Principals resolved via the oauth token. In such > > > cases > > > > > the ExternalConsumer contains the metadata coordinating the > three-way > > > > > handshake between the data-provider, data-consumer, and IdP, but > > there > > > is > > > > > no PrincipalEntity underlying it. > > > > > 3. Longer-term, we may want to tackle "on-behalf-of" semantics - > > > initially, > > > > > the way an ExternalConsumer works is that the finer-grained > consumer > > > users > > > > > share a single ExternalConsumer identity, just like how a FEDERATED > > > catalog > > > > > shares a single service principal in its ConnectionConfig. But in > > some > > > > > cases the consumer may want some end-to-end attribution of > > fine-grained > > > > > users, like "on-behalf-of: bob @ consumercompany.com" when the > > > > > ExternalConsumer implicit principal is something like > > > > > "externalconsumer$consumercompany" > > > > > 4. We can iterate more rapidly on a draft spec PR and also expand > on > > > the > > > > > detailed technical design > > > > > > > > > > Cheers, > > > > > Dennis > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo <[email protected]> > wrote: > > > > > > > > > > > Thanks JB for reviving this thread, and belated thanks to all > > > reviewers > > > > > > who have left feedback in the doc or this thread so far! And > > > apologies > > > > > for > > > > > > the silence on this, it's still been on my radar but I got > > > distracted by > > > > > > some other work for awhile. > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed that we're mostly > just > > > > > > reusing existing Authz/RBAC machinery, and the translation layer > to > > > the > > > > > new > > > > > > sharing concepts sits *on top of* the existing authz engine as > much > > > as > > > > > > possible. > > > > > > > > > > > > So even if we're adding layers of indirection on > > > ExternalConsumer/Listing > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the parts that end > > up > > > > > doing > > > > > > the underlying authz just reuse existing Authorizer logic. > > > > > > > > > > > > Based on some other offline discussion with Dmitri, I realized > why > > > the > > > > > > discussion around "requirements" may have been confusing: > > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut out some of > the > > > more > > > > > > "abstract requirements" formulations > > > > > > 2. The project in itself is somewhat more of a "form *as* > function" > > > > > > feature than *strictly* functional in nature -- for example, many > > > > > correctly > > > > > > pointed out that "sharing" via the existing RBAC/Principals > syntax > > is > > > > > > already *possible* just by letting a cross-org "consumer" be a > > > > > first-class > > > > > > principal and access your own company's main Polaris catalog > > > endpoints. > > > > > > > > > > > > For (1) I added this to the Requirements section in the doc to > > better > > > > > > clarify the abstract form of the data model element: > > > > > > > > > > > > *A. Sharing Container*: Container concept for holding things you > > > want to > > > > > > share - “ShareEntity” > > > > > > *B. Consumer Identity*: Special flavor of principal-like entity > > > > > > representing a consumer relationship - “ExternalConsumer” > > > > > > *C. Map Containers to Consumers*: A way to specify letting a > given > > > > > > consumer access a given sharing container - “Listing” > > > > > > *D. Consumer Endpoint*: A segregated API access-point for > > consumers - > > > > > > “EndpointConfig” > > > > > > *E. Objects in Containers*: A way to specify which in-catalog > > objects > > > > > > belong in the big sharing container - “ShareMembership” > > > > > > > > > > > > This still bakes in a bit of implementation detail since it begs > > the > > > > > > question why not just let core RBAC/grants handle (C) and (E). To > > > that > > > > > end, > > > > > > and addressing problem (2) of where the "form *is* function", I > > > added a > > > > > > "Concept Mapping" tab: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > > > > > > > I think the details of more advanced features can still be hashed > > out > > > > > > incrementally without blocking work on an MVP (and certainly the > > MVP > > > > > scope > > > > > > could be trimmed so we can launch-and-iterate), so as long as > > there's > > > > > > community alignment on the general direction I propose we could > > move > > > > > > forward on basic building blocks. Or if building blocks > themselves > > > are > > > > > > controversial we could still proceed with some prototyping across > > > > > different > > > > > > options. > > > > > > > > > > > > Cheers, > > > > > > Dennis > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > >> Hi everyone, > > > > > >> > > > > > >> I would like to resume the discussion about Data Sharing in > > Polaris > > > > > >> and propose focusing on three major items to help us move > forward. > > > > > >> > > > > > >> 1. What is data sharing? > > > > > >> In a nutshell, it is an open protocol proposal allowing > > > > > >> organizations to share live data without copying or replication. > > The > > > > > >> main goal is zero-copy sharing achieved through shared metadata > > and > > > > > >> authorization. > > > > > >> > > > > > >> On Polaris, this would mean: > > > > > >> a. The consumer is authenticated as usual. > > > > > >> b. The Polaris server provides the consumer with direct access > to > > > the > > > > > >> data files (e.g., Parquet, as specified in the metadata.json). > > > > > >> c. The consumer reads the data files directly using their > > preferred > > > > > >> tool (e.g., a query engine). > > > > > >> > > > > > >> While Polaris already has most of the necessary underlying > > > > > >> capabilities, we need clear branding/packaging around this use > > case. > > > > > >> Specifically, this means: > > > > > >> a. A share is a dedicated catalog in the Polaris server, or a > > > catalog > > > > > >> role scoped to specific namespaces. > > > > > >> b. A recipient is represented by a principal and a principal > role. > > > > > >> c. A consumer profile consists of the catalog URI and the OAuth2 > > > > > >> client_id/client_secret. > > > > > >> d. We use credential vending (temporary STS credentials) for > > > consumer > > > > > >> access to the data files. > > > > > >> e. We use the Polaris IRC layer to expose the metadata and the > > file > > > > > >> locations/access. > > > > > >> f. We can also support Polaris-to-Polaris sharing via > Federation. > > > > > >> > > > > > >> Concretely: > > > > > >> > > > > > >> - On the provider side, we use the standard Polaris RBAC model > > > > > >> (assigning grants to catalog roles, and assigning those catalog > > > roles > > > > > >> to principal roles): > > > > > >> - Create a dedicated principal with its own > > > > > client_id/client_secret. > > > > > >> - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, > > and > > > > > >> NAMESPACE_LIST grants on the namespaces to be shared (no write > > > > > >> grants). > > > > > >> - Create a principal role per destination, attach the > > catalog > > > > > >> role, and assign the principal. > > > > > >> - The destination engine uses loadTable with the > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" header. > Polaris > > > then > > > > > >> emits temporary, read-only storage credentials, allowing the > > engine > > > to > > > > > >> read the data files directly from the owner's bucket without > > > accessing > > > > > >> the rest of the bucket. This achieves zero-copy sharing. > > > > > >> - On the recipient/consumer side, no special configuration is > > > > > >> needed; any IRC client can use it. > > > > > >> > > > > > >> 2. What is data sharing not? > > > > > >> I previously considered adding an Arrow Flight endpoint to > > > > > >> Polaris, but I no longer think it is a good idea for the initial > > > > > >> phase. An Arrow Flight endpoint would serve the data directly, > > which > > > > > >> would require Polaris to read the data and expose it. I am > against > > > > > >> embedding a query engine in Polaris, as this tight coupling > feels > > > like > > > > > >> an anti-pattern. While we could use a delegation service to > serve > > > the > > > > > >> Flight endpoint, I think it should be considered for a later > > phase, > > > if > > > > > >> at all, since a Flight endpoint might be better suited as a > > consumer > > > > > >> responsibility and could encourage data copying. > > > > > >> > > > > > >> The first phase should focus on packaging and branding the > > > > > >> capabilities we already have. > > > > > >> > > > > > >> 3. What could the Data Sharing "package" look like in Polaris? > > > > > >> To make Data Sharing obvious and easy to use, we should > > > introduce > > > > > >> the missing concept of a "share"—a named definition that users > can > > > > > >> list, discover, and define. The goal is to materialize the > user's > > > > > >> intent ("I want to share these tables with this consumer until > > this > > > > > >> date") into standard Polaris entities (principal, principal > role, > > > > > >> catalog role, and grants). > > > > > >> > > > > > >> Therefore, I propose adding a /shares endpoint to the Polaris > > > > > >> Management API to package and materialize these share > definitions > > > into > > > > > >> Polaris entities. This would provide a usable, first-version > > package > > > > > >> for Polaris Data Sharing. > > > > > >> > > > > > >> I hope this addresses Dmitri's questions and provides a more > > > concrete > > > > > >> scenario for Polaris Data Sharing. > > > > > >> > > > > > >> Thoughts? > > > > > >> > > > > > >> Regards, > > > > > >> JB > > > > > >> > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo <[email protected]> > > wrote: > > > > > >> > > > > > > >> > Hi All, > > > > > >> > > > > > > >> > One big emerging enterprise use case coming up as more people > > > > > >> consolidate > > > > > >> > Data Lakehouses and Catalogs is something commonly known as > > "Data > > > > > >> Sharing", > > > > > >> > an more specifically over the course of adoption of open table > > > formats > > > > > >> > "Open Data Sharing". > > > > > >> > > > > > > >> > Examples of existing managed service providers' Data Sharing > > > features: > > > > > >> > > > > > > >> > https://www.databricks.com/product/delta-sharing > > > > > >> > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > > > > >> > > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > > > > >> > > > > > > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > > > >> > > > > > > >> > > > > > > > > > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > > >> > > > > > > >> > The basic idea is that when you share data between different > > > > > companies, > > > > > >> you > > > > > >> > need a first-class governance/management layer and extra > > > > > >> bells-and-whistles > > > > > >> > that are distinct from just the basic capabilities of RBAC or > > > > > >> generalized > > > > > >> > access-control (i.e. if you're sharing across > > partially-untrusted > > > org > > > > > >> > boundaries, you don't just let the consumer organization log > > into > > > your > > > > > >> > datalake like one of your own employees). > > > > > >> > > > > > > >> > JB and I put together this high-level proposal for supporting > > Open > > > > > >> Sharing > > > > > >> > in Polaris: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > > >> > > > > > > >> > Tentatively, it means adding ~5 logical data model constructs, > > > some of > > > > > >> > which may be a first-class PolarisEntity type, others subtypes > > of > > > > > >> existing > > > > > >> > entities, and others just a nested construct: > > > > > >> > > > > > > >> > - ShareEntity (would behave similarly to a Catalog) > > > > > >> > - ExternalConsumer (mostly inherits from Principal) > > > > > >> > - Listing (Similar to a "role grant" but has different > > > metadata) > > > > > >> > - EndpointConfig (nested config under Listing) > > > > > >> > - ShareMembership (Similar to a "securable grant" but > > different > > > > > >> metadata) > > > > > >> > > > > > > >> > Feedback/comments welcome! I'll also bring it up for live > > > discussion > > > > > if > > > > > >> > there's time in the community sync. > > > > > >> > > > > > > >> > Cheers, > > > > > >> > Dennis > > > > > >> > > > > > > > > > > > > > > > > >
