Hi everyone, Thanks Robert for writing these down while the contract is still separate from storage. And thanks EJ for the design, the follow-ups on consistency and authorization, and for splitting delivery so this discussion can still change the API.
I think these four points should be resolved in https://github.com/apache/polaris/pull/5366 before that PR merges. They are contract questions, not JDBC layout questions, and they match what EJ asked the list to call out. 1) Target query encoding I agree the GET form needs a specified, reversible encoding. There are two separate problems: multipart namespace boundaries, and characters in a namespace element, table name, or column name that are special in query strings (&, ?, =, +, %). Policy's getApplicablePolicies uses the same GET shape and is equally underspecified, so Tags can document this correctly from the start. I would like the spec to define one reversible namespace-element codec, then apply ordinary URI query-value encoding to that complete output (and to target-name and column). Decoding failures should be 400. A few conformance examples, including reserved characters, would help. Client libraries should expose the codec. The structured TagAttachmentTarget on assign/unassign should stay the canonical write representation. I would not copy Iceberg's helpers blindly here. Iceberg has more than one encoding path (percent-encode each level and join with %1F for paths; join with raw 0x1F for some query params), and %1F has also been a Servlet-container problem. Nessie's printable escaping is a useful precedent for an unambiguous codec that does not put a control character on the wire. I am fine either reusing a well-specified Iceberg-compatible encoding or using a printable codec for this Polaris GET, as long as it is written down, reversible, and not double-encoded. 2) Opaque revision token I agree, and I think this is the one most worth changing now. The current spec makes version an integer that starts at 0 and increments. That matches Policy, and it will not age well for backends whose native conditional-write token is a commit id, row revision, or ETag. The observable rule is enough: the server returns a token; an update succeeds only if the client supplies the current token; otherwise 409. Matching semantics, not integer type, initial value, increment-by-one, or history. I would make the wire type a string. A JDBC implementation can still store an integer internally. Policy already shipped integers; Tags does not have to copy that constraint. 3) Reverse lookup and detach-all The REST shape is already a paginated GET /tags/{tag-name}/objects, which is the right resource. I agree with the stronger point behind it: assignments are a first-class relationship, not a collection embedded on the definition. That implies: - a forward access path by target, and a reverse path by tag/value - effective tags stay computed from the target and its ancestors; inherited assignments are not materialized onto descendants - detach-all distinguishes observable deletion from physical cleanup EJ already said a backend that cannot provide the required result must reject the operation. I would make the observable rule explicit: after a successful detach-all, no assignment of that definition is visible to API callers. Physical cleanup may complete later. If a backend cannot provide that guarantee, it should 501 / capability-gate the operation rather than return 204 with partial deletes. https://github.com/apache/polaris/pull/5391 already returns 501 until assignments exist; that is a good hook :) 4) Column identity independent of Iceberg I agree, and this already matches the intent in https://github.com/apache/polaris/issues/5442. The OpenAPI still only shows column names. I would like the contract to say: request and display use names; the durable column reference is a stable field identity. Iceberg's field id is the v1 instance of that, not the general model. Other table implementations can opt in later when they have an equally stable identity. That keeps rename-safe assignments without treating a name-based column mapping as the abstraction. None of this pulls tag-based authorization into v1. I agree with the split already on the thread: leave room for JB's permission-on-tags direction, and treat Policy as the later place for tag-aware enforcement. If the list is aligned on these, I think they belong in https://github.com/apache/polaris/pull/5366 so https://github.com/apache/polaris/pull/5391 does not freeze an integer version and the later assignment-writes PR does not freeze a name-only column mapping. Thanks, Prithvi S On Tue, Sep 8, 2026 at 4:12 PM Robert Stupp <[email protected]> wrote: > Hi, > > I have been thinking about what a full implementation would require. > > I like that the proposal separates definitions, assignments, and effective > reads. I have a few API-contract questions that seem worth resolving while > the contract is still separate from the implementation. > > > First, I think the target query parameters need a defined encoding for > identifier elements. > > This is not only about the unit separator. Namespace elements and object > names may themselves contain characters such as &, ?, =, +, or %. > Those must be preserved rather than interpreted as query syntax. > > Ordinary URI query-value encoding handles those characters, but it does not > solve the separate problem of representing the boundaries between multipart > namespace elements. > > I suggest defining a small, reversible namespace-element codec, then > applying ordinary URI encoding to its complete output. Nessie's escaped > path > representation is a useful precedent: it has an unambiguous element > separator > and escape syntax, while avoiding control characters in the transport > representation. > > The contract should specify the codec, its decoding failures, and > conformance examples. Client libraries should expose it rather than > requiring > every client to reproduce it. The structured target used by the write APIs > would still be the clearest canonical representation; this codec would make > the GET form safe and interoperable. > > > Second, I think the revision token should be opaque at the API boundary. > > The backend should be free to use a native row revision, commit ID, ETag, > or > another conditional-write token. However, the contract should define the > observable precondition: the server returns a token, and an update succeeds > only if the client supplies the token for the current tag definition. > Otherwise the server returns a conflict. > > That requires token matching semantics, but not an integer type, an initial > value, ordering, increment-by-one behavior, or history semantics. A > catalog-wide commit token would also be valid, although it could create > avoidable conflicts for unrelated changes. > > > Third, the direct reverse lookup is useful, but I would treat it as a > first-class, paginated relationship rather than a tag record containing a > collection of targets. > > A common tag can legitimately be attached to a very large number of objects > or columns. A backend will normally need one forward access path for direct > assignments by target, and one reverse access path by tag/value, with > backend-specific partitioning or sharding. Effective assignments should > remain > computed from the target and its ancestors; materializing inherited > assignments onto descendants would have very different scaling behavior. > > This also affects detach-all. Deleting an unbounded number of assignment > records atomically is not a portable primitive for all backends. The > contract > should distinguish observable deletion semantics from physical cleanup, or > state the backend capability required for a synchronous detach-all > operation. > > > Finally, I agree with the V1 boundary of top-level Iceberg columns, but I > would > keep the core tag model independent of Iceberg. For Iceberg, the durable > column reference should be the field ID, with a column name used only for > request-time resolution and display. Other table implementations could opt > in > later once they provide an equally stable field identity. This avoids > treating > a name-based column mapping as a general abstraction. > > > None of this requires tags to become authorization inputs in V1. It is > mainly > about leaving the assignment and read contract implementable by more than > one > persistence model when those slices arrive. > > Thanks, > Robert > > > On Fri, Aug 28, 2026 at 2:19 AM EJ Wang <[email protected]> > wrote: > > > Hi folks, > > > > A quick update on the Tag work. *Current status*: > > - PR1: API contract (https://github.com/apache/polaris/pull/5366): ready > > for review > > - *NEW! *PR2: Definition CRUD ( > https://github.com/apache/polaris/pull/5391 > > ): > > open as Draft, ready for review if PR1 LGTY > > - PR3: Assignment writes and storage: planned > > - PR4: Reads, inheritance, and reverse lookup: planned > > > > *More on PR2:* > > - PR2 makes Tag definitions usable through create, list, load, update, > > rename, and delete. It intentionally stops before assignments, so their > > persistence model remains open for the next slice. > > - The PR is stacked on #5366 and will be rebased once that PR merges. > > > > *Asks:* > > - For #5366, please call out any remaining API contract concerns. For > > #5391, I would especially appreciate feedback on the slice boundary and > the > > decision to reuse the existing entity persistence model. > > - The design doc remains here: > > > > > https://docs.google.com/document/d/1rIJGzcsmGhfrBiRXPac51hr-jeJuuKQQBYjgBdOb9-k/edit?usp=sharing > > > > I’ll keep using this thread for new delivery slices, material status > > changes, and specific community asks. > > > > Thanks, > > -ej > > > > On Mon, Aug 24, 2026 at 5:04 PM EJ Wang <[email protected]> > > wrote: > > > > > Hi folks, > > > > > > Following up on this thread, I have opened a PR to land the public API > > > contract for Tags: https://github.com/apache/polaris/pull/5366 > > > > > > The PR defines Tag management, assignment and unassignment, direct and > > > inherited reads, and reverse lookup. V1 covers catalogs, namespaces, > > > Iceberg and generic tables as whole objects, and top-level Iceberg > table > > > columns. Views, generic-table columns, nested fields, multi-value > > > assignments, and tag-based authorization are deferred. > > > > > > I plan to deliver the capability through four PRs that merge in order: > > the > > > API contract in this PR, Tag definition CRUD, assignment writes and > > > storage, then reads and reverse lookup. A separate follow-up will add > > > grants on Tag resources to the management APIs. That grant surface is > > > distinct from using Tags to control access to tagged objects, which > > remains > > > outside v1. > > > > > > The updated design doc is here: > > > > > > https://docs.google.com/document/d/1rIJGzcsmGhfrBiRXPac51hr-jeJuuKQQBYjgBdOb9-k/edit?usp=sharing > > > > > > The PR is currently Draft while we finish aligning on the public > > contract. > > > It is intended to merge as the first delivery slice, not remain as a > > > design-only artifact. Please call out any remaining scope or contract > > > concerns. If the list is aligned, I will mark it ready for review. > > > > > > Thanks, > > > -ej > > > > > > On Wed, Aug 12, 2026 at 2:05 PM EJ Wang < > [email protected]> > > > wrote: > > > > > >> Thanks Dmitri, these comments were very useful. > > >> > > >> I went through the three areas you called out and updated the proposal > > >> accordingly. > > >> > > >> On the permission/policy direction, *I agree the Tag model should > leave > > >> room for permissions or policies to consume tags later*, including the > > >> direction JB proposed. I am keeping that outside the v1 Tag contract, > > >> though. In v1, tags classify resources; they do not themselves grant > or > > >> deny access. Polaris Policy looks like the closest existing foundation > > if > > >> we later want a portable tag-aware policy model, but I think that > > deserves > > >> a separate proposal rather than baking policy semantics into the Tag > > >> storage model now. > > >> > > >> I also made the authorizer path more explicit. *A future OPA, Ranger, > or > > >> other authorizer could receive the target's complete effective tags as > > >> resource attributes*. The authorization path would resolve those tags > > >> internally, applying target-types, inheritance, closest-wins, > > grandfathered > > >> values, and the same coherent-read guarantees as the Tag API. At > > minimum, > > >> the portable input can include the tag definition ID, current name, > and > > >> selected value; provenance can be additional context. If Polaris > cannot > > >> resolve the complete effective state, authorization should fail closed > > >> rather than treat the resource as untagged. > > >> > > >> That also makes the persistence expectation on the read path clearer: > an > > >> implementation needs to resolve the target and relevant ancestors, > > obtain > > >> the applicable tag definitions and assignments, and produce one > coherent > > >> effective result. *Those observable semantics are the backend > contract; > > >> the physical lookup/indexing strategy is not.* > > >> > > >> On the Java interface suggestion, I added Java-shaped records for the > > >> durable logical model so the definition, target identity, and > assignment > > >> shapes are easier to review from JDBC and NoSQL perspectives. I > stopped > > >> short of proposing operation interfaces in pseudo-code, though. My > > current > > >> thinking is that we should first agree on the durable facts and > required > > >> behavior, then design the actual persistence SPI around the needs of > the > > >> implementations. I did not want an illustrative interface in this > > design to > > >> accidentally become the persistence contract. > > >> > > >> So Part 2 now separates the two intentionally: > > >> > > >> *logical data + behavior/conformance requirements are specified; > > >> transaction, CAS, atomic batch, provider-native operations, and the > > >> eventual Java SPI remain implementation/design choices.* > > >> > > >> Thanks again for the review, and definitely keep the comments coming > :) > > >> > > >> I've updated the doc, please check it out the latest and the greatest: > > >> > > >> > > > https://docs.google.com/document/d/1rIJGzcsmGhfrBiRXPac51hr-jeJuuKQQBYjgBdOb9-k/edit?pli=1&tab=t.0 > > >> > > >> -ej > > >> > > >> On Fri, Aug 7, 2026 at 3:39 PM Dmitri Bourlatchkov <[email protected]> > > >> wrote: > > >> > > >>> Hi EJ, JB, > > >>> > > >>> I left some comments on EJ's doc. I actually have a lot of comments > on > > >>> the > > >>> REST API design, I only posted some of them to start a discussion > > >>> without overloading the doc. > > >>> > > >>> Overall, I believe EJ's proposal should also allow permission > > assignments > > >>> on tags that JB proposed (eventually). We just need to clearly define > > the > > >>> persistence expectations for looking up related tags on the read > path. > > >>> > > >>> We should probably specify whether and how tags are exposed to > > >>> authorizers > > >>> (OPA, Ranger). I imagine people will want to use them in external > > policy > > >>> engines the moment the feature is available. > > >>> > > >>> On the persistence side, I believe it would be nice to define actual > > java > > >>> interfaces (perhaps in pseudo code) to allow easier review from the > > NoSQL > > >>> persistence perspective (also commented in the doc). > > >>> > > >>> Cheers, > > >>> Dmitri. > > >>> > > >>> On Thu, Jul 30, 2026 at 12:39 AM Jean-Baptiste Onofré < > [email protected] > > > > > >>> wrote: > > >>> > > >>> > Hi EJ > > >>> > > > >>> > Thanks for starting this discussion. > > >>> > > > >>> > For the record, here's my initial proposal about tagging: > > >>> > https://lists.apache.org/thread/nmqmmjfmocfllb71fcmyp9syc9gyn820 > > >>> > > > >>> > At that time, only Dmitri replied :) > > >>> > So, I would be happy to work with you on this, as I still have the > > PoC > > >>> > I created for my initial proposal. > > >>> > > > >>> > I will try to join the scheduled meeting (no guarantee). > > >>> > > > >>> > Regards > > >>> > JB > > >>> > > > >>> > On Fri, Jul 17, 2026 at 6:53 AM EJ Wang < > > >>> [email protected]> > > >>> > wrote: > > >>> > > > > >>> > > Hi folks, > > >>> > > > > >>> > > I have prepared a Google Doc > > >>> > > < > > >>> > > > >>> > > > https://docs.google.com/document/d/1rIJGzcsmGhfrBiRXPac51hr-jeJuuKQQBYjgBdOb9-k/edit?usp=sharing > > >>> > > > > >>> > > for the Polaris tag spec proposal. > > >>> > > > > >>> > > The goal is simple: add a native tag model to Polaris so users > can > > >>> > classify > > >>> > > catalog objects, read those classifications back, and find > objects > > by > > >>> > tag. > > >>> > > > > >>> > > The proposal covers: > > >>> > > * tag definitions as catalog-scoped Polaris entities > > >>> > > * tag assignments on catalogs, namespaces, table-like objects, > and > > >>> > columns > > >>> > > * allowed values on tag definitions > > >>> > > * direct and inherited tag reads > > >>> > > * direct by-tag lookup > > >>> > > * the durable model behind the API > > >>> > > * how this compares with the existing Polaris Policy API (tag > > design > > >>> > > referenced policy heavily, given their pattern similarity) > > >>> > > > > >>> > > Please take a look and leave comments in the doc. Let me know > WDYT! > > >>> > > > > >>> > > I would also like to discuss this in the July 23 community sync. > A > > >>> > separate > > >>> > > dedicated review meeting will be scheduled separately, likely > > within > > >>> the > > >>> > > next two weeks. > > >>> > > > > >>> > > Thanks, > > >>> > > -ej > > >>> > > > >>> > > >> > > >
