I actually think we already have precedent for how to handle situations like this in the spec (TLDR; (1 - Disallow Ambiguous States)) from our earlier discussions with default values.
For those who remember <https://github.com/apache/iceberg/pull/12841>, we spent a lot of time discussing how parent and child defaults should interact when both could supply a value for the same leaf. We discussed all the various options almost *identically *to what we are discussing now. After much back and forth we just decided to disallow the ambiguous configuration. Struct defaults are only null or {}, and field defaults live on the fields themselves. Simple, with no extra rules that a client can mess up. I think we are in basically the same situation here. Overlapping projections on a nested container and one of its descendants don't have an obvious resolution. Similar to the default case, we could try to establish client rules that every client must implement and could potentially implement incorrectly, or we could simply declare that state is invalid. So I am +1 on Option A / Nevin’s contract (1): catalogs MUST NOT emit overlapping nested projections, and readers MUST fail closed if they receive them. Similar to a table with multiple overlapping initial-defaults, the reader should simply throw an error and state that the input is not valid. On Fri, Aug 7, 2026 at 3:10 PM Sung Yun <[email protected]> wrote: > > While changing this requirement in the future is structurally > compatible, it is not behaviorally compatible. > > Dan, that is a fair point, and I withdraw my claim that relaxing the > constraint later would be entirely backward-compatible. It would not be, > given the data access regression you described. > > There is a real behavioral consequence whenever a new capability is > introduced to ReadRestrictions, and I think the community would benefit > from working out what the cost of introducing new capabilities is, beyond > the immediate scope of nested policy handling. I would rather > compartmentalize this discussion to its own thread than litigate the > question on nested policy conflict resolution based on an uncalibrated > weighting of that cost. I will raise it at the 8/18 sync. > > Setting that aside, Nevin is right that this is partly a question of which > tradeoffs we value. I think Option A holds up well on interoperability with > existing policy engines. I put together a survey of how current systems > handle nested policies to help with that analysis, and I am happy to walk > through it together during the 8/18 sync [1]. > > Here is a short summary. Every surveyed system resolves to a single policy > per field value, and no system composes more than one masking transforms > over one value. Most make an ancestor-descendant overlap un-authorable, > either by excluding the container from masking or by permitting exactly one > attachment point per column. Redshift is the only system that permits both > to be attached, and it requires administrator-authored priorities and > resolves to a single winner before the reader is involved. > > Two additional properties seem worth weighing in the discussion. First, > whether moving conflict resolution from the policy engine to the client is > valuable to the community, and whether a defined semantic in Iceberg can do > that effectively. Second, whether a composition rule, where transforms > apply on top of each other, is a capability we want to introduce without > industry precedent. > > As Nevin noted, we have not dug into what those semantics could be. I > hoped laying them out would help move the discussion forward. > > Sung > > [1] https://s.apache.org/0w1mu > > On 2026/08/07 00:00:55 Nevin Zheng wrote: > > Hi all, > > > > Thanks everyone for the thoughtful discussion. I wanted to offer a > > slightly different framing that may help us align on the decision > > we're making. > > > > I see two dimensions: > > > > - Catalog: MUST NOT send overlaps vs. MAY send overlaps > > - Reader: fail closed vs. apply defined semantics > > > > That gives us four possible contracts: > > > > (1) MUST NOT + fail closed — strict contract > > (2) MUST NOT + defined semantics — defensive fallback > > (3) MAY + fail closed — partial interoperability > > (4) MAY + defined semantics — supported semantics > > > > My read is that the current proposal is (1). I would place Dan's > > position as a version of (4), with the concrete semantics still to be > > defined. > > > > At this level, I think this is partly a values/tradeoff decision > > rather than a question with one technically correct answer. We can > > value simplicity and fail-closed behavior, or place more value on > > expressiveness and interoperability. Until we align on those > > priorities, or find a compromise, I think we may continue disagreeing > > on the lower-level mechanics. > > > > There are of course finer questions around precedence/composition, > > security, compatibility, implementation complexity, and extensibility. > > I think choosing the contract at this level first could help clarify > > those discussions. > > > > Another useful lens may be the critical user journey: when overlapping > > projections occur, who bears the cost? With (1), the query fails > > safely and the operator/end user has to resolve the configuration. > > With (4), the query can continue, but Iceberg readers take on the > > complexity of correctly interpreting the overlap. > > > > My preference is (1). Catalogs should produce unambiguous > > restrictions, and readers should fail closed with a clear error when > > they don't. Catalogs will inevitably have bugs or misconfigurations; I > > prefer a loud, safe, diagnosable failure over requiring every reader > > to interpret ambiguous policy. > > > > This also gives us a safe failure mode if we expand the semantics > > later: older implementations may fail, but they fail closed. > > > > That said, I'm comfortable with another coherent contract if the > > community prefers its tradeoffs and we specify and implement it well. > > > > I hope this framing is useful. Please let me know if I've > > misrepresented the design space or anyone's position. > > > > Nevin > > > > On Thu, 6 Aug 2026 16:17:39 -0700, Steven Wu <[email protected]> > wrote: > > > > This is true regardless. The client is not involved in resolving > anything. > > > > > > I am only discussing main question of this thread: failing the query > vs resolving precedence rules at client/reader-side for nested fields with > conflicting policies? > > > > > > If a projection targets a nested-typed field (struct, list, or map), > other projections in the same ReadRestrictions must not target any nested > field-id (struct subfields, list elements, or > > > > > > map keys/values) at any depth. This specification does not define how > such actions combine. > > > > > > A reader that receives such a response must fail the query. > > > > > > > Relaxing the requirement is not backward compatible for clients. > > > > > > If we relax this in the future, here is how we can do so compatibly. > > > > > > A reader that receives such a response must fail the query. > > > > > > --> > > > > > > A reader that receives such a response may fail the query or may apply > the following precedence rule to deconflict. > > > > > > > we're taking a shortcut by not implementing full schema support for > this feature > > > > > > What does the "full schema support" mean in this context of > conflicting policies for nested fields? > > > > > > Thanks, > > > Steven > > > > > > On Thu, Aug 6, 2026 at 2:43 PM Daniel Weeks <[email protected]> wrote: > > > > > > Current spec requires clients to fail if the catalog server returns > conflicting policies. If we relax this requirement in the future with > precedence rules for client-side de-conflicting, older clients would still > fail. I don't see it as "an unpredictable behavior change" for older > clients. I would also the relaxation would probably use the wording of MAY > (not MUST) applying these precedence rules to deconflict. > > > > > > Relaxing the requirement is not backward compatible for clients. That > would be equivalent to saying that we can make a required field optional > because the clients need to validate required fields so it's ok if they > break. Backward compatibility is about not breaking clients, not making it > ok to break clients. > > > > > > I heard comments that this rule is very easy for the policy engine to > enforce . . . > > > > > > Handling nested fields doesn't change how hard it is for a policy > engine to enforce. They can always fallback to not allowing it (they would > already be required to detect these cases). > > > > > > If we ask clients to resolve conflicting policies > > > > > > This makes it sound like we're shifting the burden to the client, but > that's not the case. We are just defining an order in which they are > applied. Policy engines are required to resolve conflicts. The client just > applies the restrictions in a well defined order. > > > > > > I would let the policy engine either enforce that > conflicting/ambiguous policies are not accepted or perform the > deconflicting/precedence rules before returning to the client. > > > > > > This is true regardless. The client is not involved in resolving > anything. > > > > > > The issue here is that we're taking a shortcut by not implementing > full schema support for this feature and presenting it as a reasonable path > based on a misconception that it's backward compatible. > > > > > > -Dan > > > > > > On Thu, Aug 6, 2026 at 11:44 AM Steven Wu <[email protected]> > wrote: > > > > > > > Changing that in the future would break older clients that validate > and result in an unpredictable changes in behior of clients that assume > correctness from the catalog. > > > > > > Current spec requires clients to fail if the catalog server returns > conflicting policies. If we relax this requirement in the future with > precedence rules for client-side de-conflicting, older clients would still > fail. I don't see it as "an unpredictable behavior change" for older > clients. I would also the relaxation would probably use the wording of MAY > (not MUST) applying these precedence rules to deconflict. > > > > > > I agree with Prashanth and Sung that it is hard for Iceberg to force a > specific behavior considering the divergent behaviors in the industry. > > > > > > As I mentioned during the sync, I also see it the same as the other > requirement we put on the catalog server. I heard comments that this rule > is very easy for the policy engine to enforce. If we ask clients to resolve > conflicting policies for nested fields with precedence rules, I can argue > the same thing here for this rule. Can't we also ask clients to define > resolution precedence with the projection policies? > > > > > > 3. A server must not return more than one projection for the same > field-id > > > in required-column-projections. If a duplicate field-id appears, the > reader > > > must fail the query. > > > > > > Since IRC doesn't define policy management APIs and behaviors, I would > let the policy engine either enforce that conflicting/ambiguous policies > are not accepted or perform the deconflicting/precedence rules before > returning to the client. Hence, i agree with the current spec writing that > clients must fail in the case of conflicting policies with nested fields. > > > > > > On Thu, Aug 6, 2026 at 9:40 AM Daniel Weeks <[email protected]> wrote: > > > > > > I'd like to clarify the point on backward compatibility claims here > first as it indicates a dangerous misunderstanding of compatibility in the > REST protocol. The statement that "Forbidding is reversible; precedence > isn't. Relaxing a MUST NOT later is backward-compatible" is incorrect, and > we need to be very careful regarding backward compatibility. It also > creates a false sense of security by deferring this, as it implies the > issue can be addressed later without consequence. > > > > > > While changing this requirement in the future is structurally > compatible, it is not behaviorally compatible. Given the current spec > wording, a responsible client should validate that they do not receive a > invalid set of redactions from the catalog as this is prohibited by the > spec and there's no guidance on how the client should apply them. Changing > that in the future would break older clients that validate and result in an > unpredictable changes in behior of clients that assume correctness from the > catalog. This is neither safe nor backward compatible. > > > > > > I think addressing this now is well within scope and will prevent > unnecessary future protocol evolution. Regardless of what type of > precedence we define, we already know that some policies (e.g. Redshift) > that will express more than we will likely include in the protocol. It's > also likely that different policy systems will have conflicting views on > this matter. Since we're already taking an opinionated stance and levying > requirements on the catalog implementation to align their policy with the > protocol, this isn't any more complicated than prohibiting conflicting > policy. > > > > > > Read restrictions have a number of unexpected cliffs regarding how > policy changes affect what clients can consume (e.g. ABAC policies that > update from simple redaction to more contextual redaction) and this is just > one more scenario that limits interoperability. > > > > > > The backward compatibility issue further highlights why we need a > solid reference implementation as I think it would make the problem clear. > I don't want to repeat the challenges we had with scan/plan where we > released the protocol with issues that we, fortuntately, were able to > address as the reference implementation was built. > > > > > > -Dan > > > > > > On Thu, Aug 6, 2026 at 7:56 AM Sung Yun <[email protected]> wrote: > > > > > > Thanks Prashant for doing such extensive research on this topic. > > > > > > I believe the industry divergence on how nested type policies are > handled is a good indicator that we should hold off on forcing a standard > for nested conflict resolution right now. Defining a standard is a highly > sensitive design decision that deserves its own focused, long-term > community discussion rather than being rushed. > > > > > > During our syncs, we’ve discussed several potential evaluation models, > each with distinct trade-offs: > > > > > > - Outermost wins (single policy is enforced) > > > > > > - Innermost wins (single policy is enforced) > > > > > > - Innermost evaluated first (pipelined and layered sequentially) > > > > > > - Outermost evaluated first (pipelined and layered sequentially) > > > > > > These models carry different security implications, and hence I'm in > favor of choosing option A (forbidding overlapping nested policies and > failing-closed on the client) to not rush this discussion and unblock > ReadRestrictions today. > > > > > > If the community later decides to define a precedence standard, > relaxing this rule is entirely backward-compatible. On the client/reader > side, we would simply be introducing a new handling capability for what > used to be an invalid response. On the server/catalog side, the timeline > remains entirely in the catalog administrator's control because the new > behavior only triggers once they choose to update their catalog > implementations to emit multiple nested policies. > > > > > > One final observation on the industry examples: > > > > > > Even in Redshift, which is the only example where multiple policies > can technically be registered on a nested path, the engine treats > overlapping policies as a conflict. It resolves this conflict by enforcing > only the single, highest-priority policy at query time. In practice, this > "highest-priority wins" model is highly consistent with Option A. It > already assumes that the system must ultimately resolve the conflict down > to a single, clean policy rather than attempting to chain and layer > multiple policies sequentially. I believe that this can be the > responsibility of the policy engine, rather than becoming a preemptive > standard in the ReadRestrictions spec. > > > > > > All in all, I strongly support keeping the current restrictive > instruction in the spec (Option A) for the initial release and updating the > words with normative RFC 2119 terminology (using MUST NOT and MUST fail) to > clearly define the behavior on the client and server side. [1] > > > > > > Sung > > > > > > [1] > https://github.com/apache/iceberg/pull/13879#discussion_r3716145795 > > > > > > On 2026/08/04 23:25:40 Prashant Singh wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > One open thread on the Read Restrictions spec PR [1] is the last > item I'd > > > > > > > like to settle before calling a vote [2]. The last sync leaned > towards the > > > > > > > restrictive option, but there was an ask by Dan that the alternative > be > > > > > > > explored properly first, so I'm bringing it here. > > > > > > > > > > > > > > The question > > > > > > > > > > > > > > required-column-projections binds an action to a field-id, and a > nested > > > > > > > type has a field-id for the container and separate ones beneath it > (struct > > > > > > > subfields; list element; map key and value). > > > > > > > > > > > > > > Concretely, with 2: address struct<3: street string, 4: city string>: > > > > > > > > > > > > > > - field-id: 2 action: mask-to-fixed-value > > > > > > > > > > > > > > - field-id: 4 action: replace-with-null > > > > > > > > > > > > > > mask-to-fixed-value overwrites everything beneath the container, so > it > > > > > > > yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore > > > > > > > returns {"street": > > > > > > > "XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns {"street": > null, > > > > > > > "city": "XXXXXXXX"}, though the catalog asked for the whole struct > to be > > > > > > > masked. > > > > > > > > > > > > > > if a catalog returns two projections (one on the outer container and > one on > > > > > > > a field-id below it), how should clients handle it? > > > > > > > > > > > > > > (A) forbid the overlap; the existing fail-closed rule then makes a > reader > > > > > > > > > > > > > > that receives one fail the query. The spec PR currently says: > > > > > > > > > > > > > > If a projection targets a nested-typed field (struct, list, or map), > other > > > > > > > projections in the same ReadRestrictions must not target any nested > > > > > > > field-id (struct subfields, list elements, or > > > > > > > > > > > > > > map keys/values) at any depth. This specification does not define > how such > > > > > > > actions combine. > > > > > > > > > > > > > > A reader that receives such a response must fail the query. > > > > > > > > > > > > > > (B) define precedence (e.g. outer-most wins) and allow it > > > > > > > > > > > > > > Why current spec PR choose option A > > > > > > > > > > > > > > No semantics to borrow. There are different behaviors in the current > > > > > > > industry practice > > > > > > > > > > > > > > - > > > > > > > > > > > > > > BigQuery: > > > > > > > - > > > > > > > > > > > > > > "Policy tags can't be applied to columns that use the STRUCT data > > > > > > > type" [3]; > > > > > > > - > > > > > > > > > > > > > > Redshift both experience based on data-types: > > > > > > > - > > > > > > > > > > > > > > "You can only apply masking policies to scalar values on the SUPER > > > > > > > path" [4] rejects the overlap. > > > > > > > - > > > > > > > > > > > > > > Redshift is the only one that lets both be expressed, and it calls > > > > > > > the pair a conflict, resolved by an admin-supplied priority [5] > > > > > > > - > > > > > > > > > > > > > > Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, Impala > and > > > > > > > Ranger define no parent-vs-nested resolution at all, and where the > question > > > > > > > has been raised it has stayed open: RANGER-3525 since 2021 [6], > HIVE-22823 > > > > > > > [7], trino#16964 [8]. > > > > > > > - > > > > > > > > > > > > > > SQL:2016 defines nothing - <privilege column list> is a flat column > name > > > > > > > list. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Forbidding is reversible; precedence isn't. Relaxing a MUST NOT > later is > > > > > > > backward-compatible: invalid responses become valid, and no reader > changes > > > > > > > what it returns for a response that was already legal. If we define > > > > > > > outer-most-wins now and later want the inner action to contribute, > we've > > > > > > > silently changed the values a compliant reader returns for the same > > > > > > > response. > > > > > > > > > > > > > > On the future-actions concern: I don't think the prohibition limits > any > > > > > > > action. A future action can target a container, or a nested field - > just > > > > > > > not both in one response, which is the only combination with no > defined > > > > > > > meaning today. > > > > > > > > > > > > > > Asks > > > > > > > > > > > > > > 1. Opinions on option A (forbid) vs B (define precedence) > > > > > > > > > > > > > > 2. If (B), is outer-most-wins the rule — and are we comfortable that > > > > > > > > > > > > > > nothing in the current action set can show it’s the right choice? > > > > > > > > > > > > > > I am looking forward to hear what other people think and if they have > > > > > > > objection to the current proposed spec > > > > > > > > > > > > > > We time-boxed this to the next sync (Aug 18) in the last one. If > there's no > > > > > > > further input by then, we will keep the current restrictive wording > and say > > > > > > > so in this thread rather than treat silence as agreement - the > precedence > > > > > > > question should be closed explicitly, not by default. And if a use > case > > > > > > > turns up after we ship, Option A lets us relax the rule > compatibility, > > > > > > > which is most of why the PR takes it. Notes from sync [10]. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Prashant > > > > > > > > > > > > > > References > > > > > > > > > > > > > > [1] Spec PR: https://github.com/apache/iceberg/pull/13879 > > > > > > > > > > > > > > [2] Review thread: > > > > > > > https://github.com/apache/iceberg/pull/13879#discussion_r3211673048 > > > > > > > > > > > > > > [3] BigQuery, column data masking: > > > > > > > https://cloud.google.com/bigquery/docs/column-data-masking-intro > > > > > > > > > > > > > > [4] Redshift, DDM with SUPER paths: > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html > > > > > > > > > > > > > > [5] Redshift, DDM policy hierarchy: > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html > > > > > > > > > > > > > > [6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525 > > > > > > > > > > > > > > [7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823 > > > > > > > > > > > > > > [8] trinodb/trino#16964: > https://github.com/trinodb/trino/issues/16964 > > > > > > > > > > > > > > [9] IMPALA-15130: https://issues.apache.org/jira/browse/IMPALA-15130 > > > > > > > > > > > > > > [10] Note from ReadRestrictions syncs: > > > > > > > > https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx > > > > > > > > > >
