xiaoxuandev commented on code in PR #17764:
URL: https://github.com/apache/iceberg/pull/17764#discussion_r3876514936
##########
core/src/main/java/org/apache/iceberg/DeleteFileIndex.java:
##########
@@ -485,6 +486,24 @@ Builder ignoreResiduals() {
return this;
}
+ /**
+ * Indexes deletion vectors as file-scoped position deletes, allowing a
data file to have more
+ * than one DV.
+ *
+ * <p>A data file has at most one live DV in a single snapshot, so by
default {@link #build()}
+ * rejects a second DV for the same data file. That invariant does not
hold across snapshots,
+ * and commit validation indexes delete manifests from every snapshot in
the validation window,
+ * so it requires an index that accepts them.
+ *
+ * <p>The resulting index only answers whether deletes exist for a data
file, not which deletes
+ * apply to it: {@code forDataFile} no longer suppresses partition-scoped
position deletes when
+ * a DV is present. Scans must use the default so that corrupt live
metadata is still detected.
+ */
+ Builder indexDVsAsPositionDeletes() {
Review Comment:
Thanks @amogh-jahagirdar for the review, agreed, the per-snapshot
`DeleteFileIndex` is the better abstraction. This version works but shouldn't
put a validation-only knob on `DeleteFileIndex` that changes `forDataFile`'s
semantics.
Streaming upserts is the case we're targeting here. An optimized version
might end up looking like "single index tolerating multiple DVs," but doing it
at the right layer keeps forDataFile intact and still catches within-snapshot
corruption. Will update this PR and do the optimization in a separate PR later.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
