Hi all, I'd like to gauge appetite for making concurrent UPDATE/MERGE on V3 deletion vectors fail less often when the operations touched disjoint rows.
Background: with one DV per data file, validateAddedDVs fails the commit when a concurrent op added a DV for a file this op also DV'd, at file granularity, regardless of whether the deleted positions actually overlap. That ValidationException is not retryable (the commit loop is onlyRetryOn CommitFailedException), so the whole operation is recomputed and re-run. For two operations that modified disjoint rows in the same file, that is wasted work on a spurious conflict. Delta Lake's row-level concurrency already reduces this to row granularity on DV-enabled tables; I would like to pursue the same for Iceberg V3 DVs. Core idea: refine validateAddedDVs from file granularity to position granularity. When a concurrent DV exists for a file this op also DV'd, compare the newly-deleted positions: (dv_concurrent intersect dv_self) minus dv_base == empty If disjoint, union the two into a single DV and proceed; if they overlap, fail as today. This reuses existing DV primitives (DVUtil.readDV, PositionDeleteIndex.merge, BaseDVFileWriter); the new code is a bitmap-overlap test plus wiring a cross-operation union into the validation path. It does not change isolation semantics, which stay in validateNoConflictingDeletes and validateNoConflictingData. I wrote up the full design (soundness conditions, a sequenced set of follow-up refinements, non-goals including compaction-vs-DML which is not reconciled today, and configuration) as a GitHub issue: https://github.com/apache/iceberg/issues/18020 Questions: 1. Is the file-level false-failure in validateAddedDVs worth refining to position granularity? 2. Should the disjoint-case union happen in the validation path (union in place), or via a retryable commit that re-reads and unions the base DV on each refreshed apply? (Details in the issue.) 3. Any prior art or earlier design decision I have missed? Thanks, EJ
