kerwin-zk commented on code in PR #8214:
URL: https://github.com/apache/paimon/pull/8214#discussion_r3458047583
##########
paimon-core/src/main/java/org/apache/paimon/operation/commit/ConflictDetection.java:
##########
@@ -643,30 +643,81 @@ Optional<RuntimeException> checkRowIdExistence(
.collect(Collectors.toList());
RowRangeIndex existingIndex = RowRangeIndex.create(existingRanges,
false);
+ // Row-id ranges removed by this same commit, keyed by partition and
bucket. A
+ // copy-on-write update on a data-evolution table deletes whole row-id
groups and re-adds
+ // rewritten rows with their original row ids. File rolling may make
an added file cover
+ // only a sub-range of a deleted group and not mirror an existing file
exactly; it is still
+ // consistent as long as its range is fully covered by ranges deleted
in this commit
+ // (concurrent rewrites of those files are caught by the regular
deleted-file conflict
+ // checks).
+ Map<Pair<BinaryRow, Integer>, List<Range>> deletedRanges = new
HashMap<>();
Review Comment:
@JingsongLi Yes, this is needed by `UPDATE`.
This logic is not for supporting SQL `DELETE`. It exists for the
copy-on-write replacement that `UPDATE` performs in one commit: old physical
data files are recorded as manifest `DELETE` entries, and rewritten files are
added back with the original row ids.
The rewritten `ADD` files may be sub-ranges of a deleted file because of
file rolling, so they do not necessarily have the exact same row-id range as
any file in the current snapshot.
For example, an existing file covers row ids `[0, 100)`. The `UPDATE`
deletes that physical file and may add rewritten files `[0, 40)` and `[40,
100)`. The old `existingIndex.containsExactly(rowRange)` check would require
each `ADD` range to exactly match an existing snapshot file and would falsely
report `Row ID existence conflict`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
