szehon-ho commented on a change in pull request #3069:
URL: https://github.com/apache/iceberg/pull/3069#discussion_r702477592
##########
File path: core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
##########
@@ -283,6 +283,44 @@ protected void validateAddedDataFiles(TableMetadata base,
Long startingSnapshotI
}
}
+ /**
+ * Validates that no delete files matching a filter have been added to the
table since a starting snapshot.
+ *
+ * @param base table metadata to validate
+ * @param startingSnapshotId id of the snapshot current at the start of the
operation
+ * @param conflictDetectionFilter an expression used to find new conflicting
delete files
+ * @param caseSensitive whether expression evaluation should be case
sensitive
+ */
+ protected void validateAddedDeleteFiles(TableMetadata base, Long
startingSnapshotId,
+ Expression conflictDetectionFilter,
boolean caseSensitive) {
+ // if there is no current table state, no files have been added
+ if (base.currentSnapshot() == null) {
+ return;
+ }
+
+ Pair<List<ManifestFile>, Set<Long>> history =
+ validationHistory(base, startingSnapshotId,
VALIDATE_ADDED_DELETE_FILES_OPERATIONS, ManifestContent.DELETES);
+ List<ManifestFile> deleteManifests = history.first();
+
+ DeleteFileIndex.Builder deleteIndexBuilder =
DeleteFileIndex.builderFor(ops.io(), deleteManifests)
+ .filterData(conflictDetectionFilter)
+ .caseSensitive(caseSensitive)
+ .specsById(ops.current().specsById());
+
+ // the starting snapshot could have been expired concurrently
+ if (startingSnapshotId != null && base.snapshot(startingSnapshotId) !=
null) {
+ Snapshot startingSnapshot = base.snapshot(startingSnapshotId);
+ long startingSequenceNumber = startingSnapshot.sequenceNumber();
+ deleteIndexBuilder.afterSequenceNumber(startingSequenceNumber);
+ }
+
+ DeleteFileIndex deletes = deleteIndexBuilder.build();
+
+ ValidationException.check(deletes.isEmpty(),
+ "Found new conflicting delete files that can apply to records matching
%s",
Review comment:
For debugging some complex concurrency use case, we find the list of
data files in exception message very helpful to identify where it's coming
from. Is a similar thing possible with delete files (or at least print a
sample of them)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
