karan-canva commented on code in PR #10575:
URL: https://github.com/apache/paimon/pull/10575#discussion_r4289018801
##########
paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java:
##########
@@ -1108,39 +1110,33 @@ private void createMetadataWithBase(
addedFiles.values().stream().mapToLong(p ->
p.getRight().rowCount()).sum();
metrics.addedFilesSize =
addedFiles.values().stream().mapToLong(p ->
p.getRight().fileSize()).sum();
- metrics.deletedDataFiles = removedFiles.size();
+ metrics.deletedDataFiles = deletedFiles.size();
metrics.deletedRecords =
- removedFiles.values().stream().mapToLong(p ->
p.getRight().rowCount()).sum();
+
deletedFiles.stream().mapToLong(IcebergDataFileMeta::recordCount).sum();
metrics.deletedFilesSize =
- removedFiles.values().stream().mapToLong(p ->
p.getRight().fileSize()).sum();
- metrics.changedPartitionCount = modifiedPartitionsSet.size();
-
- IcebergSnapshot baseSnapshot = baseMetadata.currentSnapshot();
+
deletedFiles.stream().mapToLong(IcebergDataFileMeta::fileSizeInBytes).sum();
+ Set<BinaryRow> changedPartitions = new HashSet<>();
Review Comment:
Thanks, confirmed. A commit that only changes deletion vectors reported
`changed-partition-count=0`. Fixed in 8991d5bab.
When deletion vectors are exported, the count now also includes the
partitions whose deletion vector index files were added or removed. It doesn't
use the ADDED status in the new DELETES manifest, because that manifest is
rebuilt with every entry ADDED whenever a deletion vector changes, and a retry
passes every index file of the snapshot as ADD. Instead, the index files are
compared with the live entries of the base DELETES manifests, so a partition
counts only when its delete files actually changed.
Reading the base DELETES manifests showed that
`IcebergDataFileMetaSerializer#fromRow` failed on delete file entries, which
are written with null `null_value_counts`, `lower_bounds` and `upper_bounds`.
`fromRow` now handles these null fields, and
`IcebergDataFileMetaTest#testDeleteFileMetaRoundTrip` covers it.
`IcebergDVCompatibilityTest#testChangedPartitionCountWithDeletionVectorOnlyChange`
is the v3 + bitmap64 regression case. It fully compacts two partitions, then
deletes one existing key in each and lets minor compaction write deletion
vectors, which must report 2 changed partitions. Then it does the same in one
partition only, which must report 1, and checks the rows through the Iceberg
reader. Without the fix the first of these commits reports 0. The helper in
`IcebergCompatibilityTest` still skips DELETES manifests, since those tests
don't export deletion vectors. The v3 case is covered in `paimon-iceberg`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
