karan-canva opened a new pull request, #10575: URL: https://github.com/apache/paimon/pull/10575
### Purpose Linked issue: close #10574 The snapshot summary that `IcebergCommitCallback` writes can undercount live files. `total-data-files`, `total-records` and `total-files-size` drift down, and can reach 0 while the manifests still have live files. `deleted-data-files`, `deleted-records` and `removed-files-size` can count more files than Iceberg removed. Engines that read these fields, such as Snowflake during refresh, can then decline to refresh the table. There are two causes: 1. `collectFileChanges` filters ADD entries with `shouldAddFileToIceberg`, but not DELETE entries. Removed Paimon files that were never exported, such as low-level files of a primary-key table, were counted as deletions and subtracted from the running totals. Each compaction lowered the totals further. 2. `computeLiveFileCount` and `computeLiveRowCount` computed `added + existing - deleted` for each manifest. These counts are disjoint, so the live count is `added + existing`. The helpers undercounted whenever a manifest had DELETED entries. They were used when the previous snapshot had no totals, and for the delete-file totals. The fix: - The deleted counters come from the entries that `createWithDeleteManifestFileMetas` marks as DELETED, so they match what Iceberg removed. The DELETE entries are not filtered with `shouldAddFileToIceberg`. The first snapshot is built from raw-convertible splits and can export a level-0 file, and filtering would stop such a file from ever being marked DELETED. - `total-data-files` and `total-records` are computed from the new data manifest list, the same way `total-delete-files` already is. They now always match the manifests, and tables that already have wrong totals are corrected on their next commit. - File sizes are not in the manifest list, so `total-files-size` is still carried forward. When the previous `total-data-files` does not match the base manifests, the previous size total is not trusted and is recomputed from the manifests. - `IcebergManifestFileMeta#liveFilesCount` (`added + existing`) is added next to `liveRowsCount`, and both helpers use them. ### Tests - `IcebergCompatibilityTest#testSnapshotSummaryTotalsMatchManifests`, with and without deletion vectors: a primary-key table goes through three rounds of uncompacted writes followed by a full compaction with a delete. After every commit, `total-data-files`, `total-records`, `total-files-size`, `total-delete-files` and `total-position-deletes` must equal the live entries in the snapshot's manifests. `deleted-data-files`, `deleted-records` and `removed-files-size` must equal the entries the snapshot marked as DELETED. On master the first compaction writes `deleted-data-files=4` while Iceberg removed 1 file, and `total-data-files=0` while 1 file is live. - `IcebergCompatibilityTest#testSnapshotSummaryTotalsRepairedWhenBaseTotalsAreUnusable`: the latest metadata has its totals either removed, as in metadata written before the counters existed, or set to 0, as in metadata written with the miscounted totals. The next append and compaction must write totals that match the manifests. On master both cases write `total-data-files=0`. - `IcebergManifestFileMetaTest#testLiveCountsExcludeDeletedEntries`: a manifest with ADDED, EXISTING and DELETED entries reports live counts of `added + existing`. - `mvn -pl paimon-core -Dtest='Iceberg*Test' test`: 192 tests pass. `mvn -Ppaimon-iceberg -pl paimon-iceberg test`: 65 tests pass, 2 skipped as on master. Checkstyle and spotless are clean. ### API and Format No API change. The optional summary fields in Iceberg metadata now have correct values. No format change. ### Documentation No. Generated-by: Cursor (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
