karan-canva opened a new issue, #10574: URL: https://github.com/apache/paimon/issues/10574
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.0.0. The same code is in 2.1.0 and on master (b63437353). The counters were added in #6503, so 1.x releases are not affected: they write an operation-only summary. ### Compute Engine Flink (Paimon sink), Iceberg compatibility enabled (`metadata.iceberg.storage`), Iceberg format version 2. ### Minimal reproduce step 1. Create a primary-key table with Iceberg compatibility enabled (one bucket is enough). 2. Commit three small writes without compaction, then commit a write with a full compaction. 3. Compare the summary of the latest Iceberg snapshot with the live entries in its data manifests: ``` jq '.snapshots[-1].summary | {operation, "total-data-files", "total-records", "total-files-size", "deleted-data-files"}' metadata/vN.metadata.json ``` After step 2 the manifests contain 1 live data file and this commit marked 1 entry as DELETED, but the summary has: ``` "operation": "replace", "added-data-files": "1", "deleted-data-files": "4", "total-data-files": "0", "total-records": "0", "total-files-size": "0" ``` On a staging table running 2.0.0 we saw `total-data-files = 0` while the current manifests had 943 live data files. ### What doesn't meet your expectations? The `total-*` fields should match the live entries in the snapshot's manifests, and `deleted-*` should match the entries the commit marked as DELETED. Instead the totals drift down, until the `Math.max(0, ...)` clamp holds them at 0. These fields are optional in the Iceberg spec, but some engines read them. In our case Snowflake declined to refresh an Iceberg table whose totals did not match its manifests, and the table went stale. There are two causes in `IcebergCommitCallback`: **1. The deleted counters include files that were never exported to Iceberg.** `collectFileChanges` only records an ADD if `shouldAddFileToIceberg` is true. For a primary-key table that means the max level, or level > 0 with deletion vectors. DELETE entries have no such check, so every removed Paimon file goes into `removedFiles`. `deleted-data-files`, `deleted-records` and `removed-files-size` are computed from `removedFiles`, and the running totals subtract them. Every compaction removes low-level files that Iceberg never saw, so the totals drop on each compaction. In the example above, Iceberg deleted 1 file but the summary counts 4. **2. The live-count helpers subtract deleted entries from a count that already excludes them.** `computeLiveFileCount` and `computeLiveRowCount` return `added + existing - deleted` for each manifest. In a manifest list these three counts are disjoint (`IcebergManifestFile` puts each entry in exactly one of them), so the live count is `added + existing`, the same as `IcebergManifestFileMeta#liveRowsCount`. The helpers undercount whenever a manifest has DELETED entries. They are used when the previous snapshot has no totals (for example metadata written before #6503), and for `total-delete-files` and `total-position-deletes`. Since every total is the previous total plus this commit's changes, a table that already has wrong totals keeps them, even after both causes are fixed. ### Anything else? - Affected code: `paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java` (`createMetadataWithBase`, `collectFileChanges`, `computeLiveFileCount`, `computeLiveRowCount`). - Filtering DELETE entries with `shouldAddFileToIceberg` is not a safe fix. The first Iceberg snapshot is built from raw-convertible splits, which can export a level-0 file. When that file is compacted away, the DELETE has level 0 and would be skipped, so Iceberg would never mark it DELETED. - Proposed fix: take the deleted counters from the entries the commit actually marks as DELETED, and compute `total-data-files` and `total-records` from the new manifest list, as `total-delete-files` already is. Then tables with wrong totals are corrected on their next commit. File sizes are not in the manifest list, so `total-files-size` can still be carried forward, recomputed from the manifests when the previous totals do not match them. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
