plusplusjiajia opened a new pull request, #10649: URL: https://github.com/apache/paimon/pull/10649
### Purpose When a commit removes files, `IcebergCommitCallback#createWithDeleteManifestFileMetas` prunes the base manifests by their partition summaries before it marks the removed files as DELETED. Two summary fields were read wrongly, so a manifest that holds a removed file could be skipped. The file then stays live in the Iceberg metadata after Paimon has removed it. Iceberg readers keep returning its rows, and fail on the missing file once Paimon expires the snapshot and deletes it. The snapshot summary totals are computed from the same manifests, so they do not show the problem. - `contains_null` was passed as a null count of 1, next to the manifest's live row count. A manifest with one live row then looked all null, and the predicate of a non-null partition rejected it. `contains_null` also covers DELETED entries, so a rewritten manifest can report a null partition while its single live row is in another partition. This dates back to #3739. - Since #9188, the bounds leave out NaN. The predicate of a NaN partition compared NaN with bounds that cannot hold it, and rejected the manifest. This affects 2.1.0. The null count is now treated as unknown when `contains_null` is set, and the bounds are not used when `contains_nan` is set. Pruning still applies otherwise: only manifests with NaN partition values, or with a single live row and a null partition, are now read where they were skipped before. ### Tests - `IcebergCompatibilityTest#testDropPartitionInManifestWithOneLiveRowAndNullPartition`: one manifest holds a null partition and `pt=1`. Dropping the null partition leaves one live row, and dropping `pt=1` must then remove it from Iceberg. On master, Iceberg still returns `Record(1, 2)`. - `IcebergCompatibilityTest#testDropNanPartitionInManifestWithOtherPartitions`: one manifest holds `pt=1.0` and `pt=NaN`, with bounds `[1.0, 1.0]`. Dropping the NaN partition must remove it from Iceberg. On master, the NaN row stays. - `mvn -pl paimon-core -Dtest='Iceberg*Test'`: 196 tests pass. `mvn -Ppaimon-iceberg -pl paimon-iceberg -Dtest='Iceberg*Test'`: 68 tests pass, 2 skipped as on master. Checkstyle and spotless are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
