JingsongLi commented on PR #10730: URL: https://github.com/apache/paimon/pull/10730#issuecomment-6333896255
There is an existing-data upgrade gap in the manifest-level fix. `MosaicSimpleStatsExtractor` drops these bounds when metadata is created, but `AppendOnlyFileStoreScan.filterByStats` reads the already-persisted `entry.file().valueStats()` without re-extracting it. Files written before this change can therefore still be pruned before the fixed `MosaicRecordsReader` is reached. I reproduced this for both FLOAT and DOUBLE: write `[NaN, -5, 3]` with the pre-fix extractor behavior and `mosaic.stats-columns=v`, then read with the fixed behavior and `v < 0`. Reading the unfiltered split with the fixed reader returns row 2, while the normal filtered scan plans zero splits. The 162 existing module tests pass, but this upgrade regression test fails for both types. This is a remaining failure of the original bug, rather than a new regression introduced here. Before treating this as a production fix for existing tables, please cover persisted unsafe manifest statistics as well, or explicitly document the limitation and a verified procedure for rebuilding the affected metadata/data. Please add a test that writes legacy bounds and reads them with the fix; the current tests write all metadata using the new extractor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
