cshuo opened a new issue, #20480: URL: https://github.com/apache/hudi/issues/20480
### Bug Description **What happened:** Flink incremental reads can return incomplete results when data skipping uses current partition stats to prune partitions containing historical records. After an update changes a column’s range, current statistics may no longer cover values in files selected by an earlier incremental range. **What you expected:** Incremental reads return all records matching the requested range and predicate. Current partition stats must not prune partitions when they do not cover the selected historical files. Partition-field pruning can remain enabled. **Steps to reproduce:** 1. Create a COW table with metadata, column stats and partition stats enabled. 2. In commit 1, write `id1, age = 23` to `par1` and `id2, age = 23` to `par2`. 3. In commit 2, update `id1` to `age = 50`, so the current statistics for `par1` no longer include `23`. 4. Enable `read.data.skipping.enabled` and query `age = 23` using `read.start-commit` and `read.end-commit` set to the completion time of commit 1. Expected: both historical records are returned. Actual: `par1` can be pruned, leaving only the record from `par2`. The issue was reproduced for both batch incremental reads and streaming reads, with the legacy source and Source V2. ### Environment **Hudi version:** 1.3.0-SNAPSHOT **Query engine:** Flink 2.2 **Relevant configs:** metadata table, column stats, partition stats, and `read.data.skipping.enabled` enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
