csarushan1729 opened a new pull request, #20298: URL: https://github.com/apache/hudi/pull/20298
Flink data-skipping can silently drop matching rows after expanding the Column Stats Index column list (`hoodie.metadata.index.column.stats.column.list`). Old files lack statistics for newly indexed columns. `FileStatsIndex.foldRowsByFiles` was padding missing records as `(null, null, valueCount)`, which evaluators treated as “all values are NULL”, causing correct rows to be pruned. Closes #20289 ### Summary and Changelog - Treat missing column statistics as **unknown** instead of all-null. - Introduced `nullCnt = -1` as an explicit “statistics unavailable” sentinel in `FileStatsIndex.foldRowsByFiles`. - Added `ColumnStats.isUnknown()` helper. - Updated all leaf evaluators in `ExpressionEvaluators` (`EqualTo`, `NotEqualTo`, `IsNull`, `IsNotNull`, `LessThan`, `GreaterThan`, `LessThanOrEqual`, `GreaterThanOrEqual`, `In`) to return `true` (keep the file) when stats are unknown. - True all-null columns continue to be pruned correctly. ### Impact - Fixes silent data loss / incorrect query results under data-skipping after CSI column list expansion. - No public API change. - No performance regression for fully-indexed columns. Files with incomplete statistics are retained (correct and conservative behavior). ### Risk Level low The change is narrowly scoped to the data-skipping path. It only makes the system more conservative (keeps extra files when stats are missing) and does not change behavior for fully indexed columns. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
