voonhous commented on issue #20500: URL: https://github.com/apache/hudi/issues/20500#issuecomment-6600688471
Two more sets of Flink data skipping findings, from reading `ExpressionEvaluators` and `ColumnStatsProbe` at b0d16f9. **Evaluator defects that would prune files holding matching rows.** Each returns "cannot match" for a file that can match. A Flink SQL test (column stats on, with an explicit column list; one file with `a` in 1..10, one with 100..110) returned the same rows with data skipping on and off for `NOT (a < 5)`, `id = 1000 OR ts = 999 OR a = 3` and `a IN (1, CAST(NULL AS INT))`, most likely because the planner rewrites these filters before `applyFilters`. The evaluator is still wrong for any expression that reaches it in these shapes: - NOT negates the stats answer instead of the predicate: [`Not#eval`](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java#L472) returns `!evaluator.eval(stats)`. For `min=1, max=10`, `a < 5` might match, so `NOT (a < 5)` prunes the file although it holds `a >= 5`. - AND and OR bind only their first two children ([L82-L94](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java#L82-L94)). - Any null literal turns the leaf into `AlwaysFalse` ([L107-L113](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java#L107-L113)), so `a IN (1, NULL)` would prune every file. **Missed pruning (results stay correct):** - DATE stats are read with `getLong` but stored as int; the `ClassCastException` is caught in `FileStatsIndex#computeCandidateFiles`, so DATE filters never prune ([`ColumnStatsProbe`](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/ColumnStatsProbe.java#L122)). - The partitioned record index is skipped above 3 candidate partitions, while Spark allows 10 ([`RecordLevelIndex.MAX_PARTITIONS`](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/RecordLevelIndex.java#L69)). - Flink and Java writers leave the column stats index definition without columns: #17351. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
