cshuo commented on issue #20500: URL: https://github.com/apache/hudi/issues/20500#issuecomment-6649205412
> Two more sets of Flink data skipping findings, from reading `ExpressionEvaluators` and `ColumnStatsProbe` at [b0d16f9](https://github.com/apache/hudi/commit/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e). > > **Evaluator defects that would prune files holding matching rows.** Each returns "cannot match" for a file that can match. A Flink SQL test (column stats on, with an explicit column list; one file with `a` in 1..10, one with 100..110) returned the same rows with data skipping on and off for `NOT (a < 5)`, `id = 1000 OR ts = 999 OR a = 3` and `a IN (1, CAST(NULL AS INT))`, most likely because the planner rewrites these filters before `applyFilters`. The evaluator is still wrong for any expression that reaches it in these shapes: > > * NOT negates the stats answer instead of the predicate: [`Not#eval`](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java#L472) returns `!evaluator.eval(stats)`. For `min=1, max=10`, `a < 5` might match, so `NOT (a < 5)` prunes the file although it holds `a >= 5`. > * AND and OR bind only their first two children ([L82-L94](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java#L82-L94)). > * Any null literal turns the leaf into `AlwaysFalse` ([L107-L113](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java#L107-L113)), so `a IN (1, NULL)` would prune every file. > > **Missed pruning (results stay correct):** > > * DATE stats are read with `getLong` but stored as int; the `ClassCastException` is caught in `FileStatsIndex#computeCandidateFiles`, so DATE filters never prune ([`ColumnStatsProbe`](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/ColumnStatsProbe.java#L122)). > * The partitioned record index is skipped above 3 candidate partitions, while Spark allows 10 ([`RecordLevelIndex.MAX_PARTITIONS`](https://github.com/apache/hudi/blob/b0d16f94b3b5f4ce790a9a09e41ebf891b35a53e/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/RecordLevelIndex.java#L69)). > * Flink and Java writers leave the column stats index definition without columns: [Add support to track list of columns indexed with col stats index for flink and java #17351](https://github.com/apache/hudi/issues/17351). For the AND/OR case, Flink’s filter-pushdown path converts the expression into a binary `CallExpression` tree before passing it to the source. For example, `a OR b OR c` becomes `OR(OR(a, b), c)`, so Hudi’s recursive evaluator visits all three operands. See [`RexNodeToExpressionConverter.visitCall`](https://github.com/apache/flink/blob/release-2.2.1/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/RexNodeExtractor.scala#L506-L521). This issue should not occur on the normal filter-pushdown path. As a low-priority follow-up, Hudi could either handle AND/OR expressions with more than two children or add a comment explaining Flink’s expression rewrite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
