cshuo opened a new issue, #20508: URL: https://github.com/apache/hudi/issues/20508
## Description Flink data skipping can incorrectly discard files or partitions when an `IN` predicate contains both non-null values and `NULL`. For example, `x IN (1, NULL)` evaluates to `TRUE` when `x = 1`. However, `ExpressionEvaluators.fromExpression` currently returns `AlwaysFalse` if any literal in the expression is `NULL`. `ExpressionEvaluators.In.eval` also returns `false` when its list contains `NULL`. As a result, if this expression reaches the evaluator, data skipping can discard files or partitions that contain matching records. ## Expected behavior Non-null candidates should still be considered when evaluating whether an `IN` predicate can match. A list containing only `NULL` values cannot match any row in a `WHERE` clause. ## Reproduction notes The issue can be reproduced by constructing a Flink `IN` expression with a non-null literal and a `NULL` literal, and evaluating it against column statistics whose range includes the non-null value. The evaluator currently returns `false`. In testing with Flink 1.20, the SQL optimizer rewrites a literal `IN` list before it reaches Hudi and removes the `NULL` candidate from the filter. Therefore, a SQL integration test on that version does not reproduce this evaluator behavior. ## Validation Add regression tests covering: - A matching non-null candidate alongside `NULL`. - A non-matching non-null candidate alongside `NULL`. - A list containing only `NULL` values. - A large list containing both non-null values and `NULL`. Parent issue: #20500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
