raghavyadav01 opened a new pull request, #26548: URL: https://github.com/apache/datafusion/pull/26548
## Which issue does this PR close? Closes #26542. ## Rationale for this change Under filter semantics only rows for which the predicate is `TRUE` are kept, so `NULL` rejects a row exactly like `FALSE`. That makes two rewrites sound for `IN` lists that hold `NULL` literals: - `x IN (a, NULL)` is `TRUE` exactly when `x` matches `a`. A `NULL` member can only turn `FALSE` into `NULL`, and both reject. The `NULL` members can be dropped, and a list of nothing but `NULL`s rejects every row. - `x NOT IN (a, NULL)` is `FALSE` when `x` matches `a` and `NULL` otherwise, so it is never `TRUE` and rejects every row. Leaving the `NULL`s in place keeps a filter that `PruningPredicate` does not handle well, which is the pruning degradation described in the issue. #24711 fixed this only for the compact path (lists over 20 members) and #25187 made short lists on several types reach the same path. ## What changes are included in this PR? An `Expr::InList` arm in `simplify_filter_predicate`, plus a `simplify_filter_in_list` helper and an `is_null_literal` predicate, in `datafusion/optimizer/src/eliminate_filter.rs`. The rewrite stays in `EliminateFilter` deliberately. It is **not** sound in `ExprSimplifier`: in a general expression context `SELECT 3 IN (1, NULL)` must remain `NULL`, and dropping the `NULL` would make it `FALSE`. The negated case keeps the original `InList` inside `FilterPredicate::RejectsAll` rather than substituting a literal, so nothing is discarded that the `AND`/`OR` reconstruction may still need. ## Are these changes tested? Yes — five tests in `eliminate_filter.rs`: | test | asserts | |---|---| | `filter_in_list_with_null_drops_null` | `a IN (1, 2, NULL)` → `a IN (1, 2)` | | `filter_in_list_of_only_nulls_rejects_all` | `a IN (NULL)` → `EmptyRelation` | | `filter_not_in_list_with_null_rejects_all` | `a NOT IN (1, NULL)` → `EmptyRelation` | | `filter_in_list_without_null_is_untouched` | no `NULL` → unchanged | | `filter_not_in_list_without_null_is_untouched` | no `NULL` → unchanged | I wrote the tests first and confirmed the three `NULL` cases **fail on `main`** before the change and pass after, while the two control tests pass in both states. Also run: the full `datafusion-optimizer` suite (948 + 29 + 5 passing), `cargo fmt --all -- --check`, and `cargo clippy -p datafusion-optimizer --all-targets -- -D warnings`, all clean. `in_list.slt` and the `null_aware_*` slt files pass. `expr`, `predicates`, `scalar` and `aggregate` slt files each fail identically with and without this change on my machine (a local test-data issue), so they are unaffected. ## Are there any user-facing changes? Explain output for filters whose `IN` list contains `NULL` literals: the `NULL` members no longer appear, and a filter that can never be `TRUE` becomes an empty relation. Query results are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
