lxc512157407 opened a new issue, #24942: URL: https://github.com/apache/datafusion/issues/24942
## Summary Join planning inserts null-rejecting `IS NOT NULL` filters on nullable join keys (`filter_null_join_keys`), which are then pushed down to scans. When the underlying data contains no NULLs — the overwhelmingly common case for join keys, e.g. TPC-H primary/foreign keys — these filters pass 100% of rows, yet every input batch still pays a full predicate evaluation plus a full batch copy. On TPC-H SF=10 we measured `selectivity = 100% (6.00M/6.00M)` for such filters on join keys. ## Why this is wasteful `FilterExec::statistics_helper` already derives the forward direction: a surviving `IS NOT NULL` conjunct implies `null_count = Exact(0)` for the output. The inverse is missing: when *input* statistics already prove `null_count = Exact(0)` for a column, a bare `Column IS NOT NULL` conjunct is vacuously true and could be dropped from the predicate at plan-construction time (`FilterExecBuilder::build`). If all conjuncts are dropped, the predicate degenerates to `lit(true)` and the filter becomes a no-op pass-through (or can be elided entirely). Non-provable cases (statistics `Absent` or null_count `Inexact`) must of course keep the predicate unchanged — this is strictly a statistics-driven refinement. ## Impact - Every join query over sources that report exact zero null-counts (Parquet with statistics, custom TableProviders with exact stats) executes no-op filter evaluations per batch. - For plans with many joins (TPC-H), the redundant filters stack up on the scan side. ## Related - PR implementing the conjunct-drop at `FilterExecBuilder::build`: #24821 (filed this issue per review discussion there) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
