kumarUjjawal opened a new pull request, #25648: URL: https://github.com/apache/datafusion/pull/25648
## Which issue does this PR close? Closes #25620. ## Rationale for this change Bare `IS NULL` and `IS NOT NULL` filters currently use the default selectivity even when the input has a null count. For five rows containing two NULLs, they both estimate one row instead of two and three. Using those counts also requires keeping them valid through joins and pushed-down fetch limits. Joins can repeat input rows and add NULL padding, so copying input null counts can give the filter an incorrect estimate or an unjustified exact result. ## What changes are included in this PR? - Estimate bare column null checks from available null counts, preserve their precision, and retain the existing fallback for missing statistics and other predicates. - Apply per-partition fetch limits in both filter statistics paths. Keep all-null, empty, and singleton column statistics consistent with the output rows. - Adjust join null counts for repeated rows, unmatched rows, and NULL padding. Reuse semi-join estimates to count unmatched input rows, and share the NULL-match estimate between single-key null-safe joins. Unknown contributions remain unknown. - Normalize semi/anti-join column statistics and rescale null counts when a statistics provider replaces the row estimate. Avoid applying a non-null distinct-count bound to an `IS NULL` result. These rules use general operator semantics; there are no TPC-H or TPC-DS special cases. ## What is the testing strategy for this PR? Unit coverage includes exact, inexact, missing, and inconsistent statistics; empty and all-null inputs; fetch limits across partitions; projections; all join types; both null-equality modes; duplicate matches; and unknown null counts. SQL logic tests check the estimates and results of null filters, including filters above outer joins. Completed locally: - Extended workspace tests with `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`: 12,075 Rust tests passed and all 522 SQL logic test files passed. - `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check` - Runtime reproductions confirmed the duplicate-match and null-safe key NULL counts against actual join output. Release-mode benchmarks compared this branch with `main` at `7570366fd`. Ad hoc statistics microbenchmarks measured about 20% less time for a bare null check and 30–45 ns more per join-statistics calculation. The existing `compute_statistics` filter-chain comparison was inconclusive because unchanged controls also varied. These measurements cover statistics calculation, not end-to-end query performance. ## Are there any user-facing changes? Null-check row estimates and related `EXPLAIN` statistics improve. Plan choices can change as a result. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
