gabotechs opened a new issue, #25620: URL: https://github.com/apache/datafusion/issues/25620
## Describe the bug A bare null check falls back to 20% selectivity even when input column statistics contain a null count. IS NULL and IS NOT NULL receive the same estimate. ## To Reproduce From the repository root, with [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench repro_dir=$(mktemp -d) mkdir -p "$repro_dir/data" curl --fail --location \ https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/store_sales.parquet \ -o "$repro_dir/data/store_sales.parquet" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.execution.target_partitions = 1; SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NULL; SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NOT NULL; SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed at [this revision](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67) with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the empty SET reports. | Predicate | FilterExec node | Estimated rows | Actual rows | | ----------- | --------------- | -------------: | ----------: | | IS NULL | `0` | 576,081 | 129,975 | | IS NOT NULL | `0` | 576,081 | 2,750,429 | ## Expected behavior Use the input null count for IS NULL and subtract it from the input row count for IS NOT NULL, retaining the appropriate precision. Here those estimates would be 129,975 and 2,750,429. ## Additional context Reduced from TPC-DS Q76. The input has 2,880,404 rows; a diagnostic capture confirms its ss_addr_sk null count is available as Inexact(129975). Related: [#24942](https://github.com/apache/datafusion/issues/24942) concerns removing provably redundant null checks; this report concerns estimating checks that actually remove rows. Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
