gabotechs opened a new issue, #25612: URL: https://github.com/apache/datafusion/issues/25612
## Describe the bug Adding an unsupported AND condition discards the useful estimate for the supported condition. The extra restriction increases the estimate tenfold because the whole predicate falls back to 20% selectivity. ## To Reproduce From the repository root, with the CLI fix from [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed repro_dir=$(mktemp -d) tpchgen-cli --scale-factor 1 --format parquet \ --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.execution.target_partitions = 1; SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SELECT p_partkey FROM part WHERE p_size = 15; SELECT p_partkey FROM part WHERE p_size = 15 AND p_type LIKE '%BRASS%'; SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed with `tpchgen-cli` 1.1.1 at [6c320561b5](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67). Inspect the SELECT reports; ignore the empty `SET` reports. | SELECT | FilterExec node | Estimated rows | Actual rows | | ------------------------------- | --------------- | -------------: | ----------: | | `p_size = 15` | `0` | 4,000 | 3,907 | | Add `AND p_type LIKE '%BRASS%'` | `0` | 40,000 | 747 | ## Expected behavior Retain supported conjunct estimates and combine unsupported conjuncts conservatively. Adding an AND condition should not increase estimated rows; OR requires different treatment. ## Additional context Related: [filter selectivity #14237](https://github.com/apache/datafusion/issues/14237). Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
