gabotechs opened a new issue, #25613: URL: https://github.com/apache/datafusion/issues/25613
## Describe the bug Decimal types are excluded from interval analysis. Adding quantity and discount restrictions to a date range causes the entire filter to receive a blanket 20% estimate. ## To Reproduce From the repository root, with the CLI fix from [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed repro_dir=$(mktemp -d) tpchgen-cli --scale-factor 1 --format parquet \ --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.execution.target_partitions = 1; SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SELECT l_orderkey FROM lineitem WHERE l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1995-01-01'; SELECT l_orderkey FROM lineitem WHERE l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1995-01-01' AND l_discount >= 0.05 AND l_discount <= 0.07 AND l_quantity < 24; SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed with `tpchgen-cli` 1.1.1 at [6c320561b5](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67). Inspect the SELECT reports; ignore the empty `SET` reports. | SELECT | FilterExec node | Estimated rows | Actual rows | | ------------------------- | --------------- | -------------: | ----------: | | Date range | `0` | 869,535 | 909,455 | | Date + Decimal conditions | `0` | 1,200,243 | 114,160 | ## Expected behavior Support Decimal comparison selectivity with correct scales, coercions and boundaries. Preserving the supported date condition is a separate improvement from estimating the Decimal restrictions. ## Additional context This isolates TPC-H Q6’s predicate. Related: [filter selectivity #14237](https://github.com/apache/datafusion/issues/14237). Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
