gabotechs opened a new issue, #25615: URL: https://github.com/apache/datafusion/issues/25615
## Describe the bug The SUM result has no useful statistics for a downstream HAVING filter. A simplified Q18 query separates this loss from the already inaccurate group-count estimate. ## To Reproduce From the repository root, with the CLI fix from [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed repro_dir=$(mktemp -d) tpchgen-cli --scale-factor 1 --format parquet \ --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.execution.target_partitions = 1; SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey HAVING SUM(l_quantity) > 300; SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed with `tpchgen-cli` 1.1.1 at [6c320561b5](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67). Inspect the SELECT reports; ignore the empty `SET` reports. | Operator | Node | Estimated rows | Actual rows | | ------------------------ | ------- | -------------: | ----------: | | AggregateExec | `0.0.0` | 6,001,215 | 1,500,000 | | FilterExec (`SUM > 300`) | `0.0` | 1,200,243 | 57 | ## Expected behavior Propagate conservative aggregate-result statistics and use them for downstream selectivity. Rare SUM tails may require distribution summaries; bounds alone are insufficient. ## Additional context The q-error grows from 4× at the aggregate to 21,057× at the filter. Improving group counts alone does not model the SUM distribution. Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
