gabotechs opened a new issue, #25611: URL: https://github.com/apache/datafusion/issues/25611
## Describe the bug An accurate filter estimate becomes a 1.45-million-fold error at aggregation: input rows are reused as the number of groups when grouping keys lack explicit NDVs. Grouping by a derived year has the same fallback. ## To Reproduce From the repository root, with the CLI fix from [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed repro_dir=$(mktemp -d) tpchgen-cli --scale-factor 1 --format parquet \ --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.execution.target_partitions = 1; SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SELECT l_returnflag, l_linestatus, COUNT(*) FROM lineitem WHERE l_shipdate <= DATE '1998-09-02' GROUP BY l_returnflag, l_linestatus; SELECT EXTRACT(YEAR FROM l_shipdate) AS y, COUNT(*) FROM lineitem GROUP BY EXTRACT(YEAR FROM l_shipdate); SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed with `tpchgen-cli` 1.1.1 at [6c320561b5](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67). Inspect the SELECT reports; ignore the empty `SET` reports. | SELECT | Node | Estimated groups | Actual groups | | ------------------ | ---------------------- | ---------------: | ------------: | | Return flag/status | `AggregateExec`, `0.0` | 5,787,395 | 4 | | Shipping year | `AggregateExec`, `0.0` | 6,001,215 | 7 | ## Expected behavior Use available group-domain or expression statistics instead of immediately falling back to input rows. Keep uncertain range bounds distinct from measured NDVs. ## Additional context The first aggregate’s filter input estimates 5,787,395 rows versus 5,916,591 actual, within 3%. Related: [NDV epic #20766](https://github.com/apache/datafusion/issues/20766). Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
