gabotechs opened a new issue, #25619: URL: https://github.com/apache/datafusion/issues/25619
## Describe the bug Local TopK statistics cap the total estimate at one partition’s limit, while runtime metrics sum rows across all partitions. A simplified grouped query exposes the mismatch without the joins in TPC-H Q3. ## To Reproduce From the repository root, with the CLI fix from [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed repro_dir=$(mktemp -d) tpchgen-cli --scale-factor 1 --format parquet \ --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SET datafusion.execution.target_partitions = 1; SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10; SET datafusion.execution.target_partitions = 4; SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10; SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed with `tpchgen-cli` 1.1.1 at [6c320561b5](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67). Inspect the SELECT reports; ignore the empty `SET` reports. | SELECT | Operator / node | Estimated rows | Actual rows | | --------------- | ------------------------------ | -------------: | ----------: | | One partition | `SortExec(TopK)`, `0.0` | 10 | 10 | | Four partitions | `SortExec(TopK)`, `0.0.0` | 10 | 40 | | Four partitions | `SortPreservingMergeExec`, `0` | 10 | 10 | ## Expected behavior Account for the emitter count in local TopK estimates while preserving the global merge’s limit. Per-partition and overall statistics must use consistent units. ## Additional context Dynamic filtering is disabled. Early cancellation can reduce intermediate output in other plans; this reproducer emits all 40 local rows. Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
