comphead opened a new pull request, #27351: URL: https://github.com/apache/datafusion/pull/27351
## Which issue does this PR close? - Closes #27350. ## Rationale for this change Under memory pressure the Partial aggregate emits its groups early and continues with an empty table. The skip probe then compared all input rows with only the groups since the last emit, so it stayed below `skip_partial_aggregation_probe_ratio_threshold` even for unique keys. The query sends the same rows downstream either way, but spends several times the Partial CPU hashing them. ## What changes are included in this PR? `PartialHashAggregateStream` and the legacy `GroupedHashAggregateStream` count the groups they emit early and pass them to `SkipAggregationProbe` together with the groups still in the table. The sum is the output row count so far, so the probe measures the reduction the Partial actually achieves. ## What is the testing strategy for this PR? `test_partial_aggregation_skips_after_early_emits` runs both streams over unique keys with a memory limit that forces early emits before the probe threshold. On `main` both skip 0 rows. `datafusion-cli` with the query from the issue (2M unique keys, `target_partitions = 2`), `AggregateExec: mode=Partial`, 3 runs: | Memory pool | `main` emits / skipped / CPU | This PR emits / skipped / CPU | |---|---|---| | None | 0 / 1.79 M / 6.4 ms | 0 / 1.79 M / 6.7 ms | | `memory_limit = '4M'` | 87–102 / 0 / 33 ms | 5–7 / 1.79 M / 6.9 ms | | `-m 16M --mem-pool-type fair` | 46–47 / 0 / 32 ms | 4 / 1.79 M / 6.8 ms | With 20M rows over 200K keys, 8 partitions and a fair 32M pool, Partial CPU drops from 345 ms to 46 ms. Queries that still reduce well under memory pressure (20K or 2K keys) keep aggregating. ## Are there any user-facing changes? No API changes. Memory-limited queries over high-cardinality keys now skip partial aggregation, as they already do without a limit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
