yashaggarwal85d opened a new pull request, #26637: URL: https://github.com/apache/datafusion/pull/26637
## Which issue does this PR close? - Closes #26628. ## Rationale for this change A `Final` or `Single` aggregate over input sorted on all grouping keys fails with `ResourcesExhausted` under a memory limit that the legacy `GroupedHashAggregateStream` handles. After each input batch, `ClusteredFinalAggregateStream` and `ClusteredSingleAggregateStream` reserve memory for the whole table before emitting completed groups. If that reservation is refused and the stream cannot spill (full group clustering, or temporary files disabled), it returns the error, even though emitting the completed groups would leave only the active group in memory. This is the same problem as #26575 for the partial stream, fixed in #26576. ## What changes are included in this PR? - `ClusteredFinalAggregateStream`: when the reservation is refused and there is no spill context, take the completed groups and move to the output stage, then resume aggregation. The error is returned only when no group is complete. - `ClusteredSingleAggregateStream`: the same behavior, emitting through the existing `start_outputting` path and resuming `ReadingInput`. - Updated doc comments to describe the new behavior. The spilling path (partial group clustering with temporary files enabled) is unchanged. ## What is the testing strategy for this PR? New unit tests, modeled on the one in #26576: - `clustered_final_stream::tests::completed_groups_under_memory_pressure` - `clustered_single_stream::tests::fully_sorted_completed_groups_under_memory_pressure` Each runs `SUM(k) GROUP BY k` over two batches sorted on `k` that share a boundary key. It first runs with an unbounded pool to measure the most memory the stream holds when it hands off output, then reruns with a `GreedyMemoryPool` of that size and checks that the results match. Both tests fail on `main` with `ResourcesExhausted` and pass with this change. All `aggregates` unit tests in `datafusion-physical-plan` pass, and `cargo fmt` and `cargo clippy --all-targets --all-features -D warnings` on the crate are clean. ## Are there any user-facing changes? No API changes. Queries that used to fail with `ResourcesExhausted` in this situation now complete. --- _This solution was developed with AI assistance; I reviewed and tested the changes._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
