jayzhan211 opened a new pull request, #26563: URL: https://github.com/apache/datafusion/pull/26563
Based on #26562, which adds the tests this refactor keeps unchanged. Only the last commit is new here; this stays a draft until #26562 merges. ## Which issue does this PR close? - None. Pure refactor. ## Rationale for this change The seven grouped aggregate streams (the Partial, Final, Single and PartialReduce hash streams and the three clustered ones) each carry their own copy of the same memory rules: - reserve the initial table; - add 4 bytes per group for the sort indices when the stream can spill (four helpers); - shrink to the table after a spill; - release the reservation before a replay; - hold a materialized batch next to the table while handing it out in `batch_size` slices, emitting it whole if it does not fit, and dropping it from the reservation before the last slice (five copies). A change to one of these rules has to be made in every stream. Reserving the initial table (#25900) touched six stream files, and holding the output batch while slicing was added to the hash streams in #24852 and then written again for the clustered streams in #25312, #25639 and #25984. When the copies drift, a stream overruns its limit or fails a query that should succeed. ## What changes are included in this PR? - `StreamMemory` owns a stream's reservation and the arithmetic above: `try_new`, `try_fit_table` (table plus the spill headroom), `shrink_after_spill`, `hold`, `release_for_replay`, `release`. - `SlicedOutput` is a batch being handed out in slices while it stays reserved; `next_slice` drops it from the reservation before the last slice. - Each stream keeps its own decisions exactly where they are on main: how it registers (`can_spill` and the comments about it), and what it does when the pool refuses its final output (Partial and PartialReduce ignore it, Final returns it, Single hands the output off unreserved). Everything is private to the `aggregates` module; there is no public API change. ## What is the testing strategy for this PR? The pool call snapshots added in #26562 stay unchanged, so every stream makes the same calls with the same sizes in the same order and gets the same results and errors. The physical-plan tests, the `memory_limit` and aggregate fuzz integration tests, and all sqllogictests pass. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
