viirya opened a new issue, #26911: URL: https://github.com/apache/datafusion/issues/26911
### Is your feature request related to a problem or challenge? When a clustered aggregate stream (partial, final or single, with full group clustering) fails to grow its reservation, it first emits the completed groups and then calls `ClusteredAggregateTable::shrink_to_fit` (#26726, #26782). If no group is complete, for example when one key spans the whole batch, the stream returns `ResourcesExhausted` (partial), or spills or errors (final/single) without shrinking. Shrinking alone could make the reservation fit, because the table still holds capacity it grew for the batch, such as the per-batch accumulator scratch. For example, a partial `APPROX_DISTINCT(v) GROUP BY k` over three 8192-row batches of a single key fails a 64 KiB limit asking for 132.1 KB, although the table holds about 4.3 KB after `shrink_to_fit`. Raised by @comphead in https://github.com/apache/datafusion/pull/26782. ### Describe the solution you'd like On a refused reservation with no completed group, call `shrink_to_fit` and retry the reservation before returning the error or spilling. The existing `memory_pressure_without_completed_groups` tests in the final and single streams would need to reflect this. ### Describe alternatives you've considered Bound the per-batch scratch itself, for example by hashing in fixed-size chunks in `HllGroupsAccumulator`, which also avoids regrowing the scratch for every batch under sustained pressure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
