gruuya opened a new issue, #25904: URL: https://github.com/apache/datafusion/issues/25904
### Is your feature request related to a problem or challenge? We ran into a case where a DataFusion process hit memory resource limits (and got OOMed) before it actually got near the set memory pool limit. The nature of the query (many join operators, many partitions), led claude to believe a significant part of it was down to unaccounted memory in the `SharedCoalescer`, which was coincidentally also discussed in the original PR https://github.com/apache/datafusion/pull/22010#discussion_r3235192165 ### Describe the solution you'd like Thread a memory reservation down to `SharedCoalescer` and grow/shrink it as the batches are pushed/drained from it. Even though an individual `SharedCoalescer` by default buffers only a modest number of rows, this problem can compound otherwise, since `SharedCoalescer` are per partition and per operator. ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
