alamb commented on issue #25183: URL: https://github.com/apache/datafusion/issues/25183#issuecomment-5766397985
> [@alamb](https://github.com/alamb) [@jayzhan211](https://github.com/jayzhan211) Following the discussion in [#24923](https://github.com/apache/datafusion/pull/24923), one possible approach is to reserve memory for spill output buffers before materializing the batch. > , then hold that budget until the async write finishes. Yes, i think this sounds reasonable to me -- in my mind if we can't get a reservation the system shouldn't be allocating memory. I think we should aim for zero overshoot, even though that will not always be possible. If the process is using memory we should try to keep a reservation for it > If the reservation cannot be satisfied, even with available workspace, we could reduce the output row count and try again I view this as a optimization under memory pressure. Like all optimizations it will add complexity but that maybe the right tradeoff. > than introducing a hard failure for that shortfall. Again, think a hard failre for shortfall is my suggested approach as it is simple to explain. If users want to work around the issue or potentially overcommit memory, they can increase the memory pool size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
