alamb commented on issue #9417: URL: https://github.com/apache/arrow-datafusion/issues/9417#issuecomment-1989328293
> But I also believe there's a distinction between optimizing AggExec's output pattern and handling memory accounting. I agree > Regarding the memory accounting side, I'm curious if you have considered alternatives that allow for more accurate accounting for different batches. The idea of having sliced batches not reporting their memory usage or using `get_slice_memory_size` for reporting is a good starting point. What do you think about this? I think it is tricky business and depends on what we are using the memory accounting for At the moment I think the memory accounting is mostly to prevent OOM kills (over commit of memory), since memory for a sliced `RecordBatch` is not returned to the OS , a plan with a 3 row slice of a 1M row RecordBatch still is "using" all 1M rows from the OS perspective However, ensuring we don't double count is important too (like two slices to the same 1M row RecordBatch will count as a total of 2M rows, even though there only a single allocation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
