mbutrovich commented on issue #23393: URL: https://github.com/apache/datafusion/issues/23393#issuecomment-4959456025
Fragmentation is inevitable. Allocations are pretty much always going to be page-sized so it's hard to take too much away from measuring small allocations against heap-reported memory since it gets amortized over time by a good allocator. Stuff like this: > SlidingDistinctCountAccumulator / SlidingDistinctSumAccumulator — size() returns only size_of_val(self), ignoring the HashMap bucket capacity and ScalarValue key heap. > functions-aggregate/src/count.rs:557, sum.rs:619. That's valuable to surface since we should at least be counting non-ephemeral data structures. More broadly, I think the biggest challenge with accurate memory accounting is when batches share underlying buffers, particularly the first stage out of a parquet read. For example in this PR, really anywhere we make decisions about data being accumulated and possibly spilling, I had to introduce logic to dedupe underlying buffers https://github.com/apache/datafusion-comet/pull/4507. This commit was the subset https://github.com/apache/datafusion-comet/pull/4507/changes/f909136e04ac1d242bafaa064b4a0608c872f945 related to memory accounting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
