andygrove opened a new issue, #25650: URL: https://github.com/apache/datafusion/issues/25650
### Is your feature request related to a problem or challenge? When running in containers (e.g. in k8s), if a process tries to allocate more memory than is available then the process will get OOM killed. This happens in downstream projects using DataFusion because the memory pool / accouinting is not accurate. This is a problem in Comet for example. See details at https://datafusion.apache.org/comet/contributor-guide/memory_management.html Let's use this issue to discusss potential solutions and approaches. One idea is to tie into the actuall allocator (e.g. jemalloc) so we can see actual allocated bytes. We could use that information in diferent ways: - log the actual usage alongside reported memory pool sizes so that we can at least see where they drift - implement a memory pool that rejects allocations if allocated bytes is over some limit Related: https://github.com/apache/datafusion/issues/22758 ### Describe the solution you'd like _No response_ ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
