Anakin100100 commented on issue #49474: URL: https://github.com/apache/arrow/issues/49474#issuecomment-4116599785
I looked at the code and the system one seems to be mimalloc. The documentation is slightly misleading because both available allocators, mimalloc and jemalloc are malloc implementations but there is no code path for compiling with libmalloc directly. Alternatively you could compile arrow with jemalloc https://github.com/jemalloc/jemalloc but I don't think it would bring you any benefits. When I worked with ClickHouse I learned that it has a similar memory release philosophy as mimalloc. For a performance critical software like arrow there doesn't seem to have been a lot of demand for aggressively releasing memory to the os by default. One think I may suggest would be to set the memory limits on processes/containers that are in line with expected memory use. This would allow for mimalloc to allocate and use the memory that it needs without affecting performance. You can tune mimalloc for your needs based on the env vars described here https://microsoft.github.io/mimalloc/environment.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
