pitrou commented on issue #44599: URL: https://github.com/apache/arrow/issues/44599#issuecomment-2479338047
As I said, RSS can be misleading as it shows some memory to be in use even though the kernel might be able to reuse it if desired (the `MADV_FREE` case I linked to). I'm not saying this is exactly what happens here, but it's quite likely given that the system allocator returns the memory more eagerly. One way to clear this up would be: run this reproducer and keep the Python process running (for example using `time.sleep(3600)` at the end). Then start another process that tries to use all memory, and see if the Python process' RSS finally shrinks. You may also want to open a question on the https://github.com/microsoft/mimalloc issue tracker, they'll probably be able to give more precise insight. Also, you could experiment with some of the environment variables here: https://microsoft.github.io/mimalloc/environment.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
