We just ran into this problem when running our aiootp package's memory hard password hashing function (https://github.com/rmlibre/aiootp/). The memory was not being cleared after the function finished running but the script was still live. We tried making sure everything went out of scope and deleting things explicitly inside the function, but that didn't help. We even tried forcing the garbage collector to free up unreferenced memory with import gc; gc.collect(). But that only had a small dent in the memory that was being built up.
The most useful answer online had to do with Python's free lists being created automatically when very large datasets were being processed (https://stackoverflow.com/questions/23937189/how-do-i-use-subprocesses-to-force-python-to-release-memory/24126616#24126616). After putting the memory intensive work into a separate process, as the answer suggested, the memory was finally freed after execution. In our case, we wound up passing the result back in a ``multiprocessing.Manager().List()``. Though, passing your whole numpy dataset back doesn't seem feasible. I'd recommend doing the necessary memory intensive work in a separate process and passing only the necessary conclusions back to the main process. Or you could set up a multiprocessing queue to pass control messages to the spawned (daemon) process so it returns desired results on demand. Here's an excerpt from our code after the fix in case it's helpful. > @classmethod > def passcrypt(cls, password, salt, kb=1024, cpu=1024, hardness=256): > """ > The ``passcrypt`` function can be highly memory intensive. > These resources may not be freed up, & often are not, because of > python quirks around memory management. This is a huge problem. > So to force the release of those resources, we run the function > in another process which is guaranteed to release them. > """ > cls._validate_passcrypt_args(kb, cpu, hardness) > state = Manager().list() > process = Process( > target=cls._passcrypt, > args=(password, salt), > kwargs=dict(kb=kb, cpu=cpu, hardness=hardness, state=state), > ) > process.start() > process.join() > return state.pop() On 2020-05-28 16:00, python-list-requ...@python.org wrote: > Send Python-list mailing list submissions to > python-list@python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-list > or, via email, send a message with subject or body 'help' to > python-list-requ...@python.org > > You can reach the person managing the list at > python-list-ow...@python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-list digest..." > > Today's Topics: > > 1. Re: Custom logging function (zljubi...@gmail.com) > 2. Ram memory not freed after executing python script on ubuntu > system (Rahul Gupta) > 3. Re: Ram memory not freed after executing python script on > ubuntu system (Chris Angelico) > 4. Re: Custom logging function (Peter Otten) > 5. Re: Elegant hack or gross hack? TextWrapper and escape codes > (Peter Otten) > 6. Re: Elegant hack or gross hack? TextWrapper and escape codes > (Chris Angelico) > 7. Re: Elegant hack or gross hack? TextWrapper and escape codes > (Peter Otten) > 8. Re: Ram memory not freed after executing python script on > ubuntu system (Rahul Gupta) > 9. Re: Ram memory not freed after executing python script on > ubuntu system (Chris Angelico) > 10. Re: Behaviour of os.path.join (BlindAnagram) > 11. Constructing mime image attachment (Joseph L. Casale) > 12. Re: Behaviour of os.path.join (Eryk Sun) > 13. Re: Behaviour of os.path.join (Eryk Sun) > 14. Re: Behaviour of os.path.join (BlindAnagram) > 15. Re: Behaviour of os.path.join (Eryk Sun) -- https://mail.python.org/mailman/listinfo/python-list
