Thank you Thomas, that's helpful. Though I really like the future interface, I've used it many times while I never used `multiprocessing.pool.Pool` or its `AsyncResult`. I worry that some of the expectations I've grown to have from `Future` objects would be broken here.
It would be nice if this feature existed in `concurrent.futures`. Thanks, Ram. On Sat, Jun 26, 2021 at 7:28 PM Thomas Grainger <tagr...@gmail.com> wrote: > And it's on Python stdlib 3 > https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool > > On Sat, 26 Jun 2021, 17:24 Thomas Grainger, <tagr...@gmail.com> wrote: > >> billiard a multiprocessing py2 fork/backport has >> https://billiard.readthedocs.io/en/latest/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool >> with maxtasksperchild >> >> On Sat, 26 Jun 2021, 16:57 Ram Rachum, <r...@rachum.com> wrote: >> >>> Hi guys, >>> >>> I want to have a version of `concurrent.futures.ProcessPoolExecutor` in >>> which every worker process shuts down after each task. I want this because >>> I believe I have a memory leak in my program that I wasn't able to fix. >>> (Possibly it's in C extensions that I use.) >>> >>> Since I'm going to use a process pool anyway, I'm hoping to sidestep >>> this memory leak by launching every task on a worker process that shuts >>> down after it's done. >>> >>> A few years back when I was working at Dell, they did something similar. >>> They had a Python server that had a memory leak, and they worked around it >>> by having a process that can be easily restarted, and then they restarted >>> it often. >>> >>> The plan is that each new work item will run on a fresh process that has >>> no memory leaks from previous work items. >>> >>> I looked at the `futures/process.py` file to see whether I could >>> subclass `ProcessPoolExecutor` and add this functionality, and boy oh boy >>> this is not easy. >>> >>> Do you think it's worthwhile to add this feature to >>> `ProcessPoolExecutor`? >>> >>> >>> Thanks, >>> Ram. >>> _______________________________________________ >>> Python-ideas mailing list -- python-ideas@python.org >>> To unsubscribe send an email to python-ideas-le...@python.org >>> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >>> Message archived at >>> https://mail.python.org/archives/list/python-ideas@python.org/message/UNDY7YKBJ4DOB6H45N7T6QK7PVJBIKZV/ >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NZD5U73MFEZX3A3GQTJBYAQ66D76GPCC/ Code of Conduct: http://python.org/psf/codeofconduct/
