New submission from Harsh Patel <phars...@gmail.com>:
We have a long-running process, which uses thread pool to get tasks and these threads then schedule the tasks and send them to multiprocessing.Pool (with maxtasksperchild limit set) using apply function. The original codebase is large but I have extracted this logic into a simple script for convenience so that you can see the issue. The issue is that after sometime the apply function call gets blocked infinitely. This only happens when we use maxtasksperchild and cause the library for multiple process recreation. How script works: Script creates an object of the Manager class which is responsible for creating a pool of threads and creating multiprocessing pool. Each thread is provided jobs to execute which is done by Scheduler.get_ready_jobs function call. I have designed this function in a way that 100 jobs are returned after every 5 calls to this function else it returns no jobs. Each thread while performing job sends processing_func to multiprocessing pool for execution which just prints "processing something" and exits. Environment: Python 3.7.10 Clang 12.0.5 MacOS v11.6 intel Steps to reproduce: 1. Download the attached Python script and execute it 2. Let it run for 10-15 seconds and interrupt the execution You will notice that some threads will exit successfully but for some, you will see the last message printed is "executing cpu_pool apply <thread_name>" which indicates these threads are blocked even though the actual processing_func is just calling a print function. Warning: You will also not be able to exit the process without killing it. As a note, this is my first Python issue report so please let me know if you need more information. ---------- components: Library (Lib) files: example.py messages: 416331 nosy: davin, harsh8398, pitrou priority: normal severity: normal status: open title: multiprocessing.pool.Pool.apply block infinitely when stressed while using maxtasksperchild type: behavior versions: Python 3.7 Added file: https://bugs.python.org/file50706/example.py _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue47159> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
