[issue14148] Option to kill stuck workers in a multiprocessing pool

2012-03-07 Thread Brian Quinlan
Changes by Brian Quinlan br...@sweetapp.com: -- nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14148 ___ ___ Python-bugs-list

[issue14148] Option to kill stuck workers in a multiprocessing pool

2012-02-28 Thread Paul Moore
New submission from Paul Moore p.f.mo...@gmail.com: I have an application which fires off a number of database connections via a multiprocessing pool. Unfortunately, the database software occasionally gets stuck and a connection request hangs indefinitely. This locks up the whole process

[issue14148] Option to kill stuck workers in a multiprocessing pool

2012-02-28 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: The problem is that queues and other synchronization objects can end up in an inconsistent state when a worker crashes, hangs or gets killed. That's why, in concurrent.futures, a crashed worker makes the ProcessPoolExecutor become broken. A

[issue14148] Option to kill stuck workers in a multiprocessing pool

2012-02-28 Thread Paul Moore
Paul Moore p.f.mo...@gmail.com added the comment: As an alternative, maybe leave the stuck worker, but allow the pool to recognise when a worker has not processed new messages for a long period and spawn an extra worker to replace it. That would avoid the starvation issue, and the stuck workers