[issue11635] concurrent.futures uses polling
Changes by s7v7nislands s7v7nisla...@gmail.com: -- nosy: +s7v7nislands ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Roundup Robot devnull@devnull added the comment: New changeset 4390d6939a56 by Antoine Pitrou in branch '3.2': Issue #11635: Don't use polling in worker threads and processes launched by http://hg.python.org/cpython/rev/4390d6939a56 New changeset a76257a99636 by Antoine Pitrou in branch 'default': Issue #11635: Don't use polling in worker threads and processes launched by http://hg.python.org/cpython/rev/a76257a99636 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: I've now pushed the patch. I hope this won't break anything, closing. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Roundup Robot devnull@devnull added the comment: New changeset 76a898433a02 by Antoine Pitrou in branch '3.2': Add tests for the atexit hook in concurrent.futures (part of #11635) http://hg.python.org/cpython/rev/76a898433a02 New changeset d6bbde982c1c by Antoine Pitrou in branch 'default': Add tests for the atexit hook in concurrent.futures (part of #11635) http://hg.python.org/cpython/rev/d6bbde982c1c -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: Tests now committed, here is a patch without them. -- Added file: http://bugs.python.org/file21373/cfpolling4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: After studying the multiprocessing code, it turns out that Queue.get() with a timeout does its own rather high-frequency polling under Windows (see Modules/_multiprocessing/pipe_connection.c). Therefore, here is an updated patch which doesn't have a security timeout at all. -- Added file: http://bugs.python.org/file21378/cfpolling5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Brian Quinlan br...@sweetapp.com added the comment: Your approach seems workable but your patch allows the interpreter to exit while work items are still being processed. See the comment at the top of concurrent/futures/thread.py. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: Your approach seems workable but your patch allows the interpreter to exit while work items are still being processed. See the comment at the top of concurrent/futures/thread.py. Why are you saying that? In my patch, _python_exit() still takes care of joining worker threads. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Brian Quinlan br...@sweetapp.com added the comment: Sorry, I didn't read an error message very carefully. When I apply your patch I see: from concurrent.futures import * from time import * t = ThreadPoolExecutor(5) t.submit(sleep, 100) Future at 0x8acc94c state=running ctrl-D Error in atexit._run_exitfuncs: NameError: global name 'thread' is not defined Does that not happen in your environment? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: Sorry, I didn't read an error message very carefully. When I apply your patch I see: from concurrent.futures import * from time import * t = ThreadPoolExecutor(5) t.submit(sleep, 100) Future at 0x8acc94c state=running ctrl-D Error in atexit._run_exitfuncs: NameError: global name 'thread' is not defined Does that not happen in your environment? Ah, thank you, it does happen here too. I should fix the error (a typo) and add tests for this too. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: Ok, here is a new patch with an additional test for the atexit hook. If you don't object, I would like to start committing the test changes, and then the code changes themselves. -- stage: needs patch - patch review versions: +Python 3.2 Added file: http://bugs.python.org/file21360/cfpolling3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: Oops, test didn't work under Windows. Here is a new patch. -- Added file: http://bugs.python.org/file21361/cfpolling3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file21360/cfpolling3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
New submission from Antoine Pitrou pit...@free.fr: concurrent.futures uses polling in its worker threads and processes (with a timeout of 0.1). It means that: 1) this prevents CPUs to enter low power states durably 2) it incurs latency when calling shutdown() on an executor (this seems to be the main source of slowness in test_concurrent_futures under Linux) -- components: Library (Lib) messages: 131756 nosy: bquinlan, gregory.p.smith, jyasskin, pitrou priority: normal severity: normal stage: needs patch status: open title: concurrent.futures uses polling type: resource usage versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Changes by Daniel Stutzbach stutzb...@google.com: -- nosy: +stutzbach ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: Ok, here is a patch. Summary: - leave a minimal amount of polling (every 600 seconds) to avoid blocking forever if there's a bug (shouldn't happen of course, but who knows? especially with multiprocessing) - when wanting to wakeup a worker, put None in its receiving queue - remove periodic cleanup of thread references by using a weak dict instead - in tests, compute runtime and make the test fail if the runtime exceeds 60 seconds (to report aforementioned synchronization bugs) Tested under Linux (extensively) and Windows 7 (briefly). -- keywords: +patch Added file: http://bugs.python.org/file21351/cfpolling2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: (patch is for 3.2, by the way. Perhaps this should only be fixed in default?) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Brian Quinlan br...@sweetapp.com added the comment: I would suggest that you base your patch on 3.3/default. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Antoine Pitrou pit...@free.fr added the comment: I would suggest that you base your patch on 3.3/default. Well, since the module is new, I think it would be nice to fix such quirks in the bugfix branch as well. So, following the recommended workflow, I've started with a 3.2 patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
