New submission from STINNER Victor: Since Python 3.5, distutils is able to build extensions in parallel, nice enhancement! But setup.py of CPython is 2x slower in parallel mode when all modules are already built: (1 sec vs 500 ms).
Building extensions calls newer_group() which calls os.stat() 6,856 times. I wrote a cache for os.stat() but it has no impact on performance. It seems like threads are fighting to death for the GIL in the os.stat() race... Attached pull request calls newer_group() before spawning threads in parallel mode, so "setup.py build" takes the same time with and without parallel module, when all extensions are already built. I didn't measure performance when all extensions must be built. ---------- components: Distutils messages: 292530 nosy: dstufft, haypo, merwok priority: normal severity: normal status: open title: distutils build_ext: don't run newer_group() in parallel in multiple threads when using parallel type: performance versions: Python 3.7 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30198> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
