[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: What can I do to move this forward? I believe all concerns have been addressed and it seems ready to me. -- ___ Python tracker ___ __

[issue16104] Use multiprocessing in compileall script

Changes by Claudiu.Popa : Added file: http://bugs.python.org/file34404/issue16104_7.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: Here's a new version which catches ImportError for concurrent.futures and raises ValueError in `compile_dir` if `processes` was specified and concurrent.futures is unavailable. The only issue is that I don't know if this should be a ValueError or not. For instan

[issue16104] Use multiprocessing in compileall script

Antoine Pitrou added the comment: > Here's a new patch which addresses Éric's last comments. > Antoine, I don't have at my disposal a system without multiprocessing > support. How does it crash? Neither do I, but you will probably get an ImportError of some sort. -- __

[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: Here's a new patch which addresses Éric's last comments. Antoine, I don't have at my disposal a system without multiprocessing support. How does it crash? -- Added file: http://bugs.python.org/file34400/issue16104_5.patch

[issue16104] Use multiprocessing in compileall script

Antoine Pitrou added the comment: Importing ProcessExecutor at the top-level means compileall will crash on systems which don't have multiprocessing support. -- nosy: +pitrou ___ Python tracker ___

[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: Here's a test for j0 == os.cpu_count. -- Added file: http://bugs.python.org/file34384/issue16104_4.patch ___ Python tracker ___ __

[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: regrtest does that, checking for j <=0. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Un

[issue16104] Use multiprocessing in compileall script

Éric Araujo added the comment: +if args.processes <= 0: Is that correct? For make, I think I’ve always seen “-j0”, not negative values. Could you add a test for -j0? (i.e. check that “compileall -j0” calls the function with “processes=os.cpu_count()”) -- _

[issue16104] Use multiprocessing in compileall script

Changes by Claudiu.Popa : Added file: http://bugs.python.org/file34383/issue16104_3.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: I agree. I'll modify the patch. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue16104] Use multiprocessing in compileall script

Brett Cannon added the comment: I vote for changed so that -j0 uses all available cores as os.cpu_count() states. -- ___ Python tracker ___ _

[issue16104] Use multiprocessing in compileall script

Éric Araujo added the comment: This looks ready to me. One thing: “make -j0” is the spelling for “run using all available cores”, whereas “compileall -j0” will use one process. I don’t know if this should be documented, changed or ignored. -- stage: patch review -> commit review ___

[issue16104] Use multiprocessing in compileall script

Changes by Claudiu.Popa : Added file: http://bugs.python.org/file34381/issue16104_2.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue16104] Use multiprocessing in compileall script

Éric Araujo added the comment: FTR, py_compile and compileall use importlib in 3.4. -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: Thank you for the review, Éric! Here's the updated patch. -- Added file: http://bugs.python.org/file34368/issue16104_1.patch ___ Python tracker __

[issue16104] Use multiprocessing in compileall script

Éric Araujo added the comment: Patch looks good. Some comments on Rietveld. -- nosy: +eric.araujo ___ Python tracker ___ ___ Python-b

[issue16104] Use multiprocessing in compileall script

Changes by Éric Araujo : -- stage: -> patch review type: -> enhancement versions: +Python 3.5 -Python 3.4 ___ Python tracker ___ ___

[issue16104] Use multiprocessing in compileall script

Changes by Claudiu.Popa : Added file: http://bugs.python.org/file34339/issue16104.patch ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue16104] Use multiprocessing in compileall script

Claudiu.Popa added the comment: Hello! Here's a draft patch. It adds a new *processes* parameter to *compile_dir* and a new command line parameter as well. -- keywords: +patch nosy: +Claudiu.Popa Added file: http://bugs.python.org/file33079/compileall_v1.patch

[issue16104] Use multiprocessing in compileall script

Changes by Brett Cannon : -- assignee: brett.cannon -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://

[issue16104] Use multiprocessing in compileall script

Changes by Brett Cannon : -- assignee: -> brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://

[issue16104] Use multiprocessing in compileall script

Changes by Steven D'Aprano : -- nosy: +stevenjd ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyt

[issue16104] Use multiprocessing in compileall script

Brett Cannon added the comment: This should probably use concurrent.futures instead of multiprocessing directly, but yes it would be useful. Then again, the whole module should probably be rewritten to use importlib as well. -- components: +Library (Lib) nosy: +brett.cannon priority:

[issue16104] Use multiprocessing in compileall script

New submission from Daniel Holth: compileall would benefit approximately linearly from additional CPU cores. There should be an option. The noisy output would have to change. Right now it prints "compiling" and then "done" synchronously with doing the actual work. -- messages: 171744