[issue3448] Multi-process 2to3

Benjamin Peterson added the comment: I added some support for concurrent running in r70999. -- resolution: -> accepted status: open -> closed ___ Python tracker ___

[issue3448] Multi-process 2to3

Nick Edds <[EMAIL PROTECTED]> added the comment: I had very little experience with the processing module prior to the creation of this patch, and because pool objects are listed last in the documentation, I did not read about them because I saw a way to achieve what I wanted using Process. But ha

[issue3448] Multi-process 2to3

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: I'm not opposed to having the support available. I just don't what it enabled by default. ___ Python tracker <[EMAIL PROTECTED]> ___

[issue3448] Multi-process 2to3

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: I suggest that when using lib2to3 as a library, multiprocessing is not enabled by default; there may be uses of the library that are incompatible with multiprocessing. It may be enabled by default when using it from the command line (or

[issue3448] Multi-process 2to3

Collin Winter <[EMAIL PROTECTED]> added the comment: Benjamin, what "complexity" did you encounter when trying to use lib2to3 in your own work? Unless there's a concrete use-case where the mere existence of multiprocessing support (as opposed to actually enabling that support) made a tangible pro

[issue3448] Multi-process 2to3

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: I think it's reasonable to only enable multiprocessing if the adequate command-line option has been set. It's how `make` already works (next time you compile Python, try `make -jN` where N is your number of CPU cores). -- nosy: +pitro

[issue3448] Multi-process 2to3

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: Because I would like to use lib2to3 without the complexity of multiple processes. That it is a good performance boost is excellent, but I don't think it should be a required part of using the library. __

[issue3448] Multi-process 2to3

Collin Winter <[EMAIL PROTECTED]> added the comment: You have yet to articulate a reason for that preference. ___ Python tracker <[EMAIL PROTECTED]> ___

[issue3448] Multi-process 2to3

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: I'm just saying that a client of lib2to3 shouldn't have to use multiple processes. Just as long as the multiple processes are optional, I'm happy. :) ___ Python tracker <[EMAIL PROTECTED]>

[issue3448] Multi-process 2to3

Collin Winter <[EMAIL PROTECTED]> added the comment: "I don't think we should force people using it as a library to go multiprocess." I don't understand this. What downsides do you perceive in multiprocessing support? Multiprocessing is a significant speed-up for 2to3 on multicore systems.

[issue3448] Multi-process 2to3

Nick Edds <[EMAIL PROTECTED]> added the comment: The currently attached patch works in Python2.5 not Python2.6, so I will update it for 2.6 when I get the chance. But as it is currently written, the default behavior is not multiprocess. Instead, if you want multiprocess, you specify how many proc

[issue3448] Multi-process 2to3

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: I don't think we should force people using it as a library to go multiprocess. Also, it's trivial to just change the name of the class used if that is wanted. ___ Python tracker <[EMAIL PROTECTED]>

[issue3448] Multi-process 2to3

Collin Winter <[EMAIL PROTECTED]> added the comment: I believe the only issue I recall was that the patch didn't work out-of-the-box for Python 2.6 (changed imports, PEP 8 compliance changes in the multiprocess module). Has that been fixed? I disagree with Benjamin: this is an import speed incre

[issue3448] Multi-process 2to3

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: Nick, is there a way you could isolate the process functionality in a RefactoringTool subclass? It's an interesting idea, but I don't it needs to infect the main library. -- nosy: +benjamin.peterson ___

[issue3448] Multi-process 2to3

Nick Edds <[EMAIL PROTECTED]> added the comment: Is there still any interest in this Collin? Is there anything else you need me to do for it? ___ Python tracker <[EMAIL PROTECTED]> ___

[issue3448] Multi-process 2to3

Nick Edds <[EMAIL PROTECTED]> added the comment: Here is a version that only imports processing if the multi-process option is specified. I don't know if this is the most efficient way it can be done, and I think there's a better way to do it, but this works. Added file: http://bugs.python.org/f

[issue3448] Multi-process 2to3

New submission from Nick Edds <[EMAIL PROTECTED]>: Here is a working, multiprocess version of 2to3 with a few caveats. First, you need to already have the processing module installed for this to work. If we don't want to include processing in some way, I think I can modify this to only import pro