On Sat, Aug 27, 2011 at 2:59 AM, Ask Solem <a...@celeryproject.org> wrote:
> > On 26 Aug 2011, at 16:53, Antoine Pitrou wrote: > > > > > Hi, > > > >> I think that "deprecating" the use of threads w/ multiprocessing - or > >> at least crippling it is the wrong answer. Multiprocessing needs the > >> helper threads it uses internally to manage queues, etc. Removing that > >> ability would require a near-total rewrite, which is just a > >> non-starter. > > > > I agree that this wouldn't actually benefit anyone. > > Besides, I don't think it's even possible to avoid threads in > > multiprocessing, given the various constraints. We would have to force > > the user to run their main thread in an event loop, and that would be > > twisted (tm). > > > >> I would focus on the atfork() patch more directly, ignoring > >> multiprocessing in the discussion, and focusing on the merits of gps' > >> initial proposal and patch. > > > > I think this could also be combined with Charles-François' patch. > > > > Regards > > > > Have to agree with Jesse and Antoine here. > > Celery (celeryproject.org) uses multiprocessing, is wildly used in > production, > and is regarded as stable software that have been known to run for months > at a time > only to be restarted for software upgrades. > > I have been investigating an issue for some time, that I'm pretty sure is > caused > by this. It occurs only rarely, so rarely I have not had any actual bug > reports > about it, it's just something I have experienced during extensive testing. > The tone of the discussion on the bug tracker makes me think that I have > been very lucky :-) > > Using the fork+exec approach seems like a much more realistic solution > than rewriting multiprocessing.Pool and Manager to not use threads. In fact > this is something I have been considering as a fix for the suspected > issue for for some time. > It does have implications that are annoying for sure, but we are already > used to this on the Windows platform (it could help portability even). > +3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described "non-fork" implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. -gps
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
