> > > > > ---------- Weitergeleitete Nachricht ---------- > From: Dave Angel <d...@davea.name> > To: python-list@python.org > Cc: > Date: Thu, 10 Jan 2013 00:56:20 -0500 > Subject: Re: subprocess.Popen and multiprocessing fails to execute > external program > On 01/09/2013 11:08 PM, Niklas Berliner wrote: > > I have a pipline that involves processing some data, handing the data to > an > > external program (t_coffee used for sequence alignments in > bioinformatics), > > and postprocessing the result. Since I have a lot of data, I need to run > my > > pipeline in parallel which I implemented using the multiprocessing module > > following Doug Hellmanns blog ( > > http://blog.doughellmann.com/2009/04/pymotw-multiprocessing-part-1.html > ). > > > > My pipeline works perfectly fine when I run it with the multiprocessing > > implementation and one consumer, i.e. on one core. If I increase the > number > > of consumers, i.e. that multiple instances of my pipeline run in parallel > > the external program fails with a core dump. > > > > Could it be that the external program is not designed to have multiple > simultaneous instances? There are many such programs, some of which > check for an existing process before allowing another one to get far. > > When using the multiprocessing module, always make sure your externals > are well-behaved before looking for problems in your multi-code. > > To put it more strongly, a well-written program cannot easily be crashed > by the parent that launched it. > > > -- > > DaveA > >
Hi Dave, the developers of the external program said that they are using the program with multiple simultaneous instances. Also, when I execute multiple simultaneous instances of the external program using a bash wrapper script on my machine it works (the same wrapper scripts that fail when executed through python). Before asking here I have contacted the developers of the external program but they couldn't help me any further. Cheers, Niklas
-- http://mail.python.org/mailman/listinfo/python-list