ma3mju wrote:
On 3 Aug, 09:36, ma3mju <matt.u...@googlemail.com> wrote:
On 2 Aug, 21:49, Piet van Oostrum <p...@cs.uu.nl> wrote:
MRAB <pyt...@mrabarnett.plus.com> (M) wrote:
M> I wonder whether one of the workers is raising an exception, perhaps due
M> to lack of memory, when there are large number of jobs to process.
But that wouldn't prevent the join. And you would probably get an
exception traceback printed.
I wonder if something fishy is happening in the multiprocessing
infrastructure. Or maybe the Fortran code goes wrong because it has no
protection against buffer overruns and similar problems, I think.
--
Piet van Oostrum <p...@cs.uu.nl>
URL:http://pietvanoostrum.com[PGP8DAE142BE17999C4]
Private email: p...@vanoostrum.org
I don't think it's a memory problem, the reason for the hard and easy
queue is because for larger examples it uses far more RAM. If I run
all of workers with harder problems I do begin to run out of RAM and
end up spending all my time switching in and out of swap so I limit
the number of harder problems I run at the same time. I've watched it
run to the end (a very boring couple of hours) and it stays out of my
swap space and everything appears to be staying in RAM. Just hangs
after all "poison" has been printed for each process.
The other thing is that I get the message "here" telling me I broke
out of the loop after seeing the poison pill in the process and I get
all the things queued listed as output surely if I were to run out of
memory I wouldn't expect all of the jobs to be listed as output.
I have a serial script that works fine so I know individually for each
example the fortran code works.
Thanks
Matt
Any ideas for a solution?
A workaround is to do them in small batches.
You could put each job in a queue with a flag to say whether it's hard
or easy, then:
while have more jobs:
move up to BATCH_SIZE jobs into worker queues
create and start workers
wait for workers to finish
discard workers
--
http://mail.python.org/mailman/listinfo/python-list
