I'm using the third-party "processing" module in Python 2.5, which may have become the "multiprocessing" module in Python 2.6, to speed up the execution of a computation that takes over a week to run. The relevant code may not be relevant, but it is:
q1, q2 = processing.Queue(), processing.Queue() p1 = processing.Process(target=_findMaxMatch, args= (reciprocal, user, clusters[1:(numClusters - 1)/2], questions, copy.copy(maxMatch), q1)) p2 = processing.Process(target=_findMaxMatch, args= (reciprocal, user, clusters[(numClusters - 1)/2:], questions, copy.copy (maxMatch), q2)) p1.start() p2.start() maxMatch1 = q1.get()[0] maxMatch2 = q2.get()[0] p1.join() p2.join() if maxMatch1[1] > maxMatch2[1]: maxMatch = maxMatch1 else: maxMatch = maxMatch2 This code just splits up the calculation of the cluster that best matches 'user' into two for loops, each in its own process, rather than one. (It's not important what the cluster is.) The error I get is: [21661.903889] Out of memory: kill process 14888 (python) score 610654 or a child [21661.903930] Killed process 14888 (python) Traceback (most recent call last): ...etc. etc. ... Running this process from tty1, rather than GNOME, on my Ubuntu Hardy system allowed the execution to get a little further than under GNOME. The error was surprising because with just 1 GB of memory and a single for loop I didn't run into the error, but with 5 GB and two processes, I do. I believe that in the 1 GB case there was just a lot of painfully slow swapping going on that allowed it to continue. 'processing' appears to throw its hands up immediately, instead. Why does the program fail with 'processing' but not without it? Do you have any ideas for resolving the problem? Thanks for your help. -- http://mail.python.org/mailman/listinfo/python-list