Sorry about that, here is a summary of my complete code. I haven't cleaned it up much or anything, but this is what it does:
import time import multiprocessing test_constx =0 test_consty =0 def functionTester(x): global test_constx global test_consty print "constx " + str(test_constx) print "consty " + str(test_consty) return (test_constx*x[0]-x[1]+test_consty*x[0]+x[2]) def functionTesterMain(constx,consty): global test_constx global test_consty test_constx = constx test_consty = consty num_args = [(61,12,1),(61,12,2),(61,12,3),(61,11,4),(61,12,4),(62,33,4),(7,12,4),(16,19,4),(35,36,4),(37,38,3),(55,56,3),(57,63,3)] num_processes = multiprocessing.cpu_count() pool = multiprocessing.Pool(num_processes) rs = [] start = time.time() rs = pool.map(functionTester,num_args) end = time.time() elapsed= end - start min = elapsed/60 print "Took", elapsed, "seconds to run, which is the same as", min, "minutes" pos = 0 high = 0 n = None for r in rs: if r > high: n = num_args[pos] high = r pos+=1 print "high " + str(high) print "n " + str(n) return high,n if __name__ == '__main__': for i in range(1,4): a,b = functionTesterMain(i,7) print "-----------" print "a " + str(a) print "b " + str(a) Which doesn't seem to work because the functionTester() needs to be simpler and not use global variables. I'm using global variables because I'm also trying to pass a few other variables and I tried using a class but that just gave me a unpickleable error. I tried using zip but I'm confused with how I can get to pass the data. I know I can probably combine the data into tuples but that means that there is alot of data duplication, especially if the constx and consty are large dictionaries (or even custom objects), which might happen later. So it seems that map doesn't quite like functions like these. Anyway, I'll try and see if threads or something can substitute. I'd appriciate any help. Thanks On Sat, Dec 11, 2010 at 9:04 AM, geremy condra <debat...@gmail.com> wrote: > On Fri, Dec 10, 2010 at 4:42 AM, Astan Chee <astan.c...@gmail.com> wrote: > > On Fri, Dec 10, 2010 at 1:37 AM, Peter Otten <__pete...@web.de> wrote: > >> > >> I can't replicate the crash. However, your problem looks like there is a > >> ready-to-use solution: > >> > >> > >> > http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers > >> > >> -- > >> http://mail.python.org/mailman/listinfo/python-list > > > > > > Pool.map doesn't seem to be able to support multiple argument functions > > which is what I'm trying to do here. Any other suggestions? > > Thanks again > > 1. Post the real code you're using, and > 2. Put the arguments you want in a tuple and pass that. As an example, > let's say I have the following function: > > def my_func(x, y, z): > return x + y * z > > you could rewrite this as: > > def my_func(*args): > return args[0] + args[1] * args[2] > > > Here's a worked-out example: > > #! /usr/bin/env python3 > > import multiprocessing > > def my_func(x, y, z): > return x + y * z > > def my_func_wrapper(t): > return my_func(*t) > > # assume we can get an iterable over each argument > xs = [1, 2, 3, 4] > ys = [5, 6, 7, 8] > zs = [9, 1, 2, 3] > > # set up the pool to match the number of CPUs > num_processes = multiprocessing.cpu_count() > pool = multiprocessing.Pool(num_processes) > > #will call my_func(1, 5, 9), my_func(2, 6, 1), etc. > results = pool.map(my_func_wrapper, zip(xs, ys, zs)) > print(results) > > > Interesting factoid: you can't seem to use lambda or a decorator to do > this, which would have been my first instinct. Pickle apparently > chokes, although marshall wouldn't. > > Geremy Condra >
-- http://mail.python.org/mailman/listinfo/python-list