geremy condra wrote: > On Fri, Dec 10, 2010 at 4:42 AM, Astan Chee <astan.c...@gmail.com> wrote: >> On Fri, Dec 10, 2010 at 1:37 AM, Peter Otten <__pete...@web.de> wrote: >>> >>> I can't replicate the crash. However, your problem looks like there is a >>> ready-to-use solution: >>> >>> >>> http://docs.python.org/library/multiprocessing.html#using-a-pool-of- workers >>> >>> -- >>> http://mail.python.org/mailman/listinfo/python-list >> >> >> Pool.map doesn't seem to be able to support multiple argument functions >> which is what I'm trying to do here. Any other suggestions? >> Thanks again > > 1. Post the real code you're using, and > 2. Put the arguments you want in a tuple and pass that. As an example, > let's say I have the following function: > > def my_func(x, y, z): > return x + y * z > > you could rewrite this as: > > def my_func(*args): > return args[0] + args[1] * args[2] > > > Here's a worked-out example: > > #! /usr/bin/env python3 > > import multiprocessing > > def my_func(x, y, z): > return x + y * z > > def my_func_wrapper(t): > return my_func(*t) > > # assume we can get an iterable over each argument > xs = [1, 2, 3, 4] > ys = [5, 6, 7, 8] > zs = [9, 1, 2, 3] > > # set up the pool to match the number of CPUs > num_processes = multiprocessing.cpu_count() > pool = multiprocessing.Pool(num_processes) > > #will call my_func(1, 5, 9), my_func(2, 6, 1), etc. > results = pool.map(my_func_wrapper, zip(xs, ys, zs)) > print(results) > > > Interesting factoid: you can't seem to use lambda or a decorator to do > this, which would have been my first instinct. Pickle apparently > chokes, although marshall wouldn't.
You basically have to ensure that the resulting function is found under its __name__ in the global namespace of its __module__. This can be done with functools.wraps(): #! /usr/bin/env python3 import multiprocessing import functools def starcall(f): @functools.wraps(f) def g(args): return f(*args) return g @starcall def my_func(x, y, z): return x + y * z xs = [1, 2, 3, 4] ys = [5, 6, 7, 8] zs = [9, 1, 2, 3] num_processes = multiprocessing.cpu_count() pool = multiprocessing.Pool(num_processes) results = pool.map(my_func, zip(xs, ys, zs)) print(results) -- http://mail.python.org/mailman/listinfo/python-list