On Tue, Apr 19, 2011 at 10:11 AM, Lou Pecora <lou_boog2...@yahoo.com> wrote:
> But as many of you probably know that raises a pickle error. I did a > little > research and don't fully understand, but I thought of a way around this > that is > pedestrian, but it works on my toy example. I just make (deep) copies of my > object and let a helper function call mainclass' aMethod. So I change the > code (after my class definition) to, > This is overkill. I don't understand the precise technical explanation for the error, but I think it just means that you can't pickle an object's method - only the object itself (assuming it doesn't have features that break pickling). So what I do is this: import multiprocessing as MP class mainclass (object) : def __call__ (self, x) : # do something return result myclass=mainclass() pool= MP.Pool(processes= 3) # 3 CPU workers, for example theargs= [x1,x2,x3] resultlist= pool.map(myclass, thargs) The only time I use a helper function like what you describe is when the iterable objects contain both the method of interest and the arguments, e.g.: class mainclass (object): def __init__ (self, x) : self.x = x def run (self) : return x**2 def run_many (obj) : return obj.run() args = [] for x in [13,29,71] : args.append(mainclass(x)) p = MP.Pool(processes=3) p.map(run_many, args) Which version to use is very case-dependent - my code has both. Further question, on an 8 CPU MacBook Pro I get a 4X speedup (with 8 CPU > workers), not 8X (Activity Monitor shows all 8 CPUs are utilized ~100%). > Is > there something about dual core processors that I should know about? (since > 4 = > 8/2). > The top-of-the-line MacBook Pro only has 4 cores - I think you're confusing Intel's "hyper-threading" with actual cores (not surprising, since software tends to confuse them too, e.g. Linux kernel). You are very unlikely to see a significant speedup using 8 processes versus 4 with the kind of crude parallelization implemented in the Python multiprocessing module. (I assume that lower-level threading tools like OpenMP will use hyperthreading more efficiently, but I've never tested this.) -Nat
_______________________________________________ Pythonmac-SIG maillist - Pythonmac-SIG@python.org http://mail.python.org/mailman/listinfo/pythonmac-sig unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG