I don't think it's due to the warmup of the JIT. Here's a simpler example.
import time import multiprocessing def do_nothing(): pass if __name__ == '__main__': time1 = time.time() do_nothing() time2 = time.time() pool = multiprocessing.Pool(processes=1) time3 = time.time() result = pool.apply_async(do_nothing) result.get() time4 = time.time() result = pool.apply_async(do_nothing) result.get() time5 = time.time() pool.close() print('not multiprocessing: ' + str(time2 - time1)) print('create pool: ' + str(time3 - time2)) print('run first time: ' + str(time4 - time3)) print('run second time: ' + str(time5 - time4)) Here are the results in PyPy. The first call to do_nothing() using multiprocessing.Pool takes 0.57 seconds. not multiprocessing: 0.0 create pool: 0.30999994278 run first time: 0.575999975204 run second time: 0.00100016593933 Here are the results in CPython. It also appears to be have some overhead the first time the pool is used, but it's less severe than PyPy. not multiprocessing: 0.0 create pool: 0.00500011444092 run first time: 0.134000062943 run second time: 0.0 On Fri, Sep 30, 2011 at 6:25 AM, Maciej Fijalkowski <fij...@gmail.com>wrote: > On Fri, Sep 30, 2011 at 10:20 AM, Armin Rigo <ar...@tunes.org> wrote: > > Hi, > > > > Is the conclusion just the fact that, again, the JIT's warm-up time is > > important, which we know very well? Or is there some other effect > > that cannot be explained just by that? (BTW, Laura, it's unrelated to > > multithreading if it's based on the multiprocessing module.) > > > > I guess what people didn't realize is that if you spawn a new process, > you have to warmup the JIT *again* for each of the worker (at least in > the worst case scenario). > > > > > A bientôt, > > > > Armin. > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev@python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > > >
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev