I don't think it's due to the warmup of the JIT. Here's a simpler example.
import time
import multiprocessing
def do_nothing(): pass
if __name__ == '__main__':
time1 = time.time()
do_nothing()
time2 = time.time()
pool = multiprocessing.Pool(processes=1)
time3 = time.time()
result = pool.apply_async(do_nothing)
result.get()
time4 = time.time()
result = pool.apply_async(do_nothing)
result.get()
time5 = time.time()
pool.close()
print('not multiprocessing: ' + str(time2 - time1))
print('create pool: ' + str(time3 - time2))
print('run first time: ' + str(time4 - time3))
print('run second time: ' + str(time5 - time4))
Here are the results in PyPy. The first call to do_nothing() using
multiprocessing.Pool takes 0.57 seconds.
not multiprocessing: 0.0
create pool: 0.30999994278
run first time: 0.575999975204
run second time: 0.00100016593933
Here are the results in CPython. It also appears to be have some overhead
the first time the pool is used, but it's less severe than PyPy.
not multiprocessing: 0.0
create pool: 0.00500011444092
run first time: 0.134000062943
run second time: 0.0
On Fri, Sep 30, 2011 at 6:25 AM, Maciej Fijalkowski <fij...@gmail.com>wrote:
> On Fri, Sep 30, 2011 at 10:20 AM, Armin Rigo <ar...@tunes.org> wrote:
> > Hi,
> >
> > Is the conclusion just the fact that, again, the JIT's warm-up time is
> > important, which we know very well? Or is there some other effect
> > that cannot be explained just by that? (BTW, Laura, it's unrelated to
> > multithreading if it's based on the multiprocessing module.)
> >
>
> I guess what people didn't realize is that if you spawn a new process,
> you have to warmup the JIT *again* for each of the worker (at least in
> the worst case scenario).
>
> >
> > A bientôt,
> >
> > Armin.
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev@python.org
> > http://mail.python.org/mailman/listinfo/pypy-dev
> >
>
_______________________________________________
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev
