Hi everyone,
I just parallelized a for loop in a way that may be generally useful, and
I'm wondering whether I should write an addition to sage.parallel. It's
possible that this has been done already and in much better ways than my
own, so I'd be very happy to hear your opinions.
I had a for loop of the form
for i in iterator:
# do some stuff
# eventually result <- f(i)
yield result
for some "long" iterator and a "short" computation i -> f(i).
I tried to parallelize this using @parallel, but the result wasn't any
quicker:
@parallel()
def f(i):
# do some stuff
return result
for res in f(iterator):
yield res[1]
My guess about the lack of speed-up is that every iteration introduces
pickling and forking overhead, which adds up to a lot, collectively.
I tried to mitigate this overhead by not forking for every iteration, but
by starting a fork for every cpu and then having that fork loop over its
own slice of the iterator:
@parallel('fork')
def do_computation(start):
result = []
for args in itertools.islice(iterator, start, None, number_of_chunks):
result.append(function(args))
return result
results = do_computation(range(number_of_chunks))
for res in results:
for r in res[1]:
yield r
This resulted in the expected speedup (~3x for ncpus=4).
I collected this functionality in a decorator @unordered_parallel that lets
you replace
for i in iterator:
yield f(i)
by
@unordered_parallel(iterator, number_of_chunks)
def _(i):
return f(i)
for res in _: yield res
(As a side note: I'm really happy with how the interface lets you
parallelize your
for loop while hardly changing its body.)
Would there be any interest in an addition along these lines to the
sage.parallel module?
Best, Timo
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.
