On Sun, Mar 7, 2010 at 7:40 PM, Gokhan Sever <gokhanse...@gmail.com> wrote: > > > On Mar 7, 8:45 pm, William Stein <wst...@gmail.com> wrote: >> On Sun, Mar 7, 2010 at 5:06 PM, Gokhan Sever <gokhanse...@gmail.com> wrote: >> > Parallel Python is a separate module. IPython is the interactive >> > Python interpreter that comes with Sage. However to use the IPython >> > parallel features you need to install some additional packages. I am >> > not sure if you could get it work from within Sage. More recommended >> > way is to have an external installation. >> >> > I don't know about @parallel yet. Please do tell me and give an >> > example of its usage. I look at the Parallel Python examples and they >> > are all lengthy. multiprocessing provides much simpler solution at a >> > first sight. >> >> I wrote @parallel. Type >> >> sage: parallel? >> >> for some documentation and look at the source code in Sage. >> You just do >> >> @parallel(4) >> def f(n): >> # do some computation >> >> then >> >> for X in f([1..10]): >> print X >> >> and 4 copies of X will run at once. >> >> The actual implementation is *very* simple compared to multiprocessing >> or Parallel Python. It's about 2 pages of custom code I wrote from >> scratch just using pure Python and the fork system call (which is part >> of the os module). However, it has special support for forking Sage >> itself -- there are a number of issues having to do with pexpect >> interfaces, etc., which @parallel takes care of, but multiprocessing >> or Parallel python wouldn't know about. >> >> Another nice (imho) thing about @parallel is that it fork's a new >> process for each evaluation of the function f. This is good because >> (1) forking is cheap, (2) the entire *state* of your process is copied >> to the forked processes as is (e.g., you can have @parallel's inside >> of functions you define on the command line or notebook, etc.), (3) if >> the computation of f(n) leaks memory or segfaults, the calling program >> doesn't die. >> >> I use @parallel constantly for my research. >> >> -- William > > I see that @parallel provides a very quick solution for the problem > and it is also the easiest to use. However consider my comparison (I > am running on my local Sage v4.3.3 build): > > > from timeit import default_timer as clock > > @parallel(2) > def f(n): > return n**3 + n**2 + n > > t1 = clock() > map(f, range(1000000)) > t2 = clock() > print "Elapsed time:", t2-t1 > > #Elapsed time: 7.36224603653 > #Without @parallel; Elapsed time: 4.58867001534 > > Seems like something wrong. Could that be due to the map usage?
With a function as *trivial* as your f above, the overhead of @parallel will kill your benchmark. As I explained, for *every* single call, an entire copy of Sage is forked off. This is no problem if evaluating f takes at least a second (say), but it kind of pointless for something like the above. -- William > > from timeit import default_timer as clock > > def f(x): > return x**3 + x**2 + x > > t1 = clock() > map(f, range(1000000)) > t2 = clock() > print "Elapsed time using single process:", t2-t1 > > #Elapsed time using single process: 4.68043208122 > > > from timeit import default_timer as clock > from multiprocessing import Pool > > def f(x): > return x**3 + x**2 + x > > if __name__ == '__main__': > t1 = clock() > pool = Pool(processes=2) # start 2 worker processes > dummy = pool.map(f, range(1000000)) > t2 = clock() > print "Elapsed time using two processes:", t2-t1 > > # Elapsed time using two processes: 3.65069890022 > > With mp module it gets faster and threading works correctly --more > processes more speed > > > > -- > To post to this group, send email to sage-support@googlegroups.com > To unsubscribe from this group, send email to > sage-support+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-support > URL: http://www.sagemath.org > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
