[sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Mar 7, 9:47 pm, William Stein wst...@gmail.com wrote: With a function as *trivial* as your f above, the overhead of @parallel will kill your benchmark. As I explained, for *every* single call, an entire copy of Sage is forked off. This is no problem if evaluating f takes at least a second (say), but it kind of pointless for something like the above. -- William I have created a simple time-exhaustive function to compute some million elements FFT. Each call was taking approximately 5 secs. However I couldn't get speed-ups with @parallel on my dual core 2.5 Ghz cpu laptop. Could you provide an example showing the @parallel use with an easy-observable example? I am going to give a demonstration on Wednesday and I would like to be able to demonstrate this feature as well. Thanks again William. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
[sage-support] Re: Simple multiprocessing example in Sage-Notebook
I've tried with print pool.map(f, range(10)) instead of pool.map(f, range(10)) calling python file.py In the notebook, the output must be caughted and printed for user, I think. Pedro On Mar 7, 5:23 am, Gokhan Sever gokhanse...@gmail.com wrote: Hello, I am executing this example in Sage Notebook v4.3.3 from timeit import default_timer as clock from multiprocessing import Pool def f(x): return x**3 + x**2 + x if __name__ == '__main__': t1 = clock() pool = Pool(processes=2) # start 2 worker processes pool.map(f, range(100)); t2 = clock() print Elapsed time using two processes:, t2-t1 When I run the code in shell using python file.py I don't get any results from function mapping printed out however in Sage-Notebook I see a huge output with WARNING: Output truncated! Any ideas what might be causing this issue? Thanks. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
[sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Mar 7, 10:13 am, calcp...@aol.com wrote: from multiprocessing import Pool ... pool = Pool(processes=2) # start 2 worker processes Wow, cool, is this part of parallel python? Does this only work on a multi-core PC or can this be made to work over a cluster as well? This is standard library's multiprocessing library: http://docs.python.org/library/multiprocessing.html It comes with Python = 2.6 default. There are backports for 2.4 and 2.5 at http://pypi.python.org/pypi/multiprocessing/2.6.2.1 This module is for harnessing multiple cores. For cluster parallelism I would suggest you to give a try to IPython (http://ipython.scipy.org/ doc/stable/html/overview.html#interactive-parallel-computing) To me the multiprocessing module is the simplest approach to utilize the second lazy core in my laptop. (Intel Core 2 Duo 2.5G, 4GiB memory). I will use this example to demonstrate how quickly one can write code distributed on multiple cores in Python. In my local Sage v4.3.3 installation, in notebook using 1 million element list from timeit import default_timer as clock def f(x): return x**3 + x**2 + x t1 = clock() map(f, range(100)) t2 = clock() print Elapsed time using single process:, t2-t1 Elapsed time using single process: 4.46871089935 from timeit import default_timer as clock from multiprocessing import Pool def f(x): return x**3 + x**2 + x if __name__ == '__main__': t1 = clock() pool = Pool(processes=2)# start 2 worker processes dummy = pool.map(f, range(100)) t2 = clock() print Elapsed time using two processes:, t2-t1 Elapsed time using two processes: 3.15281605721 The gain is ~1.4x. In this example increasing the list length the ratio can go upto 1.7x. when I run the same test on sagenb.org Elapsed time using single process: 0.937326192856 Elapsed time using two processes: 0.970607995987 Heh single process execution takes less time :) If I try with 10 million element then I get this error. Traceback (click to the left of this block for traceback) ... MemoryError Maybe someone can give me a smarter example to demonstrate multiprocessing in Sage-notebook without failing on the main sagenb.org server. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
[sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Mar 7, 12:20 pm, jpc pedrocruzave...@gmail.com wrote: I've tried with print pool.map(f, range(10)) instead of pool.map(f, range(10)) calling python file.py In the notebook, the output must be caughted and printed for user, I think. Pedro I was trying with pool.map(f, range(10)); # supressing the output with semi-colon. Maybe it was another language that uses semi-colons to supress output not Python :) Anyways, Pat explained the situation. I might file a bug if I don't hear any other objections. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
Re: [sage-support] Re: Simple multiprocessing example in Sage-Notebook
pool sounds interesting! ipython is separate from parallel python? What about the @parallel decorator, is that parallel python? Its confusing in SAGE sometimes what needs importing and what doesn't. TIA, A. Jorge Garcia http://calcpage.tripod.com Teacher Professor Applied Mathematics, Physics Computer Science Baldwin Senior High School Nassau Community College -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
[sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Mar 7, 3:14 pm, calcp...@aol.com wrote: pool sounds interesting! ipython is separate from parallel python? What about the @parallel decorator, is that parallel python? Its confusing in SAGE sometimes what needs importing and what doesn't. TIA, A. Jorge Garciahttp://calcpage.tripod.com Teacher Professor Applied Mathematics, Physics Computer Science Baldwin Senior High School Nassau Community College Parallel Python is a separate module. IPython is the interactive Python interpreter that comes with Sage. However to use the IPython parallel features you need to install some additional packages. I am not sure if you could get it work from within Sage. More recommended way is to have an external installation. I don't know about @parallel yet. Please do tell me and give an example of its usage. I look at the Parallel Python examples and they are all lengthy. multiprocessing provides much simpler solution at a first sight. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
Re: [sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Sun, Mar 7, 2010 at 5:06 PM, Gokhan Sever gokhanse...@gmail.com wrote: Parallel Python is a separate module. IPython is the interactive Python interpreter that comes with Sage. However to use the IPython parallel features you need to install some additional packages. I am not sure if you could get it work from within Sage. More recommended way is to have an external installation. I don't know about @parallel yet. Please do tell me and give an example of its usage. I look at the Parallel Python examples and they are all lengthy. multiprocessing provides much simpler solution at a first sight. I wrote @parallel. Type sage: parallel? for some documentation and look at the source code in Sage. You just do @parallel(4) def f(n): # do some computation then for X in f([1..10]): print X and 4 copies of X will run at once. The actual implementation is *very* simple compared to multiprocessing or Parallel Python. It's about 2 pages of custom code I wrote from scratch just using pure Python and the fork system call (which is part of the os module). However, it has special support for forking Sage itself -- there are a number of issues having to do with pexpect interfaces, etc., which @parallel takes care of, but multiprocessing or Parallel python wouldn't know about. Another nice (imho) thing about @parallel is that it fork's a new process for each evaluation of the function f. This is good because (1) forking is cheap, (2) the entire *state* of your process is copied to the forked processes as is (e.g., you can have @parallel's inside of functions you define on the command line or notebook, etc.), (3) if the computation of f(n) leaks memory or segfaults, the calling program doesn't die. I use @parallel constantly for my research. -- William -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
[sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Mar 7, 8:45 pm, William Stein wst...@gmail.com wrote: On Sun, Mar 7, 2010 at 5:06 PM, Gokhan Sever gokhanse...@gmail.com wrote: Parallel Python is a separate module. IPython is the interactive Python interpreter that comes with Sage. However to use the IPython parallel features you need to install some additional packages. I am not sure if you could get it work from within Sage. More recommended way is to have an external installation. I don't know about @parallel yet. Please do tell me and give an example of its usage. I look at the Parallel Python examples and they are all lengthy. multiprocessing provides much simpler solution at a first sight. I wrote @parallel. Type sage: parallel? for some documentation and look at the source code in Sage. You just do @parallel(4) def f(n): # do some computation then for X in f([1..10]): print X and 4 copies of X will run at once. The actual implementation is *very* simple compared to multiprocessing or Parallel Python. It's about 2 pages of custom code I wrote from scratch just using pure Python and the fork system call (which is part of the os module). However, it has special support for forking Sage itself -- there are a number of issues having to do with pexpect interfaces, etc., which @parallel takes care of, but multiprocessing or Parallel python wouldn't know about. Another nice (imho) thing about @parallel is that it fork's a new process for each evaluation of the function f. This is good because (1) forking is cheap, (2) the entire *state* of your process is copied to the forked processes as is (e.g., you can have @parallel's inside of functions you define on the command line or notebook, etc.), (3) if the computation of f(n) leaks memory or segfaults, the calling program doesn't die. I use @parallel constantly for my research. -- William I see that @parallel provides a very quick solution for the problem and it is also the easiest to use. However consider my comparison (I am running on my local Sage v4.3.3 build): from timeit import default_timer as clock @parallel(2) def f(n): return n**3 + n**2 + n t1 = clock() map(f, range(100)) t2 = clock() print Elapsed time:, t2-t1 #Elapsed time: 7.36224603653 #Without @parallel; Elapsed time: 4.58867001534 Seems like something wrong. Could that be due to the map usage? from timeit import default_timer as clock def f(x): return x**3 + x**2 + x t1 = clock() map(f, range(100)) t2 = clock() print Elapsed time using single process:, t2-t1 #Elapsed time using single process: 4.68043208122 from timeit import default_timer as clock from multiprocessing import Pool def f(x): return x**3 + x**2 + x if __name__ == '__main__': t1 = clock() pool = Pool(processes=2)# start 2 worker processes dummy = pool.map(f, range(100)) t2 = clock() print Elapsed time using two processes:, t2-t1 # Elapsed time using two processes: 3.65069890022 With mp module it gets faster and threading works correctly --more processes more speed -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
Re: [sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Sun, Mar 7, 2010 at 7:40 PM, Gokhan Sever gokhanse...@gmail.com wrote: On Mar 7, 8:45 pm, William Stein wst...@gmail.com wrote: On Sun, Mar 7, 2010 at 5:06 PM, Gokhan Sever gokhanse...@gmail.com wrote: Parallel Python is a separate module. IPython is the interactive Python interpreter that comes with Sage. However to use the IPython parallel features you need to install some additional packages. I am not sure if you could get it work from within Sage. More recommended way is to have an external installation. I don't know about @parallel yet. Please do tell me and give an example of its usage. I look at the Parallel Python examples and they are all lengthy. multiprocessing provides much simpler solution at a first sight. I wrote @parallel. Type sage: parallel? for some documentation and look at the source code in Sage. You just do @parallel(4) def f(n): # do some computation then for X in f([1..10]): print X and 4 copies of X will run at once. The actual implementation is *very* simple compared to multiprocessing or Parallel Python. It's about 2 pages of custom code I wrote from scratch just using pure Python and the fork system call (which is part of the os module). However, it has special support for forking Sage itself -- there are a number of issues having to do with pexpect interfaces, etc., which @parallel takes care of, but multiprocessing or Parallel python wouldn't know about. Another nice (imho) thing about @parallel is that it fork's a new process for each evaluation of the function f. This is good because (1) forking is cheap, (2) the entire *state* of your process is copied to the forked processes as is (e.g., you can have @parallel's inside of functions you define on the command line or notebook, etc.), (3) if the computation of f(n) leaks memory or segfaults, the calling program doesn't die. I use @parallel constantly for my research. -- William I see that @parallel provides a very quick solution for the problem and it is also the easiest to use. However consider my comparison (I am running on my local Sage v4.3.3 build): from timeit import default_timer as clock @parallel(2) def f(n): return n**3 + n**2 + n t1 = clock() map(f, range(100)) t2 = clock() print Elapsed time:, t2-t1 #Elapsed time: 7.36224603653 #Without @parallel; Elapsed time: 4.58867001534 Seems like something wrong. Could that be due to the map usage? With a function as *trivial* as your f above, the overhead of @parallel will kill your benchmark. As I explained, for *every* single call, an entire copy of Sage is forked off. This is no problem if evaluating f takes at least a second (say), but it kind of pointless for something like the above. -- William from timeit import default_timer as clock def f(x): return x**3 + x**2 + x t1 = clock() map(f, range(100)) t2 = clock() print Elapsed time using single process:, t2-t1 #Elapsed time using single process: 4.68043208122 from timeit import default_timer as clock from multiprocessing import Pool def f(x): return x**3 + x**2 + x if __name__ == '__main__': t1 = clock() pool = Pool(processes=2) # start 2 worker processes dummy = pool.map(f, range(100)) t2 = clock() print Elapsed time using two processes:, t2-t1 # Elapsed time using two processes: 3.65069890022 With mp module it gets faster and threading works correctly --more processes more speed -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
[sage-support] Re: Simple multiprocessing example in Sage-Notebook
On Mar 6, 11:23 pm, Gokhan Sever gokhanse...@gmail.com wrote: Hello, I am executing this example in Sage Notebook v4.3.3 from timeit import default_timer as clock from multiprocessing import Pool def f(x): return x**3 + x**2 + x if __name__ == '__main__': t1 = clock() pool = Pool(processes=2) # start 2 worker processes pool.map(f, range(100)); t2 = clock() print Elapsed time using two processes:, t2-t1 When I run the code in shell using python file.py I don't get any results from function mapping printed out however in Sage-Notebook I see a huge output with WARNING: Output truncated! Any ideas what might be causing this issue? Thanks. if I do in Sage Notebook: foo = pool.map(f, range(100)) Then it works fine. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
Re: [sage-support] Re: Simple multiprocessing example in Sage-Notebook
On 03/06/2010 09:36 PM, Gokhan Sever wrote: On Mar 6, 11:23 pm, Gokhan Sever gokhanse...@gmail.com wrote: When I run the code in shell using python file.py I don't get any results from function mapping printed out however in Sage-Notebook I see a huge output with WARNING: Output truncated! Any ideas what might be causing this issue? Thanks. if I do in Sage Notebook: foo = pool.map(f, range(100)) Then it works fine. This *may* be a consequence of sagenb.misc.format.displayhook_hack which formats a cell's input so that when exec'd last line is printed if it is an expression. By the way. It seems that both sage: import sagenb.misc.format sage: sagenb.misc.format.displayhook_hack? sage: sagenb.misc.format.displayhook_hack?? don't render properly in the on the command-line. The latter does in the notebook. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
