On Thu, Mar 19, 2009 at 10:29 AM, Brian Granger <ellisonbg....@gmail.com> wrote: > > Yes, we do have built-in dynamic load balancing. Check out the following: > > client.TaskClient > client.StringTask > client.MapTask > > All of this is dynamically load balanced. Here is how I would approach this: > > 1. Use MultiEngineClient to setup all the basic imports and variables > that all the engines need. > > 2. Create a TaskClient and then submit many StringTasks/MapTasks to > the TaskClient. > > The only thing to note is that currently, the TaskClient has slightly > more overhead than the MultiEngineClient, so you want to make sure > that each tasks lasts long enough to be worth it. If your tasks are > really short and you are not seeing good speedup, you may want each > StringTask/MapTask instance handle a small batch of actual tasks. > > But all of this should "Just Work" > > Right now the best documentation for this is in the doc strings: > > http://ipython.scipy.org/doc/nightly/html/api/generated/IPython.kernel.taskclient.html > > http://ipython.scipy.org/doc/nightly/html/api/generated/IPython.kernel.task.html#IPython.kernel.task.StringTask > > http://ipython.scipy.org/doc/nightly/html/api/generated > /IPython.kernel.task.html#IPython.kernel.task.MapTask > > If these task types don't work for you, it is also possible to define > new task types.
Thanks Brian, that should do it. Yes, we have ~1400 of small tests, so I can group them in larger groups as you suggested. One more beginner question, as I am new to cluster computing. :) After I install all the necessary dependencies on our UNR cluster, and then submit a job that executes ipcluster (using mpirun), then I will have an ipcluster running. In order to connect to it, I can submit another job that will be my script (e.g. it imports ipython and all should be fine). But if I wanted to interactively deal with the ipcluster from the ipython session --- do you think there is some way? Because that would really rock. I guess it depends on how the cluster is built, which I don't know until I try it. I thought that maybe if I run ipython on the master node that it could interact with the ipcluster --- but maybe this is forbidden, in which case it will not be possible to interact with it. When you run ipython on some big cluster, are you able to actually interact with it? Ondrej --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sympy" group. To post to this group, send email to sympy@googlegroups.com To unsubscribe from this group, send email to sympy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sympy?hl=en -~----------~----~----~----~------~----~------~--~---
