There are some great comments in this thread. Process migration (like many topics in systems) can get complex fast.
The Open MPI process migration implementation is checkpoint/restart based (currently using BLCR), and uses an 'eager' style of migration. This style of migration stops a process completely on the source machine, checkpoints/terminates it, restarts it on the destination machine, then rejoins it with the other running processes. I think the only documentation that we have is at the webpage below (and my PhD thesis, if you want the finer details): http://osl.iu.edu/research/ft/ompi-cr/ We have wanted to experiment with a 'pre-copy' or 'live' migration style, but have not had the necessary support from the underlying checkpointer or time to devote to making it happen. I think BLCR is working on including the necessary pieces in a future release (there are papers where a development version of BLCR has done this with LAM/MPI). So that might be something of interest. Process migration techniques can benefit from fault prediction and 'good' target destination selection. Fault prediction allows us to move processes away from soon-to-fail locations, but it can be difficult to accurately predict failures. Open MPI has some hooks in the runtime layer that support 'sensors' which might help here. Good target destination selection is equally complex, but the idea here is to move processes to a machine where they can continue supporting the efficient execution of the application. So this might mean moving to the least loaded machine, or moving to a machine with other processes to reduce interprocess communication (something like dynamic load balancing). So there are some ideas to get you started. -- Josh On Thu, Aug 25, 2011 at 12:06 PM, Rayson Ho <raysonlo...@gmail.com> wrote: > Don't know which SSI project you are referring to... I only know the > OpenSSI project, and I was one of the first who subscribed to its > mailing list (since 2001). > > http://openssi.org/cgi-bin/view?page=openssi.html > > I don't think those OpenSSI clusters are designed for tens of > thousands of nodes, and not sure if it scales well to even a thousand > nodes -- so IMO they have limited use for HPC clusters. > > Rayson > > > > On Thu, Aug 25, 2011 at 11:45 AM, Durga Choudhury <dpcho...@gmail.com> wrote: >> Also, in 2005 there was an attempt to implement SSI (Single System >> Image) functionality to the then-current 2.6.10 kernel. The proposal >> was very detailed and covered most of the bases of task creation, PID >> allocation etc across a loosely tied cluster (without using fancy >> hardware such as RDMA fabric). Anybody knows if it was ever >> implemented? Any pointers in this direction? >> >> Thanks and regards >> Durga >> >> >> On Thu, Aug 25, 2011 at 11:08 AM, Rayson Ho <raysonlo...@gmail.com> wrote: >>> Srinivas, >>> >>> There's also Kernel-Level Checkpointing vs. User-Level Checkpointing - >>> if you can checkpoint an MPI task and restart it on a new node, then >>> this is also "process migration". >>> >>> Of course, doing a checkpoint & restart can be slower than pure >>> in-kernel process migration, but the advantage is that you don't need >>> any kernel support, and can in fact do all of it in user-space. >>> >>> Rayson >>> >>> >>> On Thu, Aug 25, 2011 at 10:26 AM, Ralph Castain <r...@open-mpi.org> wrote: >>>> It also depends on what part of migration interests you - are you wanting >>>> to look at the MPI part of the problem (reconnecting MPI transports, >>>> ensuring messages are not lost, etc.) or the RTE part of the problem >>>> (where to restart processes, detecting failures, etc.)? >>>> >>>> >>>> On Aug 24, 2011, at 7:04 AM, Jeff Squyres wrote: >>>> >>>>> Be aware that process migration is a pretty complex issue. >>>>> >>>>> Josh is probably the best one to answer your question directly, but he's >>>>> out today. >>>>> >>>>> >>>>> On Aug 24, 2011, at 5:45 AM, srinivas kundaram wrote: >>>>> >>>>>> I am final year grad student looking for my final year project in >>>>>> OpenMPI.We are group of 4 students. >>>>>> I wanted to know about the "Process Migration" process of MPI processes >>>>>> in OpenMPI. >>>>>> Can anyone suggest me any ideas for project related to process migration >>>>>> in OenMPI or other topics in Systems. >>>>>> >>>>>> >>>>>> >>>>>> regards, >>>>>> Srinivas Kundaram >>>>>> srinu1...@gmail.com >>>>>> +91-8149399160 >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> >>>>> -- >>>>> Jeff Squyres >>>>> jsquy...@cisco.com >>>>> For corporate legal information go to: >>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> >>> >>> >>> -- >>> Rayson >>> >>> ================================================== >>> Open Grid Scheduler - The Official Open Source Grid Engine >>> http://gridscheduler.sourceforge.net/ >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > > > -- > Rayson > > ================================================== > Open Grid Scheduler - The Official Open Source Grid Engine > http://gridscheduler.sourceforge.net/ > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey
