I'll still need to look at the intercomm_create issue, but I just tested both the trunk and current 1.7.3 branch for "add-host" and both worked just fine. This was on my little test cluster which only has rsh available - no Torque.
You might add "-mca plm_base_verbose 5" to your cmd line to get some debug output as to the problem. On Sep 21, 2013, at 5:48 PM, Ralph Castain <r...@open-mpi.org> wrote: > > On Sep 21, 2013, at 4:54 PM, Suraj Prabhakaran <suraj.prabhaka...@gmail.com> > wrote: > >> Dear all, >> >> Really thanks a lot for your efforts. I too downloaded the trunk to check if >> it works for my case and as of revision 29215, it works for the original >> case I reported. Although it works, I still see the following in the output. >> Does it mean anything? >> [grsacc17][[13611,1],0][btl_openib_proc.c:157:mca_btl_openib_proc_create] >> [btl_openib_proc.c:157] ompi_modex_recv failed for peer [[13611,2],0] > > Yes - it means we don't quite have this right yet :-( > >> >> However, on another topic relevant to my use case, I have another problem to >> report. I am having problems using the "add-host" info to the >> MPI_Comm_spawn() when MPI is compiled with support for Torque resource >> manager. This problem is totally new in the 1.7 series and it worked >> perfectly until 1.6.5 >> >> Basically, I am working on implementing dynamic resource management >> facilities in the Torque/Maui batch system. Through a new tm call, an >> application can get new resources for a job. > > FWIW: you'll find that we added an API to the orte RAS framework to support > precisely that operation. It allows an application to request that we > dynamically obtain additional resources during execution (e.g., as part of a > Comm_spawn call via an info_key). We originally implemented this with Slurm, > but you could add the calls into the Torque component as well if you like. > > This is in the trunk now - will come over to 1.7.4 > > >> I want to use MPI_Comm_spawn() to spawn new processes in the new hosts. With >> my extended torque/maui batch system, I was able to perfectly use the >> "add-host" info argument to MPI_Comm_spawn() to spawn new processes on these >> hosts. Since MPI and Torque refer to the hosts through the nodeids, I made >> sure that OpenMPI uses the correct nodeid's for these new hosts. >> Until 1.6.5, this worked perfectly fine, except that due to the >> Intercomm_merge problem, I could not really run a real application to its >> completion. >> >> While this is now fixed in the trunk, I found that, however, when using the >> "add-host" info argument, everything collapses after printing out the >> following error. >> >> [warn] opal_libevent2021_event_base_loop: reentrant invocation. Only one >> event_base_loop can run on each event_base at once. > > I'll take a look - probably some stale code that hasn't been updated yet for > async ORTE operations > >> >> And due to this, I am still not really able to run my application! I also >> compiled the MPI without any Torque/PBS support and just used the "add-host" >> argument normally. Again, this worked perfectly in 1.6.5. But in the 1.7 >> series, it works but after printing out the following error. >> >> [grsacc17][[13731,1],0][btl_openib_proc.c:157:mca_btl_openib_proc_create] >> [btl_openib_proc.c:157] ompi_modex_recv failed for peer [[13731,2],0] >> [grsacc17][[13731,1],1][btl_openib_proc.c:157:mca_btl_openib_proc_create] >> [btl_openib_proc.c:157] ompi_modex_recv failed for peer [[13731,2],0] > > Yeah, the 1.7 series doesn't have the reentrant test in it - so we > "illegally" re-enter libevent. The error again means we don't have > Intercomm_create correct just yet. > > I'll see what I can do about this and get back to you > >> >> In short, with pbs/torque support, it fails and without pbs/torque support, >> it runs after spitting the above lines. >> >> I would really appreciate some help on this, since I need these features to >> actually test my case and (at least in my short experience) no other MPI >> implementation seem friendly to such dynamic scenarios. >> >> Thanks a lot! >> >> Best, >> Suraj >> >> >> >> On Sep 20, 2013, at 4:58 PM, Jeff Squyres (jsquyres) wrote: >> >>> Just to close my end of this loop: as of trunk r29213, it all works for me. >>> Thanks! >>> >>> >>> On Sep 18, 2013, at 12:52 PM, Ralph Castain <r...@open-mpi.org> wrote: >>> >>>> Thanks George - much appreciated >>>> >>>> On Sep 18, 2013, at 9:49 AM, George Bosilca <bosi...@icl.utk.edu> wrote: >>>> >>>>> The test case was broken. I just pushed a fix. >>>>> >>>>> George. >>>>> >>>>> On Sep 18, 2013, at 16:49 , Ralph Castain <r...@open-mpi.org> wrote: >>>>> >>>>>> Hangs with any np > 1 >>>>>> >>>>>> However, I'm not sure if that's an issue with the test vs the underlying >>>>>> implementation >>>>>> >>>>>> On Sep 18, 2013, at 7:40 AM, "Jeff Squyres (jsquyres)" >>>>>> <jsquy...@cisco.com> wrote: >>>>>> >>>>>>> Does it hang when you run with -np 4? >>>>>>> >>>>>>> Sent from my phone. No type good. >>>>>>> >>>>>>> On Sep 18, 2013, at 4:10 PM, "Ralph Castain" <r...@open-mpi.org> wrote: >>>>>>> >>>>>>>> Strange - it works fine for me on my Mac. However, I see one >>>>>>>> difference - I only run it with np=1 >>>>>>>> >>>>>>>> On Sep 18, 2013, at 2:22 AM, Jeff Squyres (jsquyres) >>>>>>>> <jsquy...@cisco.com> wrote: >>>>>>>> >>>>>>>>> On Sep 18, 2013, at 9:33 AM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> 1. sm doesn't work between spawned processes. So you must have >>>>>>>>>> another network enabled. >>>>>>>>> >>>>>>>>> I know :-). I have tcp available as well (OMPI will abort if you >>>>>>>>> only run with sm,self because the comm_spawn will fail with >>>>>>>>> unreachable errors -- I just tested/proved this to myself). >>>>>>>>> >>>>>>>>>> 2. Don't use the test case attached to my email, I left an xterm >>>>>>>>>> based spawn and the debugging. It can't work without xterm support. >>>>>>>>>> Instead try using the test case from the trunk, the one committed by >>>>>>>>>> Ralph. >>>>>>>>> >>>>>>>>> I didn't see any "xterm" strings in there, but ok. :-) I ran with >>>>>>>>> orte/test/mpi/intercomm_create.c, and that hangs for me as well: >>>>>>>>> >>>>>>>>> ----- >>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create >>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create >>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, &inter) >>>>>>>>> [rank 4] >>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, &inter) >>>>>>>>> [rank 5] >>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, &inter) >>>>>>>>> [rank 6] >>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, &inter) >>>>>>>>> [rank 7] >>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>> [rank 4] >>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>> [rank 5] >>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>> [rank 6] >>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>> [rank 7] >>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>> [hang] >>>>>>>>> ----- >>>>>>>>> >>>>>>>>> Similarly, on my Mac, it hangs with no output: >>>>>>>>> >>>>>>>>> ----- >>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create >>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create >>>>>>>>> [hang] >>>>>>>>> ----- >>>>>>>>> >>>>>>>>>> George. >>>>>>>>>> >>>>>>>>>> On Sep 18, 2013, at 07:53 , "Jeff Squyres (jsquyres)" >>>>>>>>>> <jsquy...@cisco.com> wrote: >>>>>>>>>> >>>>>>>>>>> George -- >>>>>>>>>>> >>>>>>>>>>> When I build the SVN trunk (r29201) on 64 bit linux, your attached >>>>>>>>>>> test case hangs: >>>>>>>>>>> >>>>>>>>>>> ----- >>>>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create >>>>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create >>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, >>>>>>>>>>> &inter) [rank 4] >>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, >>>>>>>>>>> &inter) [rank 5] >>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, >>>>>>>>>>> &inter) [rank 6] >>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, >>>>>>>>>>> &inter) [rank 7] >>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0) >>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>>>> [rank 4] >>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>>>> [rank 5] >>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>>>> [rank 6] >>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) >>>>>>>>>>> [rank 7] >>>>>>>>>>> [hang] >>>>>>>>>>> ----- >>>>>>>>>>> >>>>>>>>>>> On my Mac, it hangs without printing anything: >>>>>>>>>>> >>>>>>>>>>> ----- >>>>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create >>>>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create >>>>>>>>>>> [hang] >>>>>>>>>>> ----- >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sep 18, 2013, at 1:48 AM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Here is a quick (and definitively not the cleanest) patch that >>>>>>>>>>>> addresses the MPI_Intercomm issue at the MPI level. It should be >>>>>>>>>>>> applied after removal of 29166. >>>>>>>>>>>> >>>>>>>>>>>> I also added the corrected test case stressing the corner cases by >>>>>>>>>>>> doing barriers at every inter-comm creation and doing a clean >>>>>>>>>>>> disconnect. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Jeff Squyres >>>>>>>>>>> jsquy...@cisco.com >>>>>>>>>>> For corporate legal information go to: >>>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> devel mailing list >>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> devel mailing list >>>>>>>>>> de...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jeff Squyres >>>>>>>>> jsquy...@cisco.com >>>>>>>>> For corporate legal information go to: >>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> devel mailing list >>>>>>>>> de...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> devel mailing list >>>>>>>> de...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>> _______________________________________________ >>>>>>> devel mailing list >>>>>>> de...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>> >>>>> _______________________________________________ >>>>> devel mailing list >>>>> de...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> >>> >>> -- >>> Jeff Squyres >>> jsquy...@cisco.com >>> For corporate legal information go to: >>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>> >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >
