Dave, I specifically forced the OB1 PML with the OpenIB BTL. I have "--mca pml ob1 --mca btl openib,self" on my mpirun.
Originally, I assumed that the pipeline protocol was not kicking in as expected, and that the large cost you are seeing was due to pinning the entire buffer for the communication. Thus, I tried to alter the MCA parameters driving the pipeline protocol, but failed to see any major benefit (compared with the stock version). Here is what I used: mpirun --map-by node --mca pml ob1 --mca btl openib,self --mca btl_openib_get_limit $((1024*1024)) --mca btl_openib_put_limit $((1024*1024)) ./NPmpi --nocache --start 1000000 George. On Wed, May 3, 2017 at 4:27 PM, Dave Turner <drdavetur...@gmail.com> wrote: > George, > > Our local cluster runs Gentoo which I think prevents us from > using MXM and we do not use UCX. It's a pretty standard build > of 2.0.1 (ompi_info -a for Beocat is attached). > > I've also attached the ompi_info -a dump for Comet which is > running 1.8.4. A grep shows nothing about MXM or UCX. > > Are you testing with MXM or UCX that would be giving you > the different results? > > Dave > > On Wed, May 3, 2017 at 1:00 PM, <devel-requ...@lists.open-mpi.org> wrote: > >> Send devel mailing list submissions to >> devel@lists.open-mpi.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> or, via email, send a message with subject or body 'help' to >> devel-requ...@lists.open-mpi.org >> >> You can reach the person managing the list at >> devel-ow...@lists.open-mpi.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of devel digest..." >> >> >> Today's Topics: >> >> 1. NetPIPE performance curves (Dave Turner) >> 2. Re: NetPIPE performance curves (George Bosilca) >> 3. remote spawn - have no children (Justin Cinkelj) >> 4. Re: remote spawn - have no children (r...@open-mpi.org) >> 5. Re: remote spawn - have no children (r...@open-mpi.org) >> 6. Re: remote spawn - have no children (Justin Cinkelj) >> 7. Re: remote spawn - have no children (r...@open-mpi.org) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 2 May 2017 15:40:59 -0500 >> From: Dave Turner <drdavetur...@gmail.com> >> To: Open MPI Developers <devel@lists.open-mpi.org> >> Subject: [OMPI devel] NetPIPE performance curves >> Message-ID: >> <CAFGXdkzMut=5r6pv63pqxdwxep0ca1twna2vfofqj_8uyrw...@mail.gm >> ail.com> >> Content-Type: text/plain; charset="utf-8" >> >> >> I've used my NetPIPE communication benchmark ( >> http://netpipe.cs.ksu.edu) >> >> to measure the performance of OpenMPI and other implementations on >> Comet at SDSC (FDR IB, graph attached, same results measured elsewhere >> too). >> The uni-directional performance is good at 50 Gbps, the bi-directional >> performance >> is double that at 97 Gbps, and the aggregate bandwidth from measuring >> 24 bi-directional ping-pongs across the link between 2 nodes is a little >> lower >> than I'd like to see but still respectable, and similar for MVAPICH. All >> these >> were measured by reusing the same source and destination buffers >> each time. >> >> When I measure using the --nocache flag where the data comes >> from a new buffer in main memory each time, and is therefore also >> not already registered with the IB card, and likewise gets put into a >> new buffer in main memory, I see a loss in performance of at least >> 20%. Could someone please give me a short description of whether >> this is due to data being copied into a memory buffer that is already >> registered with the IB card, or whether this is the cost of registering >> the new memory with the IB card for its first use? >> I also see huge performance losses in this case when the message >> size is not a factor of 8 bytes (factors of 8 are the tops of the spikes). >> I've seen this in the past when there was a memory copy involved and >> the copy routine switched to a byte-by-byte copy for non factors of 8. >> While I don't know how many apps fall into this worst case scenario >> that the --nocache measurements represent, I could certainly see large >> bioinformatics runs being affected as the message lengths are not >> going to be factors of 8 bytes. >> >> Dave Turner >> >> -- >> Work: davetur...@ksu.edu (785) 532-7791 >> 2219 Engineering Hall, Manhattan KS 66506 >> Home: drdavetur...@gmail.com >> cell: (785) 770-5929 >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <https://rfd.newmexicoconsortium.org/mailman/private/devel/a >> ttachments/20170502/a6fe11d2/attachment.html> >> -------------- next part -------------- >> A non-text attachment was scrubbed... >> Name: np.comet.openmpi.pdf >> Type: application/pdf >> Size: 20093 bytes >> Desc: not available >> URL: <https://rfd.newmexicoconsortium.org/mailman/private/devel/a >> ttachments/20170502/a6fe11d2/attachment.pdf> >> >> ------------------------------ >> >> Message: 2 >> Date: Tue, 2 May 2017 22:57:52 -0400 >> From: George Bosilca <bosi...@icl.utk.edu> >> To: Dave Turner <drdavetur...@gmail.com>, Open MPI Developers >> <devel@lists.open-mpi.org> >> Subject: Re: [OMPI devel] NetPIPE performance curves >> Message-ID: >> <CAMJJpkVX+yUNComKV5Oc-zodzEh4E3PQtZvW5SOZhA8n4Um1Zg@mail. >> gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> >> David, >> >> Are you using the OB1 PML or one of our IB-enabled MTLs (UCX or MXM) ? I >> have access to similar cards, and I can't replicate your results. I do see >> a performance loss, but nowhere near what you have seen (it is going down >> to 47Gb instead of 50Gb). >> >> George. >> >> >> On Tue, May 2, 2017 at 4:40 PM, Dave Turner <drdavetur...@gmail.com> >> wrote: >> >> > >> > I've used my NetPIPE communication benchmark ( >> > http://netpipe.cs.ksu.edu) >> > to measure the performance of OpenMPI and other implementations on >> > Comet at SDSC (FDR IB, graph attached, same results measured elsewhere >> > too). >> > The uni-directional performance is good at 50 Gbps, the bi-directional >> > performance >> > is double that at 97 Gbps, and the aggregate bandwidth from measuring >> > 24 bi-directional ping-pongs across the link between 2 nodes is a little >> > lower >> > than I'd like to see but still respectable, and similar for MVAPICH. >> All >> > these >> > were measured by reusing the same source and destination buffers >> > each time. >> > >> > When I measure using the --nocache flag where the data comes >> > from a new buffer in main memory each time, and is therefore also >> > not already registered with the IB card, and likewise gets put into a >> > new buffer in main memory, I see a loss in performance of at least >> > 20%. Could someone please give me a short description of whether >> > this is due to data being copied into a memory buffer that is already >> > registered with the IB card, or whether this is the cost of registering >> > the new memory with the IB card for its first use? >> > I also see huge performance losses in this case when the message >> > size is not a factor of 8 bytes (factors of 8 are the tops of the >> spikes). >> > I've seen this in the past when there was a memory copy involved and >> > the copy routine switched to a byte-by-byte copy for non factors of 8. >> > While I don't know how many apps fall into this worst case scenario >> > that the --nocache measurements represent, I could certainly see large >> > bioinformatics runs being affected as the message lengths are not >> > going to be factors of 8 bytes. >> > >> > Dave Turner >> > >> > -- >> > Work: davetur...@ksu.edu (785) 532-7791 >> > 2219 Engineering Hall, Manhattan KS 66506 >> > Home: drdavetur...@gmail.com >> > cell: (785) 770-5929 >> > >> > _______________________________________________ >> > devel mailing list >> > devel@lists.open-mpi.org >> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <https://rfd.newmexicoconsortium.org/mailman/private/devel/a >> ttachments/20170502/fedc2258/attachment.html> >> >> ------------------------------ >> >> Message: 3 >> Date: Wed, 3 May 2017 09:17:45 +0200 >> From: Justin Cinkelj <justin.cink...@xlab.si> >> To: devel@lists.open-mpi.org >> Subject: [OMPI devel] remote spawn - have no children >> Message-ID: <0c2b60ba-ecdd-902f-3b1f-50bea9c5a...@xlab.si> >> Content-Type: text/plain; charset=utf-8; format=flowed >> >> Important detail first: I get this message from significantly modified >> Open MPI code, so problem exists solely due to my mistake. >> >> Orterun on 192.168.122.90 starts orted on remote node 192.168.122.91, >> than orted figures out it has nothing to do. >> If I request to start workers on the same 192.168.122.90 IP, the >> mpi_hello is started. >> >> Partial log: >> /usr/bin/mpirun -np 1 ... mpi_hello >> # >> [osv:00252] [[50738,0],0] plm:base:setup_job >> [osv:00252] [[50738,0],0] plm:base:setup_vm >> [osv:00252] [[50738,0],0] plm:base:setup_vm creating map >> [osv:00252] [[50738,0],0] setup:vm: working unmanaged allocation >> [osv:00252] [[50738,0],0] using dash_host >> [osv:00252] [[50738,0],0] checking node 192.168.122.91 >> [osv:00252] [[50738,0],0] plm:base:setup_vm add new daemon [[50738,0],1] >> [osv:00252] [[50738,0],0] plm:base:setup_vm assigning new daemon >> [[50738,0],1] to node 192.168.122.91 >> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 0 num_procs 2 >> [osv:00252] [[50738,0],0] routed:binomial 0 found child 1 >> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 1 num_procs 2 >> [osv:00252] [[50738,0],0] routed:binomial find children of rank 0 >> [osv:00252] [[50738,0],0] routed:binomial find children checking peer 1 >> [osv:00252] [[50738,0],0] routed:binomial find children computing tree >> [osv:00252] [[50738,0],0] routed:binomial rank 1 parent 0 me 1 num_procs 2 >> [osv:00252] [[50738,0],0] routed:binomial find children returning found >> value 0 >> [osv:00252] [[50738,0],0]: parent 0 num_children 1 >> [osv:00252] [[50738,0],0]: child 1 >> [osv:00252] [[50738,0],0] plm:osvrest: launching vm >> # >> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn called >> [osv:00250] [[50738,0],1] routed:binomial rank 0 parent 0 me 1 num_procs 2 >> [osv:00250] [[50738,0],1] routed:binomial find children of rank 0 >> [osv:00250] [[50738,0],1] routed:binomial find children checking peer 1 >> [osv:00250] [[50738,0],1] routed:binomial find children computing tree >> [osv:00250] [[50738,0],1] routed:binomial rank 1 parent 0 me 1 num_procs 2 >> [osv:00250] [[50738,0],1] routed:binomial find children returning found >> value 0 >> [osv:00250] [[50738,0],1]: parent 0 num_children 0 >> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn - have no children! >> >> In the plm mca module remote_spawn() function (my plm is based on >> orte/mca/plm/rsh/), the &coll.targets list has zero length. My question >> is, which module(s) are responsible for filling in the coll.targets? >> Then I will turn on the correct mca xzy_base_verbose level, and >> hopefully narrow down my problem. I have quite a problem >> guessing/finding out what various xyz strings mean :) >> >> Thank you, Justin >> >> >> ------------------------------ >> >> Message: 4 >> Date: Wed, 3 May 2017 06:26:03 -0700 >> From: "r...@open-mpi.org" <r...@open-mpi.org> >> To: OpenMPI Devel <devel@lists.open-mpi.org> >> Subject: Re: [OMPI devel] remote spawn - have no children >> Message-ID: <98236fb7-e31f-468d-ad07-fb2006e7c...@open-mpi.org> >> Content-Type: text/plain; charset=us-ascii >> >> The orte routed framework does that for you - there is an API for that >> purpose. >> >> >> > On May 3, 2017, at 12:17 AM, Justin Cinkelj <justin.cink...@xlab.si> >> wrote: >> > >> > Important detail first: I get this message from significantly modified >> Open MPI code, so problem exists solely due to my mistake. >> > >> > Orterun on 192.168.122.90 starts orted on remote node 192.168.122.91, >> than orted figures out it has nothing to do. >> > If I request to start workers on the same 192.168.122.90 IP, the >> mpi_hello is started. >> > >> > Partial log: >> > /usr/bin/mpirun -np 1 ... mpi_hello >> > # >> > [osv:00252] [[50738,0],0] plm:base:setup_job >> > [osv:00252] [[50738,0],0] plm:base:setup_vm >> > [osv:00252] [[50738,0],0] plm:base:setup_vm creating map >> > [osv:00252] [[50738,0],0] setup:vm: working unmanaged allocation >> > [osv:00252] [[50738,0],0] using dash_host >> > [osv:00252] [[50738,0],0] checking node 192.168.122.91 >> > [osv:00252] [[50738,0],0] plm:base:setup_vm add new daemon [[50738,0],1] >> > [osv:00252] [[50738,0],0] plm:base:setup_vm assigning new daemon >> [[50738,0],1] to node 192.168.122.91 >> > [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 0 >> num_procs 2 >> > [osv:00252] [[50738,0],0] routed:binomial 0 found child 1 >> > [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> > [osv:00252] [[50738,0],0] routed:binomial find children of rank 0 >> > [osv:00252] [[50738,0],0] routed:binomial find children checking peer 1 >> > [osv:00252] [[50738,0],0] routed:binomial find children computing tree >> > [osv:00252] [[50738,0],0] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> > [osv:00252] [[50738,0],0] routed:binomial find children returning found >> value 0 >> > [osv:00252] [[50738,0],0]: parent 0 num_children 1 >> > [osv:00252] [[50738,0],0]: child 1 >> > [osv:00252] [[50738,0],0] plm:osvrest: launching vm >> > # >> > [osv:00250] [[50738,0],1] plm:osvrest: remote spawn called >> > [osv:00250] [[50738,0],1] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> > [osv:00250] [[50738,0],1] routed:binomial find children of rank 0 >> > [osv:00250] [[50738,0],1] routed:binomial find children checking peer 1 >> > [osv:00250] [[50738,0],1] routed:binomial find children computing tree >> > [osv:00250] [[50738,0],1] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> > [osv:00250] [[50738,0],1] routed:binomial find children returning found >> value 0 >> > [osv:00250] [[50738,0],1]: parent 0 num_children 0 >> > [osv:00250] [[50738,0],1] plm:osvrest: remote spawn - have no children! >> > >> > In the plm mca module remote_spawn() function (my plm is based on >> orte/mca/plm/rsh/), the &coll.targets list has zero length. My question is, >> which module(s) are responsible for filling in the coll.targets? Then I >> will turn on the correct mca xzy_base_verbose level, and hopefully narrow >> down my problem. I have quite a problem guessing/finding out what various >> xyz strings mean :) >> > >> > Thank you, Justin >> > _______________________________________________ >> > devel mailing list >> > devel@lists.open-mpi.org >> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> >> >> >> ------------------------------ >> >> Message: 5 >> Date: Wed, 3 May 2017 06:29:16 -0700 >> From: "r...@open-mpi.org" <r...@open-mpi.org> >> To: OpenMPI Devel <devel@lists.open-mpi.org> >> Subject: Re: [OMPI devel] remote spawn - have no children >> Message-ID: <c2e59f60-1d3c-43c7-be7c-45036c5c6...@open-mpi.org> >> Content-Type: text/plain; charset=utf-8 >> >> I should have looked more closely as you already have the routed verbose >> output there. Everything in fact looks correct. The node with mpirun has 1 >> child, which is the daemon on the other node. The vpid=1 daemon on node 250 >> doesn?t have any children as there aren?t any more daemons in the system. >> >> Note that the output has nothing to do with spawning your mpi_hello - it >> is solely describing the startup of the daemons. >> >> >> > On May 3, 2017, at 6:26 AM, r...@open-mpi.org wrote: >> > >> > The orte routed framework does that for you - there is an API for that >> purpose. >> > >> > >> >> On May 3, 2017, at 12:17 AM, Justin Cinkelj <justin.cink...@xlab.si> >> wrote: >> >> >> >> Important detail first: I get this message from significantly modified >> Open MPI code, so problem exists solely due to my mistake. >> >> >> >> Orterun on 192.168.122.90 starts orted on remote node 192.168.122.91, >> than orted figures out it has nothing to do. >> >> If I request to start workers on the same 192.168.122.90 IP, the >> mpi_hello is started. >> >> >> >> Partial log: >> >> /usr/bin/mpirun -np 1 ... mpi_hello >> >> # >> >> [osv:00252] [[50738,0],0] plm:base:setup_job >> >> [osv:00252] [[50738,0],0] plm:base:setup_vm >> >> [osv:00252] [[50738,0],0] plm:base:setup_vm creating map >> >> [osv:00252] [[50738,0],0] setup:vm: working unmanaged allocation >> >> [osv:00252] [[50738,0],0] using dash_host >> >> [osv:00252] [[50738,0],0] checking node 192.168.122.91 >> >> [osv:00252] [[50738,0],0] plm:base:setup_vm add new daemon >> [[50738,0],1] >> >> [osv:00252] [[50738,0],0] plm:base:setup_vm assigning new daemon >> [[50738,0],1] to node 192.168.122.91 >> >> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 0 >> num_procs 2 >> >> [osv:00252] [[50738,0],0] routed:binomial 0 found child 1 >> >> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> >> [osv:00252] [[50738,0],0] routed:binomial find children of rank 0 >> >> [osv:00252] [[50738,0],0] routed:binomial find children checking peer 1 >> >> [osv:00252] [[50738,0],0] routed:binomial find children computing tree >> >> [osv:00252] [[50738,0],0] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> >> [osv:00252] [[50738,0],0] routed:binomial find children returning >> found value 0 >> >> [osv:00252] [[50738,0],0]: parent 0 num_children 1 >> >> [osv:00252] [[50738,0],0]: child 1 >> >> [osv:00252] [[50738,0],0] plm:osvrest: launching vm >> >> # >> >> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn called >> >> [osv:00250] [[50738,0],1] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> >> [osv:00250] [[50738,0],1] routed:binomial find children of rank 0 >> >> [osv:00250] [[50738,0],1] routed:binomial find children checking peer 1 >> >> [osv:00250] [[50738,0],1] routed:binomial find children computing tree >> >> [osv:00250] [[50738,0],1] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> >> [osv:00250] [[50738,0],1] routed:binomial find children returning >> found value 0 >> >> [osv:00250] [[50738,0],1]: parent 0 num_children 0 >> >> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn - have no children! >> >> >> >> In the plm mca module remote_spawn() function (my plm is based on >> orte/mca/plm/rsh/), the &coll.targets list has zero length. My question is, >> which module(s) are responsible for filling in the coll.targets? Then I >> will turn on the correct mca xzy_base_verbose level, and hopefully narrow >> down my problem. I have quite a problem guessing/finding out what various >> xyz strings mean :) >> >> >> >> Thank you, Justin >> >> _______________________________________________ >> >> devel mailing list >> >> devel@lists.open-mpi.org >> >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> > >> > _______________________________________________ >> > devel mailing list >> > devel@lists.open-mpi.org >> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> >> >> >> ------------------------------ >> >> Message: 6 >> Date: Wed, 3 May 2017 17:15:50 +0200 (CEST) >> From: Justin Cinkelj <justin.cink...@xlab.si> >> To: Open MPI Developers <devel@lists.open-mpi.org> >> Subject: Re: [OMPI devel] remote spawn - have no children >> Message-ID: >> <99032916.130606272.1493824550110.javamail.zim...@zimbra.xlab.si> >> Content-Type: text/plain; charset=utf-8 >> >> So "remote spawn" and children refer to orted daemons only, and I was >> looking into wrong modules. >> >> Which module(s) are then responsible to send command to orted to start >> mpi application? >> Which event names should I search for? >> >> Thank you, >> Justin >> >> ----- Original Message ----- >> > From: r...@open-mpi.org >> > To: "OpenMPI Devel" <devel@lists.open-mpi.org> >> > Sent: Wednesday, May 3, 2017 3:29:16 PM >> > Subject: Re: [OMPI devel] remote spawn - have no children >> > >> > I should have looked more closely as you already have the routed verbose >> > output there. Everything in fact looks correct. The node with mpirun >> has 1 >> > child, which is the daemon on the other node. The vpid=1 daemon on node >> 250 >> > doesn?t have any children as there aren?t any more daemons in the >> system. >> > >> > Note that the output has nothing to do with spawning your mpi_hello - >> it is >> > solely describing the startup of the daemons. >> > >> > >> > > On May 3, 2017, at 6:26 AM, r...@open-mpi.org wrote: >> > > >> > > The orte routed framework does that for you - there is an API for that >> > > purpose. >> > > >> > > >> > >> On May 3, 2017, at 12:17 AM, Justin Cinkelj <justin.cink...@xlab.si> >> > >> wrote: >> > >> >> > >> Important detail first: I get this message from significantly >> modified >> > >> Open MPI code, so problem exists solely due to my mistake. >> > >> >> > >> Orterun on 192.168.122.90 starts orted on remote node >> 192.168.122.91, than >> > >> orted figures out it has nothing to do. >> > >> If I request to start workers on the same 192.168.122.90 IP, the >> mpi_hello >> > >> is started. >> > >> >> > >> Partial log: >> > >> /usr/bin/mpirun -np 1 ... mpi_hello >> > >> # >> > >> [osv:00252] [[50738,0],0] plm:base:setup_job >> > >> [osv:00252] [[50738,0],0] plm:base:setup_vm >> > >> [osv:00252] [[50738,0],0] plm:base:setup_vm creating map >> > >> [osv:00252] [[50738,0],0] setup:vm: working unmanaged allocation >> > >> [osv:00252] [[50738,0],0] using dash_host >> > >> [osv:00252] [[50738,0],0] checking node 192.168.122.91 >> > >> [osv:00252] [[50738,0],0] plm:base:setup_vm add new daemon >> [[50738,0],1] >> > >> [osv:00252] [[50738,0],0] plm:base:setup_vm assigning new daemon >> > >> [[50738,0],1] to node 192.168.122.91 >> > >> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 0 >> num_procs 2 >> > >> [osv:00252] [[50738,0],0] routed:binomial 0 found child 1 >> > >> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> > >> [osv:00252] [[50738,0],0] routed:binomial find children of rank 0 >> > >> [osv:00252] [[50738,0],0] routed:binomial find children checking >> peer 1 >> > >> [osv:00252] [[50738,0],0] routed:binomial find children computing >> tree >> > >> [osv:00252] [[50738,0],0] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> > >> [osv:00252] [[50738,0],0] routed:binomial find children returning >> found >> > >> value 0 >> > >> [osv:00252] [[50738,0],0]: parent 0 num_children 1 >> > >> [osv:00252] [[50738,0],0]: child 1 >> > >> [osv:00252] [[50738,0],0] plm:osvrest: launching vm >> > >> # >> > >> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn called >> > >> [osv:00250] [[50738,0],1] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> > >> [osv:00250] [[50738,0],1] routed:binomial find children of rank 0 >> > >> [osv:00250] [[50738,0],1] routed:binomial find children checking >> peer 1 >> > >> [osv:00250] [[50738,0],1] routed:binomial find children computing >> tree >> > >> [osv:00250] [[50738,0],1] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> > >> [osv:00250] [[50738,0],1] routed:binomial find children returning >> found >> > >> value 0 >> > >> [osv:00250] [[50738,0],1]: parent 0 num_children 0 >> > >> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn - have no >> children! >> > >> >> > >> In the plm mca module remote_spawn() function (my plm is based on >> > >> orte/mca/plm/rsh/), the &coll.targets list has zero length. My >> question >> > >> is, which module(s) are responsible for filling in the coll.targets? >> Then >> > >> I will turn on the correct mca xzy_base_verbose level, and hopefully >> > >> narrow down my problem. I have quite a problem guessing/finding out >> what >> > >> various xyz strings mean :) >> > >> >> > >> Thank you, Justin >> > >> _______________________________________________ >> > >> devel mailing list >> > >> devel@lists.open-mpi.org >> > >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> > > >> > > _______________________________________________ >> > > devel mailing list >> > > devel@lists.open-mpi.org >> > > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> > >> > _______________________________________________ >> > devel mailing list >> > devel@lists.open-mpi.org >> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> >> >> ------------------------------ >> >> Message: 7 >> Date: Wed, 3 May 2017 08:54:26 -0700 >> From: "r...@open-mpi.org" <r...@open-mpi.org> >> To: OpenMPI Devel <devel@lists.open-mpi.org> >> Subject: Re: [OMPI devel] remote spawn - have no children >> Message-ID: <25adbe2f-1fd0-4a80-8244-9b348ca9c...@open-mpi.org> >> Content-Type: text/plain; charset="utf-8" >> >> Everything operates via the state machine - events trigger moving the job >> from one state to the next, with each state being tied to a callback >> function that implements that state. If you set state_base_verbose=5, >> you?ll see when and where each state gets executed. >> >> By default, the launch_app state goes to a function in the plm/base: >> >> https://github.com/open-mpi/ompi/blob/master/orte/mca/plm/ba >> se/plm_base_launch_support.c#L477 <https://github.com/open-mpi/o >> mpi/blob/master/orte/mca/plm/base/plm_base_launch_support.c#L477> >> >> I suspect the problem is that your plm component isn?t activating the >> next step upon completion of launch_daemons. >> >> >> > On May 3, 2017, at 8:15 AM, Justin Cinkelj <justin.cink...@xlab.si> >> wrote: >> > >> > So "remote spawn" and children refer to orted daemons only, and I was >> looking into wrong modules. >> > >> > Which module(s) are then responsible to send command to orted to start >> mpi application? >> > Which event names should I search for? >> > >> > Thank you, >> > Justin >> > >> > ----- Original Message ----- >> >> From: r...@open-mpi.org >> >> To: "OpenMPI Devel" <devel@lists.open-mpi.org> >> >> Sent: Wednesday, May 3, 2017 3:29:16 PM >> >> Subject: Re: [OMPI devel] remote spawn - have no children >> >> >> >> I should have looked more closely as you already have the routed >> verbose >> >> output there. Everything in fact looks correct. The node with mpirun >> has 1 >> >> child, which is the daemon on the other node. The vpid=1 daemon on >> node 250 >> >> doesn?t have any children as there aren?t any more daemons in the >> system. >> >> >> >> Note that the output has nothing to do with spawning your mpi_hello - >> it is >> >> solely describing the startup of the daemons. >> >> >> >> >> >>> On May 3, 2017, at 6:26 AM, r...@open-mpi.org wrote: >> >>> >> >>> The orte routed framework does that for you - there is an API for that >> >>> purpose. >> >>> >> >>> >> >>>> On May 3, 2017, at 12:17 AM, Justin Cinkelj <justin.cink...@xlab.si> >> >>>> wrote: >> >>>> >> >>>> Important detail first: I get this message from significantly >> modified >> >>>> Open MPI code, so problem exists solely due to my mistake. >> >>>> >> >>>> Orterun on 192.168.122.90 starts orted on remote node >> 192.168.122.91, than >> >>>> orted figures out it has nothing to do. >> >>>> If I request to start workers on the same 192.168.122.90 IP, the >> mpi_hello >> >>>> is started. >> >>>> >> >>>> Partial log: >> >>>> /usr/bin/mpirun -np 1 ... mpi_hello >> >>>> # >> >>>> [osv:00252] [[50738,0],0] plm:base:setup_job >> >>>> [osv:00252] [[50738,0],0] plm:base:setup_vm >> >>>> [osv:00252] [[50738,0],0] plm:base:setup_vm creating map >> >>>> [osv:00252] [[50738,0],0] setup:vm: working unmanaged allocation >> >>>> [osv:00252] [[50738,0],0] using dash_host >> >>>> [osv:00252] [[50738,0],0] checking node 192.168.122.91 >> >>>> [osv:00252] [[50738,0],0] plm:base:setup_vm add new daemon >> [[50738,0],1] >> >>>> [osv:00252] [[50738,0],0] plm:base:setup_vm assigning new daemon >> >>>> [[50738,0],1] to node 192.168.122.91 >> >>>> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 0 >> num_procs 2 >> >>>> [osv:00252] [[50738,0],0] routed:binomial 0 found child 1 >> >>>> [osv:00252] [[50738,0],0] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> >>>> [osv:00252] [[50738,0],0] routed:binomial find children of rank 0 >> >>>> [osv:00252] [[50738,0],0] routed:binomial find children checking >> peer 1 >> >>>> [osv:00252] [[50738,0],0] routed:binomial find children computing >> tree >> >>>> [osv:00252] [[50738,0],0] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> >>>> [osv:00252] [[50738,0],0] routed:binomial find children returning >> found >> >>>> value 0 >> >>>> [osv:00252] [[50738,0],0]: parent 0 num_children 1 >> >>>> [osv:00252] [[50738,0],0]: child 1 >> >>>> [osv:00252] [[50738,0],0] plm:osvrest: launching vm >> >>>> # >> >>>> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn called >> >>>> [osv:00250] [[50738,0],1] routed:binomial rank 0 parent 0 me 1 >> num_procs 2 >> >>>> [osv:00250] [[50738,0],1] routed:binomial find children of rank 0 >> >>>> [osv:00250] [[50738,0],1] routed:binomial find children checking >> peer 1 >> >>>> [osv:00250] [[50738,0],1] routed:binomial find children computing >> tree >> >>>> [osv:00250] [[50738,0],1] routed:binomial rank 1 parent 0 me 1 >> num_procs 2 >> >>>> [osv:00250] [[50738,0],1] routed:binomial find children returning >> found >> >>>> value 0 >> >>>> [osv:00250] [[50738,0],1]: parent 0 num_children 0 >> >>>> [osv:00250] [[50738,0],1] plm:osvrest: remote spawn - have no >> children! >> >>>> >> >>>> In the plm mca module remote_spawn() function (my plm is based on >> >>>> orte/mca/plm/rsh/), the &coll.targets list has zero length. My >> question >> >>>> is, which module(s) are responsible for filling in the coll.targets? >> Then >> >>>> I will turn on the correct mca xzy_base_verbose level, and hopefully >> >>>> narrow down my problem. I have quite a problem guessing/finding out >> what >> >>>> various xyz strings mean :) >> >>>> >> >>>> Thank you, Justin >> >>>> _______________________________________________ >> >>>> devel mailing list >> >>>> devel@lists.open-mpi.org >> >>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> >>> >> >>> _______________________________________________ >> >>> devel mailing list >> >>> devel@lists.open-mpi.org >> >>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> >> >> >> _______________________________________________ >> >> devel mailing list >> >> devel@lists.open-mpi.org >> >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> > _______________________________________________ >> > devel mailing list >> > devel@lists.open-mpi.org >> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <https://rfd.newmexicoconsortium.org/mailman/private/devel/a >> ttachments/20170503/5f5d7b7a/attachment.html> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> devel mailing list >> devel@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >> >> ------------------------------ >> >> End of devel Digest, Vol 3468, Issue 1 >> ************************************** >> > > > > -- > Work: davetur...@ksu.edu (785) 532-7791 > 2219 Engineering Hall, Manhattan KS 66506 > Home: drdavetur...@gmail.com > cell: (785) 770-5929 > > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
