Justin, one more question...
if you want to run n>1 MPI tasks per node, would you have ? - 1 container with one orted and n MPI tasks - n containers with one orted and one MPI task per container And btw, did you configure ompi with --disable-dlopen ? If not, I strongly encourage you to do so. Cheers, Gilles On Friday, October 16, 2015, Jeff Squyres (jsquyres) <jsquy...@cisco.com> wrote: > Let me follow up on this... > > IOF is but one of the frameworks / plugins involved in launching and > monitoring processes. > > It might actually be easier to get on a webex and give you an overview > (Ralph would be the best person for this; he's the one would does most of > the work in the ORTE layer); I'm not sure we have good documentation for it > online. > > Part of the problem is that in our current design, MPI processes are > really not designed to be in the same process as the orted. It *might* be > possible to make that happen, but I think we have a lot of assumptions > built in to the code that the orted that the process(es) that it launches > will actually be separate / different OS processes. > > That being said, it might be an easier solution to just not have the > orted. That is, ORTE is capable of "orted-less" launches when the > underlying runtime provides enough support for OMPI to not have to use the > orteds. This would allow you to launch the MPI process directly in your > container without any dlopen/orted-process-merging tomfoolery. This might > avoid running afoul of many of the assumptions we have baked into the > system. > > Ralph will need to give the details of how to support orted-less > launching. But the first question is: does OSv have some kind of > programatic mechanism to launch a process in your containers? I.e., can > mpirun programmatically launch MPI processes in OSv containers? > > > > > On Oct 16, 2015, at 6:48 AM, Justin Cinkelj <justin.cink...@xlab.si > <javascript:;>> wrote: > > > > Thank you. At least its clear now that for the immediate problem I have > > to look at IOF code. > > > > > > On 16. 10. 2015 03:32, Gilles Gouaillardet wrote: > >> Justin, > >> > >> IOF stands for Input/Output (aka I/O) Forwarding > >> > >> here is a very high level overview of a quite simple case. > >> on host A, you run > >> mpirun -host B,C -np 2 a.out > >> without any batch manager and TCP interconnect > >> > >> first, mpirun will fork&exec > >> ssh B orted ... > >> ssh C orted ... > >> > >> the orted daemons will first connect back to mpirun, using TCP and > >> ip/port passed on the orted command line. > >> > >> then the orted daemons will fork&exec a.out > >> a.out will contact its parent orted (iirc, TCP on v1.10 and Unix > >> socket from v2.x) via ip/port of port from the environment > >> when a.out want to communicate, they will connect to the remote a.out > >> via TCP using ip/port obtained from orted. > >> > >> from a.out point of view : > >> - stdin is either a pipe to orted or /dev/null > >> - stdout is a pty with orted on the other side > >> - stderr is a pipe to orted > >> > >> this is basically what happens in a quite simple case, > >> back to your question, mpi_hello.so does not contact mpirun. > >> orted.so contacts mpirun, and mpi_hello.so contacts orted.so, > >> and then mpi_hello.so contact other mpi_hello.so > >> > >> > >> note it is also possible to use direct launch (SLURM or cray alps can > >> do that) > >> instead of running > >> mpirun a.out > >> you simply do > >> srun a.out (or aprun a.out) > >> in the case of slurm (i am not sure about alps) there is no orted > >> daemons involved. > >> instead of contacting its orted, a.out contact the slurm daemons > >> (slurmd) so it can exchange information with remote a.out and figure > >> out how to contact them. > >> direct launch does not support dynamic process creation > >> (MPI_Comm_spawn and friends) > >> > >> > >> you can run > >> ompi_info --all > >> to list all the parameters. > >> and then you can do > >> mpirun --mca <name> <value> ... > >> to modify a parameter (such as timeout) > >> > >> that being said, i do not think that should be needed ... just make > >> sure there is no firewall running on your system, and you should be > fine. > >> if some hosts have several interfaces, you can restrict to the one > >> that should work (e.g. eth0) with > >> mpirun --mca oob_tcp_if_include eth0 --mca btl_tcp_if_include eth0 ... > >> > >> > >> i hope this helps > >> > >> Gilles > >> > >> > >> On 10/16/2015 2:59 AM, Justin Cinkelj wrote: > >>> I'm trying to run OpenMPI in OSv container > >>> (https://github.com/cloudius-systems/osv). It's a single process, > single > >>> address space VM, without fork, exec, openpty function. With some > >>> butchering of OSv and OpenMPI I was able to compile orted.so, and run > it > >>> inside OSv via mpirun (mpirun is on remote machine). The orted.so loads > >>> mpi_hello.so and executes its main() in new pthread. > >>> > >>> Which than aborts due to communication failure/timeout - as reported by > >>> mpirun. I assume that that mpi_hello.so should connect back to mpirun, > >>> and report 'something' about itself. What could that be? > >>> Plus, where could I extend that timeout period - once mpirun closes, > >>> output from opal_output is not shown any more. > >>> > >>> Is there some highlevel overview about OpenMPI, how are modules > >>> connected, what is 'startup' sequence etc? > >>> ompi_info lists compiled modules, but I still don't know how are they > >>> connected. > >>> > >>> So basically - I lack knowledge of OpenMPI internals, and would highly > >>> appreciate links for "rookie" developers. > >>> Say https://github.com/open-mpi/ompi/wiki/IOFDesign tells me what IOF > >>> is, and a bit about its working. So, if someone has any list of such > >>> links - could it be shared? > >>> > >>> _______________________________________________ > >>> devel mailing list > >>> de...@open-mpi.org <javascript:;> > >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > >>> Link to this post: > >>> http://www.open-mpi.org/community/lists/devel/2015/10/18181.php > >>> > >> > >> _______________________________________________ > >> devel mailing list > >> de...@open-mpi.org <javascript:;> > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > >> Link to this post: > >> http://www.open-mpi.org/community/lists/devel/2015/10/18189.php > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org <javascript:;> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/10/18190.php > > > -- > Jeff Squyres > jsquy...@cisco.com <javascript:;> > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > devel mailing list > de...@open-mpi.org <javascript:;> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/10/18191.php >
