Okay, thanks - I'll get on it tonight. Looks like a fairly simple bug, so hopefully I'll have it ironed out tonight.
On Mon, Feb 2, 2015 at 1:40 PM, Mark Santcroos <mark.santcr...@rutgers.edu> wrote: > FWIW: I see similar behaviour on my laptop (OS X Yosemite 10.10.2). > > > On 02 Feb 2015, at 21:26 , Mark Santcroos <mark.santcr...@rutgers.edu> > wrote: > > > > Ok, let me check on some other systems too though, it might be Cray > specific. > > > > > >> On 02 Feb 2015, at 19:07 , Ralph Castain <r...@open-mpi.org> wrote: > >> > >> Yikes - looks like a bug crept into there at the last minute. I > actually had it working just fine - not sure what happened here. I'm on > travel this week, but I'll try to dig into this a bit and spot the issue. > >> > >> Thanks! > >> Ralph > >> > >> > >> On Mon, Feb 2, 2015 at 3:50 AM, Mark Santcroos < > mark.santcr...@rutgers.edu> wrote: > >> Hi Ralph, > >> > >> Great, the semantics look exactly as what I need! > >> > >> (To aid in debugging I added "--debug-devel" to orte-dvm.c which was > useful to detect and come by some initial bumps) > >> > >> The current status: > >> > >> * I can submit applications and see their output on the orte-dvm console > >> > >> * The following message is reported infinitely on the orte-submit > console: > >> > >> [warn] opal_libevent2022_event_base_loop: reentrant invocation. Only > one event_base_loop can run on each event_base at once. > >> > >> * orte-submit doesn't return, while I see "[nid02819:20571] > [[2120,0],0] dvm: job [2120,9] has completed" on the orte-dvm console. > >> > >> * On the orte-dvm console I see the following when submitting (so also > for "successful" runs): > >> > >> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID] > >> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file > ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433 > >> [nid03534:31545] procdir: /tmp/openmpi-sessions-62758@nid03534_0 > /9021/1/0 > >> [nid03534:31545] jobdir: /tmp/openmpi-sessions-62758@nid03534_0/9021/1 > >> [nid03534:31545] top: openmpi-sessions-62758@nid03534_0 > >> [nid03534:31545] tmp: /tmp > >> [nid03534:31545] sess_dir_finalize: proc session dir does not exist > >> > >> * If I dont specify any "-np" on the orte-submit, then I see on the > orte-dvm console: > >> > >> [nid02434:00564] [[9021,0],0] Releasing job data for [INVALID] > >> [nid03388:26474] [[9021,0],2] ORTE_ERROR_LOG: Not found in file > ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433 > >> [nid03534:31544] [[9021,0],1] ORTE_ERROR_LOG: Not found in file > ../../../../orte/mca/odls/base/odls_base_default_fns.c at line 433 > >> > >> * It only seems to work for single nodes (probably related to the > previous point). > >> > >> > >> Is this all expected behaviour given the current implementation? > >> > >> > >> Thanks! > >> > >> Mark > >> > >> > >> > >>> On 02 Feb 2015, at 4:21 , Ralph Castain <r...@open-mpi.org> wrote: > >>> > >>> I have pushed the changes to the OMPI master. It took a little bit > more than I had hoped due to the changes to the ORTE infrastructure, but > hopefully this will meet your needs. It consists of two new tools: > >>> > >>> (a) orte-dvm - starts the virtual machine by launching a daemon on > every node of the allocation, as constrained by -host and/or -hostfile. > Check the options for outputting the URI as you’ll need that info for the > other tool. The DVM remains “up” until you issue the orte-submit -terminate > command, or hit the orte-dvm process with a sigterm. > >>> > >>> (b) orte-submit - takes the place of mpirun. Basically just packages > your app and arguments and sends it to orte-dvm for execution. Requires the > URI of orte-dvm. The tool exits once the job has completed execution, > though you can run multiple jobs in parallel by backgrounding orte-submit > or issuing commands from separate shells. > >>> > >>> I’ve added man pages for both tools, though they may not be complete. > Also, I don’t have all the mapping/ranking/binding options supported just > yet as I first wanted to see if this meets your basic needs before worrying > about the detail. > >>> > >>> Let me know what you think > >>> Ralph > >>> > >>> > >>>> On Jan 21, 2015, at 4:07 PM, Mark Santcroos < > mark.santcr...@rutgers.edu> wrote: > >>>> > >>>> Hi Ralph, > >>>> > >>>> All makes sense! Thanks a lot! > >>>> > >>>> Looking forward to your modifications. > >>>> Please don't hesitate to through things with rough-edges to me! > >>>> > >>>> Cheers, > >>>> > >>>> Mark > >>>> > >>>>> On 21 Jan 2015, at 23:21 , Ralph Castain <r...@open-mpi.org> wrote: > >>>>> > >>>>> Let me address your questions up here so you don’t have to scan thru > the entire note. > >>>>> > >>>>> PMIx rationale: PMI has been around for a long time, primarily used > inside the MPI library implementations to perform wireup. It provided a > link from the MPI library to the local resource manager. However, as we > move towards exascale, two things became apparent: > >>>>> > >>>>> 1. the current PMI implementations don’t scale adequately to get > there. The API created too many communications and assumed everything was a > blocking operation, thus preventing asynchronous progress > >>>>> > >>>>> 2. there were increasing requests for application-level interactions > to the resource manager. People want ways to spawn jobs (and not just from > within MPI), request pre-location of data, control power, etc. Rather than > having every RM write its own interface (and thus make everyone’s code > non-portable), we at Intel decided to extend the existing PMI definitions > to support those functions. Thus, an application developer can directly > access PMIx functions to perform all those operations. > >>>>> > >>>>> PMIx v1.0 is about to be released - it’ll be backward compatible > with PMI-1 and PMI-2, plus add non-blocking operations and significantly > reduce the number of communications. PMIx 2.0 is slated for this summer and > will include the advanced controls capabilities. > >>>>> > >>>>> ORCM is being developed because we needed a BSD-licensed, fully > featured resource manager. This will allow us to integrate the RM even more > tightly to the file system, networking, and other subsystems, thus > achieving higher launch performance and providing desired features such as > QoS management. PMIx is a part of that plan, but as you say, they each play > their separate roles in the overall stack. > >>>>> > >>>>> > >>>>> Persistent ORTE: there is a learning curve on ORTE, I fear. We do > have some videos on the web site that can help get you started, and I’ve > given a number of “classes" at Intel now for that purpose. I still have it > on my “to-do” list that I summarize those classes and post them on the web > site. > >>>>> > >>>>> For now, let me summarize how things work. At startup, mpirun reads > the allocation (usually from the environment, but it depends on the host > RM) and launches a daemon on each allocated node. Each daemon reads its > local hardware environment and “phones home” to let mpirun know it is > alive. Once all daemons have reported, mpirun maps the processes to the > nodes and sends that map to all the daemons in a scalable broadcast pattern. > >>>>> > >>>>> Upon receipt of the launch message, each daemon parses it to > identify which procs it needs to locally spawn. Once spawned, each proc > connects back to its local daemon via a Unix domain socket for wireup > support. As procs complete, the daemon maintains bookkeeping and reports > back to mpirun once all procs are done. When all procs are reported > complete (or one reports as abnormally terminated), mpirun sends a “die” > message to every daemon so it will cleanly terminate. > >>>>> > >>>>> What I will do is simply tell mpirun to not do that last step, but > instead to wait to receive a “terminate” cmd before ending the daemons. > This will allow you to reuse the existing DVM, making each independent job > start a great deal faster. You’ll need to either manually terminate the > DVM, or the RM will do so when the allocation expires. > >>>>> > >>>>> HTH > >>>>> Ralph > >>>>> > >>>>> > >>>>>> On Jan 21, 2015, at 12:52 PM, Mark Santcroos < > mark.santcr...@rutgers.edu> wrote: > >>>>>> > >>>>>> Hi Ralph, > >>>>>> > >>>>>>> On 21 Jan 2015, at 21:20 , Ralph Castain <r...@open-mpi.org> wrote: > >>>>>>> > >>>>>>> Hi Mark > >>>>>>> > >>>>>>>> On Jan 21, 2015, at 11:21 AM, Mark Santcroos < > mark.santcr...@rutgers.edu> wrote: > >>>>>>>> > >>>>>>>> Hi Ralph, all, > >>>>>>>> > >>>>>>>> To give some background, I'm part of the RADICAL-Pilot [1] > development team. > >>>>>>>> RADICAL-Pilot is a Pilot System, an implementation of the Pilot > (job) concept, which is in its most minimal form takes care of the > decoupling of resource acquisition and workload management. > >>>>>>>> So instead of launching your real_science.exe through PBS, you > submit a Pilot, which will allow you to perform application level > scheduling. > >>>>>>>> Most obvious use-case if you want to run many (relatively) small > tasks, then you really don;t want to go through the batch system every > time. That is besides the fact that these machines are very bad in managing > many tasks anyway. > >>>>>>> > >>>>>>> Yeah, we sympathize. > >>>>>> > >>>>>> Thats always good :-) > >>>>>> > >>>>>>> Of course, one obvious solution is to get an allocation and > execute a shell script that runs the tasks within that allocation - yes? > >>>>>> > >>>>>> Not really. Most of our use-cases have dynamic runtime properties, > which means that at t=0 the exact workload is not known. > >>>>>> > >>>>>> In addition, I don't think such a script would allow me to work > around the aprun bottleneck, as I'm not aware of a way to start MPI tasks > that span multiple nodes from a Cray worker node. > >>>>>> > >>>>>>>> I looked a bit better at ORCM and it clearly overlaps with what I > want to achieve. > >>>>>>> > >>>>>>> Agreed. In ORCM, we allow a user to request a “session” that > results in allocation of resources. Each session is given an “orchestrator” > - the ORCM “shepherd” daemon - responsible for executing the individual > tasks across the assigned allocation, and a collection of “lamb” daemons > (one on each node of the allocation) that forms a distributed VM. The > orchestrator can execute the tasks very quickly since it doesn’t have to go > back to the scheduler, and we allow it to do so according to any provided > precedence requirement. Again, for simplicity, a shell script is the > default mechanism for submitting the individual tasks. > >>>>>> > >>>>>> Yeah, similar solution to a similar problem. > >>>>>> I noticed that Exascale is also part of the motivation? How does > this relate to the pmix effort? Different part of the stack I guess. > >>>>>> > >>>>>>>> One thing I noticed is that parts of it runs as root, why is that? > >>>>>>> > >>>>>>> ORCM is a full resource manager, which means it has a scheduler > (rudimentary today) and boot-time daemons that must run as root so they can > fork/exec the session-level daemons (that run at the user level). The > orchestrator and its daemons all run at the user-level. > >>>>>> > >>>>>> Ok. Our solution is user-space only, as one of our features is that > we are able to run across different type of systems. Both approaches come > with a tradeoff obviously. > >>>>>> > >>>>>>>>> We used to have a cmd line option in ORTE for what you propose - > it wouldn’t be too hard to restore. Is there some reason to do so? > >>>>>>>> > >>>>>>>> Can you point me to something that I could look for in the repo > history, then I can see if it serves my purpose. > >>>>>>> > >>>>>>> It would be back in the svn repo, I fear - would take awhile to > hunt it down. Basically, it just (a) started all the daemons to create a > VM, and (b) told mpirun to stick around as a persistent daemon. All > subsequent calls to mpirun would reference back to the persistent one, thus > using it to launch the jobs against the standing VM instead of starting a > new one every time. > >>>>>> > >>>>>> *nod* That's what I tried to do this afternoon actually with the > "--ompi-server", but that was not meant to be. > >>>>>> > >>>>>>> For ORCM, we just took that capability and expressed it as the > “shepherd” plus “lamb” daemon architecture described above. > >>>>>> > >>>>>> ACK. > >>>>>> > >>>>>>> If you don’t want to replace the base RM, then using ORTE to > establish a persistent VM is probably the way to go. > >>>>>> > >>>>>> Indeed, thats what it sounds like. Plus that ORTE is generic enough > that I can re-use it on other type of systems too. > >>>>>> > >>>>>>> I can probably make it do that again fairly readily. We have a > developer’s meeting next week, which usually means I have some free time > (during evenings and topics I’m not involved with), so I can take a crack > at this then if that would be timely enough. > >>>>>> > >>>>>> Happy to accept that offer. At this stage I'm not sure if I would > want a CLI or would be more interested to be able to do this > programmatically though. > >>>>>> Also more than willing to assist in any way I can. > >>>>>> > >>>>>> I tried to see how it all worked, but because of the modular nature > of ompi that was quite daunting. There is some learning curve I guess :-) > >>>>>> So it seems that mpirun is persistent, and opens up a listening > port, then some orded's get launched that phone home. > >>>>>> From there I got lost in the MCA maze. How do the tasks get unto > the compute nodes and started? > >>>>>> > >>>>>> Thanks a lot again, I appreciate your help. > >>>>>> > >>>>>> Cheers, > >>>>>> > >>>>>> Mark > >>>>>> _______________________________________________ > >>>>>> users mailing list > >>>>>> us...@open-mpi.org > >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/01/26227.php > >>>>> > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> us...@open-mpi.org > >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/01/26228.php > >>>> > >>>> _______________________________________________ > >>>> users mailing list > >>>> us...@open-mpi.org > >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/01/26229.php > >>> > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/02/26249.php > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/02/26254.php > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/02/26256.php > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/02/26258.php