Re: [OMPI devel] race condition in oob/tcp
Ralph,
yes and no ...
mpi hello world with four nodes can be used to reproduce the issue,
you can increase the likelyhood of producing the race condition by hacking
./opal/mca/event/libevent2021/libevent/poll.c
and replace
i = random() % nfds;
with
if (nfds < 2) {
i = 0;
} else {
i = nfds - 2;
}
but since this is really a race condition, all i could do is show you
how to use a debugger in order to force it
here is what really happens :
- thanks to your patch, when vpid 2 cannot read the acknowledge, this is
no more a fatal error.
- that being said, the peer->recv_event is not removed from the libevent
- later, send_event will be added to the libevent
- and then peer->recv_event will be added to the libevent
/* this is clearly not supported, and the interesting behaviour is that
peer->send_event will be kicked out of libevent (!) */
The attached patch fixes this race condition, could you please review it ?
Cheers,
Gilles
On 2014/09/17 22:17, Ralph Castain wrote:
> Do you have a reproducer you can share for testing this? I'm unable to get it
> to happen on my machine, but maybe you have a test code that triggers it so I
> can continue debugging
>
> Ralph
>
> On Sep 17, 2014, at 4:07 AM, Gilles Gouaillardet
> wrote:
>
>> Thanks Ralph,
>>
>> this is much better but there is still a bug :
>> with the very same scenario i described earlier, vpid 2 does not send
>> its message to vpid 3 once the connection has been established.
>>
>> i tried to debug it but i have been pretty unsuccessful so far ..
>>
>> vpid 2 calls tcp_peer_connected and execute the following snippet
>>
>> if (NULL != peer->send_msg && !peer->send_ev_active) {
>>opal_event_add(&peer->send_event, 0);
>>peer->send_ev_active = true;
>>}
>>
>> but when evmap_io_active is invoked later, the following part :
>>
>>TAILQ_FOREACH(ev, &ctx->events, ev_io_next) {
>>if (ev->ev_events & events)
>>event_active_nolock(ev, ev->ev_events & events, 1);
>>}
>>
>> finds only one ev (mca_oob_tcp_recv_handler and *no*
>> mca_oob_tcp_send_handler)
>>
>> i will resume my investigations tomorrow
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/09/17 4:01, Ralph Castain wrote:
>>> Hi Gilles
>>>
>>> I took a crack at solving this in r32744 - CMRd it for 1.8.3 and assigned
>>> it to you for review. Give it a try and let me know if I (hopefully) got it.
>>>
>>> The approach we have used in the past is to have both sides close their
>>> connections, and then have the higher vpid retry while the lower one waits.
>>> The logic for that was still in place, but it looks like you are hitting a
>>> different code path, and I found another potential one as well. So I think
>>> I plugged the holes, but will wait to hear if you confirm.
>>>
>>> Thanks
>>> Ralph
>>>
>>> On Sep 16, 2014, at 6:27 AM, Gilles Gouaillardet
>>> wrote:
>>>
Ralph,
here is the full description of a race condition in oob/tcp i very briefly
mentionned in a previous post :
the race condition can occur when two not connected orted try to send a
message to each other for the first time and at the same time.
that can occur when running mpi helloworld on 4 nodes with the grpcomm/rcd
module.
here is a scenario in which the race condition occurs :
orted vpid 2 and 3 enter the allgather
/* they are not orte yet oob/tcp connected*/
and they call orte.send_buffer_nb each other.
from a libevent point of view, vpid 2 and 3 will call
mca_oob_tcp_peer_try_connect
vpid 2 calls mca_oob_tcp_send_handler
vpid 3 calls connection_event_handler
depending on the value returned by random() in libevent, vpid 3 will
either call mca_oob_tcp_send_handler (likely) or recv_handler (unlikely)
if vpid 3 calls recv_handler, it will close the two sockets to vpid 2
then vpid 2 will call mca_oob_tcp_recv_handler
(peer->state is MCA_OOB_TCP_CONNECT_ACK)
that will invoke mca_oob_tcp_recv_connect_ack
tcp_peer_recv_blocking will fail
/* zero bytes are recv'ed since vpid 3 previously closed the socket before
writing a header */
and this is handled by mca_oob_tcp_recv_handler as a fatal error
/* ORTE_FORCED_TERMINATE(1) */
could you please have a look at it ?
if you are too busy, could you please advise where this scenario should be
handled differently ?
- should vpid 3 keep one socket instead of closing both and retrying ?
- should vpid 2 handle the failure as a non fatal error ?
Cheers,
Gilles
___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/09/15836.php
>>> ___
>
Re: [OMPI devel] race condition in oob/tcp
The patch looks fine to me - please go ahead and apply it. Thanks!
On Sep 17, 2014, at 11:35 PM, Gilles Gouaillardet
wrote:
> Ralph,
>
> yes and no ...
>
> mpi hello world with four nodes can be used to reproduce the issue,
>
>
> you can increase the likelyhood of producing the race condition by hacking
> ./opal/mca/event/libevent2021/libevent/poll.c
> and replace
>i = random() % nfds;
> with
> if (nfds < 2) {
> i = 0;
> } else {
> i = nfds - 2;
> }
>
> but since this is really a race condition, all i could do is show you
> how to use a debugger in order to force it
>
>
> here is what really happens :
> - thanks to your patch, when vpid 2 cannot read the acknowledge, this is
> no more a fatal error.
> - that being said, the peer->recv_event is not removed from the libevent
> - later, send_event will be added to the libevent
> - and then peer->recv_event will be added to the libevent
> /* this is clearly not supported, and the interesting behaviour is that
> peer->send_event will be kicked out of libevent (!) */
>
> The attached patch fixes this race condition, could you please review it ?
>
> Cheers,
>
> Gilles
>
> On 2014/09/17 22:17, Ralph Castain wrote:
>> Do you have a reproducer you can share for testing this? I'm unable to get
>> it to happen on my machine, but maybe you have a test code that triggers it
>> so I can continue debugging
>>
>> Ralph
>>
>> On Sep 17, 2014, at 4:07 AM, Gilles Gouaillardet
>> wrote:
>>
>>> Thanks Ralph,
>>>
>>> this is much better but there is still a bug :
>>> with the very same scenario i described earlier, vpid 2 does not send
>>> its message to vpid 3 once the connection has been established.
>>>
>>> i tried to debug it but i have been pretty unsuccessful so far ..
>>>
>>> vpid 2 calls tcp_peer_connected and execute the following snippet
>>>
>>> if (NULL != peer->send_msg && !peer->send_ev_active) {
>>> opal_event_add(&peer->send_event, 0);
>>> peer->send_ev_active = true;
>>> }
>>>
>>> but when evmap_io_active is invoked later, the following part :
>>>
>>> TAILQ_FOREACH(ev, &ctx->events, ev_io_next) {
>>> if (ev->ev_events & events)
>>> event_active_nolock(ev, ev->ev_events & events, 1);
>>> }
>>>
>>> finds only one ev (mca_oob_tcp_recv_handler and *no*
>>> mca_oob_tcp_send_handler)
>>>
>>> i will resume my investigations tomorrow
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On 2014/09/17 4:01, Ralph Castain wrote:
Hi Gilles
I took a crack at solving this in r32744 - CMRd it for 1.8.3 and assigned
it to you for review. Give it a try and let me know if I (hopefully) got
it.
The approach we have used in the past is to have both sides close their
connections, and then have the higher vpid retry while the lower one
waits. The logic for that was still in place, but it looks like you are
hitting a different code path, and I found another potential one as well.
So I think I plugged the holes, but will wait to hear if you confirm.
Thanks
Ralph
On Sep 16, 2014, at 6:27 AM, Gilles Gouaillardet
wrote:
> Ralph,
>
> here is the full description of a race condition in oob/tcp i very
> briefly mentionned in a previous post :
>
> the race condition can occur when two not connected orted try to send a
> message to each other for the first time and at the same time.
>
> that can occur when running mpi helloworld on 4 nodes with the
> grpcomm/rcd module.
>
> here is a scenario in which the race condition occurs :
>
> orted vpid 2 and 3 enter the allgather
> /* they are not orte yet oob/tcp connected*/
> and they call orte.send_buffer_nb each other.
> from a libevent point of view, vpid 2 and 3 will call
> mca_oob_tcp_peer_try_connect
>
> vpid 2 calls mca_oob_tcp_send_handler
>
> vpid 3 calls connection_event_handler
>
> depending on the value returned by random() in libevent, vpid 3 will
> either call mca_oob_tcp_send_handler (likely) or recv_handler (unlikely)
> if vpid 3 calls recv_handler, it will close the two sockets to vpid 2
>
> then vpid 2 will call mca_oob_tcp_recv_handler
> (peer->state is MCA_OOB_TCP_CONNECT_ACK)
> that will invoke mca_oob_tcp_recv_connect_ack
> tcp_peer_recv_blocking will fail
> /* zero bytes are recv'ed since vpid 3 previously closed the socket
> before writing a header */
> and this is handled by mca_oob_tcp_recv_handler as a fatal error
> /* ORTE_FORCED_TERMINATE(1) */
>
> could you please have a look at it ?
>
> if you are too busy, could you please advise where this scenario should
> be handled differently ?
> - should vpid 3 keep one socket instead of closing both and retrying ?
> - should vpid 2 handle the failure as a non fatal error ?
>
> Cheers,
>
Re: [OMPI devel] Need to know your Github ID
alex -> alex-ma alinas -> alinask amikheev -> alex-mikheev vasily -> vasilyMellanox On Wed, Sep 10, 2014 at 1:46 PM, Jeff Squyres (jsquyres) wrote: > As the next step of the planned migration to Github, I need to know: > > - Your Github ID (so that you can be added to the new OMPI git repo) > - Your SVN ID (so that I can map SVN->Github IDs, and therefore map Trac > tickets to appropriate owners) > > Here's the list of SVN IDs who have committed over the past year -- I'm > guessing that most of these people will need Github IDs: > > adrian > alekseys > alex > alinas > amikheev > bbenton > bosilca (done) > bouteill > brbarret > bwesarg > devendar > dgoodell (done) > edgar > eugene > ggouaillardet > hadi > hjelmn > hpcchris > hppritcha > igoru > jjhursey (done) > jladd > jroman > jsquyres (done) > jurenz > kliteyn > manjugv > miked (done) > mjbhaskar > mpiteam (done) > naughtont > osvegis > pasha > regrant > rfaucett > rhc (done) > rolfv (done) > samuel > shiqing > swise > tkordenbrock > vasily > vvenkates > vvenkatesan > yaeld > yosefe > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15788.php >
[OMPI devel] RFC: remove the --with-threads configure option
Folks, for both trunk and v1.8 branch, configure takes the --with-threads option. valid usages are --with-threads, --with-threads=yes, --with-threads=posix and --with-threads=no /* v1.6 used to support the --with-threads=solaris */ if we try to configure with --with-threads=no, this will result in a fatal error : checking for working POSIX threads package... yes checking for type of thread support... none configure: error: User requested MPI threads, but no threading model supported bottom line, the --with-threads configure option is useless in both v1.8 and trunk. is there any plan to support --with-threads=no in the future ? if no, i'd like to simply remove the --with-threads option Thanks in advance for your feedback Gilles FYI there is still some dead code / bug related to solaris threads, and this will be removed / fixed see https://svn.open-mpi.org/trac/ompi/ticket/4911
Re: [OMPI devel] Need to know your Github ID
ompimtttester On Thu, Sep 18, 2014 at 11:38 AM, Alex Margolin < alex.margo...@mail.huji.ac.il> wrote: > alex -> alex-ma > alinas -> alinask > > amikheev -> alex-mikheev > > vasily -> vasilyMellanox > > > On Wed, Sep 10, 2014 at 1:46 PM, Jeff Squyres (jsquyres) < > jsquy...@cisco.com> wrote: > >> As the next step of the planned migration to Github, I need to know: >> >> - Your Github ID (so that you can be added to the new OMPI git repo) >> - Your SVN ID (so that I can map SVN->Github IDs, and therefore map Trac >> tickets to appropriate owners) >> >> Here's the list of SVN IDs who have committed over the past year -- I'm >> guessing that most of these people will need Github IDs: >> >> adrian >> alekseys >> alex >> alinas >> amikheev >> bbenton >> bosilca (done) >> bouteill >> brbarret >> bwesarg >> devendar >> dgoodell (done) >> edgar >> eugene >> ggouaillardet >> hadi >> hjelmn >> hpcchris >> hppritcha >> igoru >> jjhursey (done) >> jladd >> jroman >> jsquyres (done) >> jurenz >> kliteyn >> manjugv >> miked (done) >> mjbhaskar >> mpiteam (done) >> naughtont >> osvegis >> pasha >> regrant >> rfaucett >> rhc (done) >> rolfv (done) >> samuel >> shiqing >> swise >> tkordenbrock >> vasily >> vvenkates >> vvenkatesan >> yaeld >> yosefe >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/09/15788.php >> > > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15864.php >
Re: [OMPI devel] RFC: remove the --with-threads configure option
I'm unaware of a use case for --without-threads. Does anyone have one? If not, then I think removing all that dead code would be a Good Thing. On Sep 18, 2014, at 7:03 AM, Gilles Gouaillardet wrote: > Folks, > > for both trunk and v1.8 branch, configure takes the --with-threads option. > valid usages are > --with-threads, --with-threads=yes, --with-threads=posix and > --with-threads=no > > /* v1.6 used to support the --with-threads=solaris */ > > if we try to configure with --with-threads=no, this will result in a > fatal error : > > checking for working POSIX threads package... yes > checking for type of thread support... none > configure: error: User requested MPI threads, but no threading model > supported > > bottom line, the --with-threads configure option is useless in both v1.8 > and trunk. > > is there any plan to support --with-threads=no in the future ? > if no, i'd like to simply remove the --with-threads option > > Thanks in advance for your feedback > > Gilles > > FYI > there is still some dead code / bug related to solaris threads, and this > will be removed / fixed > see https://svn.open-mpi.org/trac/ompi/ticket/4911 > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15865.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] External loopback
Mellanox / LANL -- Can you guys look into this? Thanks. On Sep 15, 2014, at 12:38 PM, Håkon Bugge wrote: > From time-to-time, and have a need for running Open MPI apps using the openib > btl on a single node, where port 1 on the HCA is connected to port 2 on the > same HCA. > > Using a vintage 1.5.4, my command line would read: > > mpiexec --mca btl self,openib --mca btl_openib_cpc_include oob \ > -np 1 /usr/bin/env OMPI_MCA_btl_openib_if_include=mlx4_0:1 ./a.out : \ > -np 1 /usr/bin/env OMPI_MCA_btl_openib_if_include=mlx4_0:2 ./a.out > > > Now, I had a need for a newer Open MPI, and compiled and installed version > 1.8.2. Now the problems began ;-) Apparently, the old (and in my opinion > nice)"oob" connection management method has disappeared. However, by > modifying the command line to: > > mpiexec --mca btl self,openib --mca btl_openib_cpc_include udcm \ > -np 1 /usr/bin/env OMPI_MCA_btl_openib_if_include=mlx4_0:1 ./a.out : \ > -np 1 /usr/bin/env OMPI_MCA_btl_openib_if_include=mlx4_0:2 ./a.out > > > I get tons of: > > connect/btl_openib_connect_udcm.c:1390:udcm_find_endpoint] could not find > endpoint with port: 1, lid: 4608, msg_type: 100 > > Interestingly, the lid here is the lid for Port 2 (when port numbers start at > 1). I do suspect that the printout above counts ports from zero. > > Anyway, must I get back to an older Open MPI supporting "oob", or do I have a > flaw in my command line? > > > Thanks, Håkon > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15829.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] OPAL timing framework
I have a few comments: - This looks nice. Thanks for the contribution. - I notice that the ORTE timing stuff is now a compile-time decision, not a run-time decision. Do we care that we've now taken away the ability for users to do timings in a production build? - "clksync" -- can we use "clocksync"? It's only 2 letters. We tend to use real words in the OMPI code base; unnecessary abbreviation should be avoided. - r32738 introduced a several files into the code base that have no copyrights, and do not have the standard OMPI copyright header block. Please fix. - There's no documentation on how to use mpisync, mpirun_prof, or ompi_timing_post, even though they're installed when you --enable-timing. What are these 3 executables? Can we get man pages? - What's the purpose of the MCA param orte_rml_base_timing? A *quick* look through the code seems to indicate that it is ignored. - What's the purpose of the MCA params opal_clksync_file, opal_timing_file, and opal_timing_overhead? E.g., what is a "clksync" file, what is it for, and what is its format? Does the user have to provide one? If so, how to you get one? Or is it an output file? ...etc. The brief descriptions given in the MCA help strings don't really provide enough information for someone who has no idea what the timing stuff is. Also, can those 3 params have a common prefix? I.e., it's not obvious that opal_clksync_file is related to opal_timing_* at all. - A *quick* look at ompi/tools/mpisync shows that a bunch of that code came from an external project. Is the license compatible with OMPI's license? What do we need to do to conform to their license? - opal/util/timings.h is protected by OPAL_SYS_TIMING_H -- shouldn't it be OPAL_UTIL_TIMINGS_H? - There's commented-out code in opal/util/timings.h. - There's no doxygen-style documentation in opal/util/timings.h to tell developers how to use it. - There's "TODO" comments in opal/util/timings.c; should those be fixed? - opal_config.h should be the first include in opal/util/timings.c. - If timing support is not to be compiled in, then opal/util/timings.c should not be be compiled via the Makefile.am (rather than entirely #if'ed out). It looks like this work is about 95% complete. Finishing the remaining 5% would make it great and genuinely useful to the rest of the code base. Thanks! On Sep 16, 2014, at 10:20 AM, Artem Polyakov wrote: > Hello, > > I would like to introduce OMPI timing framework that was included into the > trunk yesterday (r32738). The code is new so if you'll hit some bugs - just > let me know. > > The framework consists of the set of macro's and routines for internal OMPI > usage + standalone tool mpisync and few additional scripts: mpirun_prof and > ompi_timing_post. The set of features is very basic and I am open for > discussion of new things that are desirable there. > > To enable framework compilation you should configure OMPI with > --enable-timing option. If the option was passed to ./configure, standalone > tools and scripts will be installed into /bin. > > The timing code is located in OPAL (opal/utils/timing.[ch]). There is a set > of macro's that should be used to preprocess out all mentions of the timing > code in case it wasn't requested with --enable-timing: > OPAL_TIMING_DECLARE(t) - declare timing handler structure with name "t". > OPAL_TIMING_DECLARE_EXT(x, t) - external declaration of a timing handler "t". > OPAL_TIMING_INIT(t) - initialize timing handler "t" > OPAL_TIMING_EVENT(x) - printf-like event declaration similar to OPAL_OUTPUT. > The information about the event will be quickly inserted into the linked > list. Maximum event description is limited by OPAL_TIMING_DESCR_MAX. > The malloc is performed in buckets (OPAL_TIMING_BUFSIZE at once) and overhead > (time to malloc and prepare the bucket) is accounted in corresponding list > element. It might be excluded from the timing results (controlled by > OMPI_MCA_opal_timing_overhead parameter). > OPAL_TIMING_REPORT(enable, t, prefix) - prepare and print out timing > information. If OMPI_MCA_opal_timing_file was specified the output will go to > that file. In other case the output will be directed using opal_output, each > line will be prefixed with "prefix" to ease grep'ing. "enable" is a > boolean/integer variable that is used for runtime selection of what should be > reported. > OPAL_TIMING_RELEASE(t) - the counterpart for OPAL_TIMING_INIT. > > There are several examples in OMPI code. And here is another simple example: > OPAL_TIMING_DECLARE(tm); > OPAL_TIMING_INIT(&tm); > ... > OPAL_TIMING_EVENT((&tm,"Begin of timing: %s", > ORTE_NAME_PRINT(&(peer->name)) )); > > OPAL_TIMING_EVENT((&tm,"Next timing event with condition x = %d", x )); > ... > OPAL_TIMING_EVENT((&tm,"Finish")); > OPAL_TIMING_REPORT(enable_var, &tm,"MPI Init"); > OPAL_TIMING_RELEASE(&tm); > > > An output from all OMPI p
Re: [OMPI devel] OPAL timing framework
Jeff, thank you for the feedback! All of mentioned issues are clear and I will fix them shortly. One important thing that needs additional discussion is compile-time vs runtime selection. Ralph, what do you think about that? Several of issues depends on that decision. 2014-09-18 20:09 GMT+07:00 Jeff Squyres (jsquyres) : > I have a few comments: > > - This looks nice. Thanks for the contribution. > > - I notice that the ORTE timing stuff is now a compile-time decision, not > a run-time decision. Do we care that we've now taken away the ability for > users to do timings in a production build? - "clksync" -- can we use "clocksync"? It's only 2 letters. We tend to > use real words in the OMPI code base; unnecessary abbreviation should be > avoided. > - r32738 introduced a several files into the code base that have no > copyrights, and do not have the standard OMPI copyright header block. > Please fix. > > - There's no documentation on how to use mpisync, mpirun_prof, or > ompi_timing_post, even though they're installed when you --enable-timing. > What are these 3 executables? Can we get man pages? > I post their description in the first e-mail. Sure I can prepare man pages for them, > > - What's the purpose of the MCA param orte_rml_base_timing? A *quick* > look through the code seems to indicate that it is ignored. > > - What's the purpose of the MCA params opal_clksync_file, > opal_timing_file, and opal_timing_overhead? E.g., what is a "clksync" > file, what is it for, and what is its format? Does the user have to > provide one? If so, how to you get one? Or is it an output file? > ...etc. The brief descriptions given in the MCA help strings don't really > provide enough information for someone who has no idea what the timing > stuff is. Also, can those 3 params have a common prefix? I.e., it's not > obvious that opal_clksync_file is related to opal_timing_* at all. > - A *quick* look at ompi/tools/mpisync shows that a bunch of that code > came from an external project. Is the license compatible with OMPI's > license? What do we need to do to conform to their license? > > - opal/util/timings.h is protected by OPAL_SYS_TIMING_H -- shouldn't it be > OPAL_UTIL_TIMINGS_H? > > - There's commented-out code in opal/util/timings.h. > > - There's no doxygen-style documentation in opal/util/timings.h to tell > developers how to use it. > > - There's "TODO" comments in opal/util/timings.c; should those be fixed? > > - opal_config.h should be the first include in opal/util/timings.c. > > - If timing support is not to be compiled in, then opal/util/timings.c > should not be be compiled via the Makefile.am (rather than entirely #if'ed > out). > > It looks like this work is about 95% complete. Finishing the remaining 5% > would make it great and genuinely useful to the rest of the code base. > > Thanks! > > > > On Sep 16, 2014, at 10:20 AM, Artem Polyakov wrote: > > > Hello, > > > > I would like to introduce OMPI timing framework that was included into > the trunk yesterday (r32738). The code is new so if you'll hit some bugs - > just let me know. > > > > The framework consists of the set of macro's and routines for internal > OMPI usage + standalone tool mpisync and few additional scripts: > mpirun_prof and ompi_timing_post. The set of features is very basic and I > am open for discussion of new things that are desirable there. > > > > To enable framework compilation you should configure OMPI with > --enable-timing option. If the option was passed to ./configure, standalone > tools and scripts will be installed into /bin. > > > > The timing code is located in OPAL (opal/utils/timing.[ch]). There is a > set of macro's that should be used to preprocess out all mentions of the > timing code in case it wasn't requested with --enable-timing: > > OPAL_TIMING_DECLARE(t) - declare timing handler structure with name "t". > > OPAL_TIMING_DECLARE_EXT(x, t) - external declaration of a timing handler > "t". > > OPAL_TIMING_INIT(t) - initialize timing handler "t" > > OPAL_TIMING_EVENT(x) - printf-like event declaration similar to > OPAL_OUTPUT. > > The information about the event will be quickly inserted into the linked > list. Maximum event description is limited by OPAL_TIMING_DESCR_MAX. > > The malloc is performed in buckets (OPAL_TIMING_BUFSIZE at once) and > overhead (time to malloc and prepare the bucket) is accounted in > corresponding list element. It might be excluded from the timing results > (controlled by OMPI_MCA_opal_timing_overhead parameter). > > OPAL_TIMING_REPORT(enable, t, prefix) - prepare and print out timing > information. If OMPI_MCA_opal_timing_file was specified the output will go > to that file. In other case the output will be directed using opal_output, > each line will be prefixed with "prefix" to ease grep'ing. "enable" is a > boolean/integer variable that is used for runtime selection of what should > be reported. > > OPAL_TIMING_RELEASE(t) - the counterpart for OPAL_TIMING_INIT
Re: [OMPI devel] CONVERSION TO GITHUB
On Sep 17, 2014, at 3:21 PM, Paul Hargrove wrote: > You sent out email asking for SVN -> github ID mapping, but did NOT ask about > IDs for Trac users who are not SVN users. > So my (minor) concern was over what happens to tickets assigned to Trac users > you didn't collect github IDs for? Good question. What I should have done was extracted the ticket owners from the Trac tickets, and ensured that I had trac/svn->github mappings for those. I've done that now. Here's the IDs I don't have -- most of them are not active in OMPI any more: adi afriedle brbarret gshipman hppritcha --> I just pinged Howard jdmason rlgraham shiqing So these tickets will end up being orphaned/assigned to "ompiteam". I think we'll look at these after the import and decide if they're really important or not. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] OPAL timing framework
I believe compile-time is preferable as there is a non-zero time impact of enabling this code. It's really more for developers to improve scalability - if a user is actually interested, I think it isn't that hard for them to configure it. On Sep 18, 2014, at 7:16 AM, Artem Polyakov wrote: > Jeff, thank you for the feedback! All of mentioned issues are clear and I > will fix them shortly. > > One important thing that needs additional discussion is compile-time vs > runtime selection. Ralph, what do you think about that? Several of issues > depends on that decision. > > 2014-09-18 20:09 GMT+07:00 Jeff Squyres (jsquyres) : > I have a few comments: > > - This looks nice. Thanks for the contribution. > > - I notice that the ORTE timing stuff is now a compile-time decision, not a > run-time decision. Do we care that we've now taken away the ability for > users to do timings in a production build? > - "clksync" -- can we use "clocksync"? It's only 2 letters. We tend to use > real words in the OMPI code base; unnecessary abbreviation should be avoided. > > - r32738 introduced a several files into the code base that have no > copyrights, and do not have the standard OMPI copyright header block. Please > fix. > > - There's no documentation on how to use mpisync, mpirun_prof, or > ompi_timing_post, even though they're installed when you --enable-timing. > What are these 3 executables? Can we get man pages? > I post their description in the first e-mail. Sure I can prepare man pages > for them, > > > - What's the purpose of the MCA param orte_rml_base_timing? A *quick* look > through the code seems to indicate that it is ignored. > > - What's the purpose of the MCA params opal_clksync_file, opal_timing_file, > and opal_timing_overhead? E.g., what is a "clksync" file, what is it for, > and what is its format? Does the user have to provide one? If so, how to > you get one? Or is it an output file? ...etc. The brief descriptions given > in the MCA help strings don't really provide enough information for someone > who has no idea what the timing stuff is. Also, can those 3 params have a > common prefix? I.e., it's not obvious that opal_clksync_file is related to > opal_timing_* at all. > > - A *quick* look at ompi/tools/mpisync shows that a bunch of that code came > from an external project. Is the license compatible with OMPI's license? > What do we need to do to conform to their license? > > - opal/util/timings.h is protected by OPAL_SYS_TIMING_H -- shouldn't it be > OPAL_UTIL_TIMINGS_H? > > - There's commented-out code in opal/util/timings.h. > > - There's no doxygen-style documentation in opal/util/timings.h to tell > developers how to use it. > > - There's "TODO" comments in opal/util/timings.c; should those be fixed? > > - opal_config.h should be the first include in opal/util/timings.c. > > - If timing support is not to be compiled in, then opal/util/timings.c should > not be be compiled via the Makefile.am (rather than entirely #if'ed out). > > It looks like this work is about 95% complete. Finishing the remaining 5% > would make it great and genuinely useful to the rest of the code base. > > Thanks! > > > > On Sep 16, 2014, at 10:20 AM, Artem Polyakov wrote: > > > Hello, > > > > I would like to introduce OMPI timing framework that was included into the > > trunk yesterday (r32738). The code is new so if you'll hit some bugs - just > > let me know. > > > > The framework consists of the set of macro's and routines for internal OMPI > > usage + standalone tool mpisync and few additional scripts: mpirun_prof and > > ompi_timing_post. The set of features is very basic and I am open for > > discussion of new things that are desirable there. > > > > To enable framework compilation you should configure OMPI with > > --enable-timing option. If the option was passed to ./configure, standalone > > tools and scripts will be installed into /bin. > > > > The timing code is located in OPAL (opal/utils/timing.[ch]). There is a set > > of macro's that should be used to preprocess out all mentions of the timing > > code in case it wasn't requested with --enable-timing: > > OPAL_TIMING_DECLARE(t) - declare timing handler structure with name "t". > > OPAL_TIMING_DECLARE_EXT(x, t) - external declaration of a timing handler > > "t". > > OPAL_TIMING_INIT(t) - initialize timing handler "t" > > OPAL_TIMING_EVENT(x) - printf-like event declaration similar to OPAL_OUTPUT. > > The information about the event will be quickly inserted into the linked > > list. Maximum event description is limited by OPAL_TIMING_DESCR_MAX. > > The malloc is performed in buckets (OPAL_TIMING_BUFSIZE at once) and > > overhead (time to malloc and prepare the bucket) is accounted in > > corresponding list element. It might be excluded from the timing results > > (controlled by OMPI_MCA_opal_timing_overhead parameter). > > OPAL_TIMING_REPORT(enable, t, prefix) - prepare and p
Re: [OMPI devel] RFC: remove the --with-threads configure option
None that I can think of - I'd remove it On Sep 18, 2014, at 5:23 AM, Jeff Squyres (jsquyres) wrote: > I'm unaware of a use case for --without-threads. > > Does anyone have one? > > If not, then I think removing all that dead code would be a Good Thing. > > > On Sep 18, 2014, at 7:03 AM, Gilles Gouaillardet > wrote: > >> Folks, >> >> for both trunk and v1.8 branch, configure takes the --with-threads option. >> valid usages are >> --with-threads, --with-threads=yes, --with-threads=posix and >> --with-threads=no >> >> /* v1.6 used to support the --with-threads=solaris */ >> >> if we try to configure with --with-threads=no, this will result in a >> fatal error : >> >> checking for working POSIX threads package... yes >> checking for type of thread support... none >> configure: error: User requested MPI threads, but no threading model >> supported >> >> bottom line, the --with-threads configure option is useless in both v1.8 >> and trunk. >> >> is there any plan to support --with-threads=no in the future ? >> if no, i'd like to simply remove the --with-threads option >> >> Thanks in advance for your feedback >> >> Gilles >> >> FYI >> there is still some dead code / bug related to solaris threads, and this >> will be removed / fixed >> see https://svn.open-mpi.org/trac/ompi/ticket/4911 >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/09/15865.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15867.php
Re: [OMPI devel] OPAL timing framework
Got that. Thank you! четверг, 18 сентября 2014 г. пользователь Ralph Castain написал: > I believe compile-time is preferable as there is a non-zero time impact of > enabling this code. It's really more for developers to improve scalability > - if a user is actually interested, I think it isn't that hard for them to > configure it. > > > On Sep 18, 2014, at 7:16 AM, Artem Polyakov > wrote: > > Jeff, thank you for the feedback! All of mentioned issues are clear and I > will fix them shortly. > > One important thing that needs additional discussion is compile-time vs > runtime selection. Ralph, what do you think about that? Several of issues > depends on that decision. > > 2014-09-18 20:09 GMT+07:00 Jeff Squyres (jsquyres) >: > >> I have a few comments: >> >> - This looks nice. Thanks for the contribution. >> >> - I notice that the ORTE timing stuff is now a compile-time decision, not >> a run-time decision. Do we care that we've now taken away the ability for >> users to do timings in a production build? > > - "clksync" -- can we use "clocksync"? It's only 2 letters. We tend to >> use real words in the OMPI code base; unnecessary abbreviation should be >> avoided. > > >> - r32738 introduced a several files into the code base that have no >> copyrights, and do not have the standard OMPI copyright header block. >> Please fix. >> >> - There's no documentation on how to use mpisync, mpirun_prof, or >> ompi_timing_post, even though they're installed when you --enable-timing. >> What are these 3 executables? Can we get man pages? >> > I post their description in the first e-mail. Sure I can prepare man pages > for them, > > >> >> - What's the purpose of the MCA param orte_rml_base_timing? A *quick* >> look through the code seems to indicate that it is ignored. >> >> - What's the purpose of the MCA params opal_clksync_file, >> opal_timing_file, and opal_timing_overhead? E.g., what is a "clksync" >> file, what is it for, and what is its format? Does the user have to >> provide one? If so, how to you get one? Or is it an output file? >> ...etc. The brief descriptions given in the MCA help strings don't really >> provide enough information for someone who has no idea what the timing >> stuff is. Also, can those 3 params have a common prefix? I.e., it's not >> obvious that opal_clksync_file is related to opal_timing_* at all. > > >> - A *quick* look at ompi/tools/mpisync shows that a bunch of that code >> came from an external project. Is the license compatible with OMPI's >> license? What do we need to do to conform to their license? >> >> - opal/util/timings.h is protected by OPAL_SYS_TIMING_H -- shouldn't it >> be OPAL_UTIL_TIMINGS_H? >> >> - There's commented-out code in opal/util/timings.h. >> >> - There's no doxygen-style documentation in opal/util/timings.h to tell >> developers how to use it. >> >> - There's "TODO" comments in opal/util/timings.c; should those be fixed? >> >> - opal_config.h should be the first include in opal/util/timings.c. >> >> - If timing support is not to be compiled in, then opal/util/timings.c >> should not be be compiled via the Makefile.am (rather than entirely #if'ed >> out). >> >> It looks like this work is about 95% complete. Finishing the remaining >> 5% would make it great and genuinely useful to the rest of the code base. >> >> Thanks! >> >> >> >> On Sep 16, 2014, at 10:20 AM, Artem Polyakov > > wrote: >> >> > Hello, >> > >> > I would like to introduce OMPI timing framework that was included into >> the trunk yesterday (r32738). The code is new so if you'll hit some bugs - >> just let me know. >> > >> > The framework consists of the set of macro's and routines for internal >> OMPI usage + standalone tool mpisync and few additional scripts: >> mpirun_prof and ompi_timing_post. The set of features is very basic and I >> am open for discussion of new things that are desirable there. >> > >> > To enable framework compilation you should configure OMPI with >> --enable-timing option. If the option was passed to ./configure, standalone >> tools and scripts will be installed into /bin. >> > >> > The timing code is located in OPAL (opal/utils/timing.[ch]). There is a >> set of macro's that should be used to preprocess out all mentions of the >> timing code in case it wasn't requested with --enable-timing: >> > OPAL_TIMING_DECLARE(t) - declare timing handler structure with name "t". >> > OPAL_TIMING_DECLARE_EXT(x, t) - external declaration of a timing >> handler "t". >> > OPAL_TIMING_INIT(t) - initialize timing handler "t" >> > OPAL_TIMING_EVENT(x) - printf-like event declaration similar to >> OPAL_OUTPUT. >> > The information about the event will be quickly inserted into the >> linked list. Maximum event description is limited by OPAL_TIMING_DESCR_MAX. >> > The malloc is performed in buckets (OPAL_TIMING_BUFSIZE at once) and >> overhead (time to malloc and prepare the bucket) is accounted in >> corresponding list element. It might be excluded from the timing results >
Re: [OMPI devel] RFC: remove the --with-threads configure option
I second the motion. -Original Message- From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Ralph Castain Sent: Thursday, September 18, 2014 8:43 AM To: Open MPI Developers Subject: Re: [OMPI devel] RFC: remove the --with-threads configure option None that I can think of - I'd remove it On Sep 18, 2014, at 5:23 AM, Jeff Squyres (jsquyres) wrote: > I'm unaware of a use case for --without-threads. > > Does anyone have one? > > If not, then I think removing all that dead code would be a Good Thing. > > > On Sep 18, 2014, at 7:03 AM, Gilles Gouaillardet > wrote: > >> Folks, >> >> for both trunk and v1.8 branch, configure takes the --with-threads option. >> valid usages are >> --with-threads, --with-threads=yes, --with-threads=posix and >> --with-threads=no >> >> /* v1.6 used to support the --with-threads=solaris */ >> >> if we try to configure with --with-threads=no, this will result in a >> fatal error : >> >> checking for working POSIX threads package... yes checking for type >> of thread support... none >> configure: error: User requested MPI threads, but no threading model >> supported >> >> bottom line, the --with-threads configure option is useless in both >> v1.8 and trunk. >> >> is there any plan to support --with-threads=no in the future ? >> if no, i'd like to simply remove the --with-threads option >> >> Thanks in advance for your feedback >> >> Gilles >> >> FYI >> there is still some dead code / bug related to solaris threads, and >> this will be removed / fixed see >> https://svn.open-mpi.org/trac/ompi/ticket/4911 >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/09/15865.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15867.php ___ devel mailing list de...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post: http://www.open-mpi.org/community/lists/devel/2014/09/15873.php
Re: [OMPI devel] Need to know your Github ID
I seem to have two ids, vvenkates and vvenkatesan I prefer to keep vvenkatesan Thanks Vish On Sep 18, 2014, at 06:34 AM, Alina Sklarevich wrote: ompimtttester On Thu, Sep 18, 2014 at 11:38 AM, Alex Margolin wrote: alex -> alex-ma alinas -> alinask amikheev -> alex-mikheev vasily -> vasilyMellanox On Wed, Sep 10, 2014 at 1:46 PM, Jeff Squyres (jsquyres) wrote: As the next step of the planned migration to Github, I need to know: - Your Github ID (so that you can be added to the new OMPI git repo) - Your SVN ID (so that I can map SVN->Github IDs, and therefore map Trac tickets to appropriate owners) Here's the list of SVN IDs who have committed over the past year -- I'm guessing that most of these people will need Github IDs: adrian alekseys alex alinas amikheev bbenton bosilca (done) bouteill brbarret bwesarg devendar dgoodell (done) edgar eugene ggouaillardet hadi hjelmn hpcchris hppritcha igoru jjhursey (done) jladd jroman jsquyres (done) jurenz kliteyn manjugv miked (done) mjbhaskar mpiteam (done) naughtont osvegis pasha regrant rfaucett rhc (done) rolfv (done) samuel shiqing swise tkordenbrock vasily vvenkates vvenkatesan yaeld yosefe -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ ___ devel mailing list de...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post: http://www.open-mpi.org/community/lists/devel/2014/09/15788.php ___ devel mailing list de...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post: http://www.open-mpi.org/community/lists/devel/2014/09/15864.php ___ devel mailing list de...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post: http://www.open-mpi.org/community/lists/devel/2014/09/15866.php
Re: [OMPI devel] Need to know your Github ID
On Sep 18, 2014, at 11:35 AM, Vishwanath Venkatesan wrote: > I seem to have two ids, vvenkates and vvenkatesan > I prefer to keep vvenkatesan Are you referring to your Github or SVN IDs? > Thanks > Vish > > On Sep 18, 2014, at 06:34 AM, Alina Sklarevich > wrote: > >> ompimtttester >> >> On Thu, Sep 18, 2014 at 11:38 AM, Alex Margolin >> wrote: >> alex -> alex-ma >> alinas -> alinask >> amikheev -> alex-mikheev >> >> vasily -> vasilyMellanox >> >> >> >> On Wed, Sep 10, 2014 at 1:46 PM, Jeff Squyres (jsquyres) >> wrote: >> As the next step of the planned migration to Github, I need to know: >> >> - Your Github ID (so that you can be added to the new OMPI git repo) >> - Your SVN ID (so that I can map SVN->Github IDs, and therefore map Trac >> tickets to appropriate owners) >> >> Here's the list of SVN IDs who have committed over the past year -- I'm >> guessing that most of these people will need Github IDs: >> >> adrian >> alekseys >> alex >> alinas >> amikheev >> bbenton >> bosilca (done) >> bouteill >> brbarret >> bwesarg >> devendar >> dgoodell (done) >> edgar >> eugene >> ggouaillardet >> hadi >> hjelmn >> hpcchris >> hppritcha >> igoru >> jjhursey (done) >> jladd >> jroman >> jsquyres (done) >> jurenz >> kliteyn >> manjugv >> miked (done) >> mjbhaskar >> mpiteam (done) >> naughtont >> osvegis >> pasha >> regrant >> rfaucett >> rhc (done) >> rolfv (done) >> samuel >> shiqing >> swise >> tkordenbrock >> vasily >> vvenkates >> vvenkatesan >> yaeld >> yosefe >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/09/15788.php >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/09/15864.php >> >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/09/15866.php > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15876.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Need to know your Github ID
tkordenbrock -> tkordenbrock On Wed, Sep 10, 2014 at 5:46 AM, Jeff Squyres (jsquyres) wrote: > As the next step of the planned migration to Github, I need to know: > > - Your Github ID (so that you can be added to the new OMPI git repo) > - Your SVN ID (so that I can map SVN->Github IDs, and therefore map Trac > tickets to appropriate owners) > > Here's the list of SVN IDs who have committed over the past year -- I'm > guessing that most of these people will need Github IDs: > > adrian > alekseys > alex > alinas > amikheev > bbenton > bosilca (done) > bouteill > brbarret > bwesarg > devendar > dgoodell (done) > edgar > eugene > ggouaillardet > hadi > hjelmn > hpcchris > hppritcha > igoru > jjhursey (done) > jladd > jroman > jsquyres (done) > jurenz > kliteyn > manjugv > miked (done) > mjbhaskar > mpiteam (done) > naughtont > osvegis > pasha > regrant > rfaucett > rhc (done) > rolfv (done) > samuel > shiqing > swise > tkordenbrock > vasily > vvenkates > vvenkatesan > yaeld > yosefe > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/09/15788.php >
