[OMPI devel] Warnings on master

[r...@open-mpi.org]

We seem to have gotten into a state again of generating a ton of warnings on 
master - can folks take a look at these and clean them up?

opal_datatype_pack.c: In function ‘pack_predefined_heterogeneous’:
opal_datatype_pack.c:421:24: warning: variable ‘_l_blength’ set but not used 
[-Wunused-but-set-variable]
 size_t _r_blength, _l_blength;
^~
opal_datatype_pack.c: In function ‘opal_pack_general_checksum’:
opal_datatype_pack.c:467:9: warning: variable ‘type’ set but not used 
[-Wunused-but-set-variable]
 int type;
 ^~~~
opal_datatype_pack.c: In function ‘pack_predefined_heterogeneous’:
opal_datatype_pack.c:421:24: warning: variable ‘_l_blength’ set but not used 
[-Wunused-but-set-variable]
 size_t _r_blength, _l_blength;
^~
opal_datatype_pack.c: In function ‘opal_pack_general’:
opal_datatype_pack.c:467:9: warning: variable ‘type’ set but not used 
[-Wunused-but-set-variable]
 int type;
 ^~~~


topology.c: In function ‘hwloc__check_children’:
topology.c:3126:9: warning: variable ‘prev_firstchild’ set but not used 
[-Wunused-but-set-variable]
 int prev_firstchild = -1; /* -1 works fine with first comparisons below */
 ^~~
At top level:
topology.c:637:31: warning: ‘obj_order_type’ defined but not used 
[-Wunused-const-variable=]
 static const hwloc_obj_type_t obj_order_type[] = {
   ^~
topology-xml.c: In function ‘opal_hwloc1113_hwloc_export_obj_userdata’:
topology-xml.c:1632:5: warning: ‘realname’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]
 hwloc__export_obj_userdata(state, encoded, realname, length, buffer, 
encoded_length);
 
^~~~
topology-xml.c:1632:5: warning: ‘encoded_length’ may be used uninitialized in 
this function [-Wmaybe-uninitialized]
topology-xml.c:1632:5: warning: ‘encoded’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]


class/opal_free_list.c: In function ‘opal_free_list_grow_st’:
class/opal_free_list.c:217:21: warning: ‘align’ may be used uninitialized in 
this function [-Wmaybe-uninitialized]
 payload_ptr = (unsigned char *) 
flist->fl_mpool->mpool_alloc(flist->fl_mpool, buffer_size, align, 0);
 
^~~~
class/opal_free_list.c:217:21: warning: ‘buffer_size’ may be used uninitialized 
in this function [-Wmaybe-uninitialized]


server/pmix_server.c: In function ‘PMIx_server_init’:
server/pmix_server.c:438:83: warning: ‘tl’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]
 pmix_show_help("help-pmix-server.txt", "listener-failed-start", 
true, tl->address.sun_path);

   ^~


rcache_grdma_module.c: In function ‘mca_rcache_grdma_check_cached’:
rcache_grdma_module.c:235:17: warning: unused variable ‘ref_cnt’ 
[-Wunused-variable]
 int32_t ref_cnt = opal_atomic_add_32 (&grdma_reg->ref_count, 1);
 ^~~
runtime/orte_init.c:129:2: warning: #ident is a GCC extension


coll_basic_reduce_scatter.c: In function ‘mca_coll_basic_reduce_scatter_inter’:
coll_basic_reduce_scatter.c:474:9: warning: ‘lbuf’ may be used uninitialized in 
this function [-Wmaybe-uninitialized]
 err = comm->c_local_comm->c_coll.coll_scatterv(lbuf, rcounts, disps, dtype,
 ^~~
rbuf, rcounts[rank], dtype, 0,
~~
comm->c_local_comm,
~~~
comm->c_local_comm->c_coll.coll_scatterv_module);

coll_basic_alltoallw.c: In function ‘mca_coll_basic_alltoallw_intra’:
coll_basic_alltoallw.c:71:16: warning: ‘gap’ may be used uninitialized in this 
function [-Wmaybe-uninitialized]
 tmp_buffer -= gap;
^~
coll_basic_alltoallw.c:47:20: note: ‘gap’ was declared here
 ptrdiff_t ext, gap;
^~~


In file included from osc_pt2pt_module.c:23:0:
osc_pt2pt.h: In function ‘osc_pt2pt_gc_clean’:
osc_pt2pt.h:668:21: warning: unused variable ‘request’ [-Wunused-variable]
 ompi_request_t *request;
 ^~~
osc_pt2pt_module.c: In function ‘ompi_osc_pt2pt_free’:
osc_pt2pt_module.c:96:28: warning: comparison between signed and unsigned 
integer expressions [-Wsign-compare]
 for (int i = 0 ; i < module->recv_frag_count ; ++i) {
^
In file included from osc_pt2pt_frag.c:15:0:
osc_pt2pt.h: In function ‘osc_pt2pt_gc_clean’:
osc_pt2pt.h:668:21: warning: unused variable ‘request’ [-Wunused-variable]
 ompi_request_t *request;
 ^~~
In file included from osc_pt2pt_active_target.c:26:0:
osc_pt2pt.h: In functi

Re: [OMPI devel] Coll/sync component missing???

[George Bosilca]

Ralph,

Bringing back the coll/sync is a cheap shot at hiding a real issue behind a
smoke curtain. As Nathan described in his email, Open MPI lacks of control
flow on eager messages is the real culprit here, and the loop around any
one-to-many collective (bcast and scatter*) was only helping to exacerbate
the issue. However, doing a loop around a small MPI_Send will also end on a
memory exhaustion issue, one that would not be easily circumvented by
adding synchronizations deep inside the library.

  George.


On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org  wrote:

> I can not provide the user report as it is a proprietary problem. However,
> it consists of a large loop of calls to MPI_Bcast that crashes due to
> unexpected messages. We have been looking at instituting flow control, but
> that has way too widespread an impact. The coll/sync component would be a
> simple solution.
>
> I honestly don’t believe the issue I was resolving was due to a bug - it
> was a simple problem of one proc running slow and creating an overload of
> unexpected messages that eventually consumed too much memory. Rather, I
> think you solved a different problem - by the time you arrived at LANL, the
> app I was working with had already modified their code to no longer create
> the problem (essentially refactoring the algorithm to avoid the massive
> loop over allreduce).
>
> I have no issue supporting it as it takes near-zero effort to maintain,
> and this is a fairly common problem with legacy codes that don’t want to
> refactor their algorithms.
>
>
> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm  wrote:
> >
> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org wrote:
> >>
> >> Hi folks
> >>
> >> I had a question arise regarding a problem being seen by an OMPI user -
> has to do with the old bugaboo I originally dealt with back in my LANL
> days. The problem is with an app that repeatedly hammers on a collective,
> and gets overwhelmed by unexpected messages when one of the procs falls
> behind.
> >
> > I did some investigation on roadrunner several years ago and determined
> that the user code issue coll/sync was attempting to fix was due to a bug
> in ob1/cksum (really can’t remember). coll/sync was simply masking a
> live-lock problem. I committed a workaround for the bug in r26575 (
> https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5
> ef0cea5e35) and tested it with the user code. After this change the user
> code ran fine without coll/sync. Since lanl no longer had any users of
> coll/sync we stopped supporting it.
> >
> >> I solved this back then by introducing the “sync” component in
> ompi/mca/coll, which injected a barrier operation every N collectives. You
> could even “tune” it by doing the injection for only specific collectives.
> >>
> >> However, I can no longer find that component in the code base - I find
> it in the 1.6 series, but someone removed it during the 1.7 series.
> >>
> >> Can someone tell me why this was done??? Is there any reason not to
> bring it back? It solves a very real, not uncommon, problem.
> >> Ralph
> >
> > This was discussed during one (or several) tel-cons years ago. We agreed
> to kill it and bring it back if there is 1) a use case, and 2) someone is
> willing to support it. See https://github.com/open-mpi/ompi/commit/
> 5451ee46bd6fcdec002b333474dec919475d2d62 .
> >
> > Can you link the user email?
> >
> > -Nathan
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Coll/sync component missing???

[r...@open-mpi.org]

I don’t disagree with anything you said - however, this problem has been 
reported in our library for more than a decade (goes way back into the old Trac 
days), and has yet to be resolved. Meantime, we have a user that is “down” and 
needs a solution. Whether it is a “cheap shot” or not is irrelevant to them.

I’ll leave it to you deeper MPI wonks to solve the problem correctly :-) When 
you have done so, I will happily remove the coll/sync component and tell the 
user “all has been resolved”.


> On Aug 20, 2016, at 11:44 AM, George Bosilca  wrote:
> 
> Ralph,
> 
> Bringing back the coll/sync is a cheap shot at hiding a real issue behind a 
> smoke curtain. As Nathan described in his email, Open MPI lacks of control 
> flow on eager messages is the real culprit here, and the loop around any 
> one-to-many collective (bcast and scatter*) was only helping to exacerbate 
> the issue. However, doing a loop around a small MPI_Send will also end on a 
> memory exhaustion issue, one that would not be easily circumvented by adding 
> synchronizations deep inside the library.
> 
>   George.
> 
> 
> On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org 
>  mailto:r...@open-mpi.org>> 
> wrote:
> I can not provide the user report as it is a proprietary problem. However, it 
> consists of a large loop of calls to MPI_Bcast that crashes due to unexpected 
> messages. We have been looking at instituting flow control, but that has way 
> too widespread an impact. The coll/sync component would be a simple solution.
> 
> I honestly don’t believe the issue I was resolving was due to a bug - it was 
> a simple problem of one proc running slow and creating an overload of 
> unexpected messages that eventually consumed too much memory. Rather, I think 
> you solved a different problem - by the time you arrived at LANL, the app I 
> was working with had already modified their code to no longer create the 
> problem (essentially refactoring the algorithm to avoid the massive loop over 
> allreduce).
> 
> I have no issue supporting it as it takes near-zero effort to maintain, and 
> this is a fairly common problem with legacy codes that don’t want to refactor 
> their algorithms.
> 
> 
> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm  > > wrote:
> >
> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org  
> >> wrote:
> >>
> >> Hi folks
> >>
> >> I had a question arise regarding a problem being seen by an OMPI user - 
> >> has to do with the old bugaboo I originally dealt with back in my LANL 
> >> days. The problem is with an app that repeatedly hammers on a collective, 
> >> and gets overwhelmed by unexpected messages when one of the procs falls 
> >> behind.
> >
> > I did some investigation on roadrunner several years ago and determined 
> > that the user code issue coll/sync was attempting to fix was due to a bug 
> > in ob1/cksum (really can’t remember). coll/sync was simply masking a 
> > live-lock problem. I committed a workaround for the bug in r26575 
> > (https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5ef0cea5e35
> >  
> > )
> >  and tested it with the user code. After this change the user code ran fine 
> > without coll/sync. Since lanl no longer had any users of coll/sync we 
> > stopped supporting it.
> >
> >> I solved this back then by introducing the “sync” component in 
> >> ompi/mca/coll, which injected a barrier operation every N collectives. You 
> >> could even “tune” it by doing the injection for only specific collectives.
> >>
> >> However, I can no longer find that component in the code base - I find it 
> >> in the 1.6 series, but someone removed it during the 1.7 series.
> >>
> >> Can someone tell me why this was done??? Is there any reason not to bring 
> >> it back? It solves a very real, not uncommon, problem.
> >> Ralph
> >
> > This was discussed during one (or several) tel-cons years ago. We agreed to 
> > kill it and bring it back if there is 1) a use case, and 2) someone is 
> > willing to support it. See 
> > https://github.com/open-mpi/ompi/commit/5451ee46bd6fcdec002b333474dec919475d2d62
> >  
> > 
> >  .
> >
> > Can you link the user email?
> >
> > -Nathan
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org 
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> > 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
> 
> ___
> devel mailing list
> dev

Re: [OMPI devel] Coll/sync component missing???

[Gilles Gouaillardet]

Ralph,

in the meantime, and if not done already, your user can simply redefine
MPI_Bcast in the app.


int MPI_Bcast(void *buffer, int count, MPI_Datatype type, int root,
MPI_Comm comm) {
PMPI_Barrier(comm);
return PMPI_Bcast(buffer, count, datatype, root, comm);
}

the root causes are
- no control flow in Open MPI for eager messages (as explained by George)
and
- some processes are much slower than others.

so even if Open MPI provides a fix or workaround, the end user will be left
with some important load imbalance, which is far from being optimal from
his/her performance point of view.


Cheers,

Gilles

On Sunday, August 21, 2016, r...@open-mpi.org  wrote:

> I don’t disagree with anything you said - however, this problem has been
> reported in our library for more than a decade (goes way back into the old
> Trac days), and has yet to be resolved. Meantime, we have a user that is
> “down” and needs a solution. Whether it is a “cheap shot” or not is
> irrelevant to them.
>
> I’ll leave it to you deeper MPI wonks to solve the problem correctly :-)
> When you have done so, I will happily remove the coll/sync component and
> tell the user “all has been resolved”.
>
>
> RROn Aug 20, 2016, at 11:44 AM, George Bosilca  > wrote:
>
> Ralph,
>
> Bringing back the coll/sync is a cheap shot at hiding a real issue behind
> a smoke curtain. As Nathan described in his email, Open MPI lacks of
> control flow on eager messages is the real culprit here, and the loop
> around any one-to-many collective (bcast and scatter*) was only helping to
> exacerbate the issue. However, doing a loop around a small MPI_Send will
> also end on a memory exhaustion issue, one that would not be easily
> circumvented by adding synchronizations deep inside the library.
>
>   George.
>
>
> On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org
>   > wrote:
>
>> I can not provide the user report as it is a proprietary problem.
>> However, it consists of a large loop of calls to MPI_Bcast that crashes due
>> to unexpected messages. We have been looking at instituting flow control,
>> but that has way too widespread an impact. The coll/sync component would be
>> a simple solution.
>>
>> I honestly don’t believe the issue I was resolving was due to a bug - it
>> was a simple problem of one proc running slow and creating an overload of
>> unexpected messages that eventually consumed too much memory. Rather, I
>> think you solved a different problem - by the time you arrived at LANL, the
>> app I was working with had already modified their code to no longer create
>> the problem (essentially refactoring the algorithm to avoid the massive
>> loop over allreduce).
>>
>> I have no issue supporting it as it takes near-zero effort to maintain,
>> and this is a fairly common problem with legacy codes that don’t want to
>> refactor their algorithms.
>>
>>
>> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm > > wrote:
>> >
>> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org
>>  wrote:
>> >>
>> >> Hi folks
>> >>
>> >> I had a question arise regarding a problem being seen by an OMPI user
>> - has to do with the old bugaboo I originally dealt with back in my LANL
>> days. The problem is with an app that repeatedly hammers on a collective,
>> and gets overwhelmed by unexpected messages when one of the procs falls
>> behind.
>> >
>> > I did some investigation on roadrunner several years ago and determined
>> that the user code issue coll/sync was attempting to fix was due to a bug
>> in ob1/cksum (really can’t remember). coll/sync was simply masking a
>> live-lock problem. I committed a workaround for the bug in r26575 (
>> https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d
>> 14ec4d2a2e5ef0cea5e35) and tested it with the user code. After this
>> change the user code ran fine without coll/sync. Since lanl no longer had
>> any users of coll/sync we stopped supporting it.
>> >
>> >> I solved this back then by introducing the “sync” component in
>> ompi/mca/coll, which injected a barrier operation every N collectives. You
>> could even “tune” it by doing the injection for only specific collectives.
>> >>
>> >> However, I can no longer find that component in the code base - I find
>> it in the 1.6 series, but someone removed it during the 1.7 series.
>> >>
>> >> Can someone tell me why this was done??? Is there any reason not to
>> bring it back? It solves a very real, not uncommon, problem.
>> >> Ralph
>> >
>> > This was discussed during one (or several) tel-cons years ago. We
>> agreed to kill it and bring it back if there is 1) a use case, and 2)
>> someone is willing to support it. See https://github.com/open-mpi/om
>> pi/commit/5451ee46bd6fcdec002b333474dec919475d2d62 .
>> >
>> > Can you link the user email?
>> >
>> > -Nathan
>> > ___
>> > devel mailing list
>> > devel@lists.open-mpi.org
>> 
>> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>> 

Re: [OMPI devel] Coll/sync component missing???

[r...@open-mpi.org]

As I said earlier, modifying these legacy apps is not a desirable solution. The 
coll/sync component was developed specifically to alleviate these problems in 
an acceptable manner, albeit not optimal. Performance, in this case, is 
secondary to just getting the app to run.


> On Aug 20, 2016, at 7:38 PM, Gilles Gouaillardet 
>  wrote:
> 
> Ralph,
> 
> in the meantime, and if not done already, your user can simply redefine 
> MPI_Bcast in the app.
> 
> 
> int MPI_Bcast(void *buffer, int count, MPI_Datatype type, int root, MPI_Comm 
> comm) {
> PMPI_Barrier(comm);
> return PMPI_Bcast(buffer, count, datatype, root, comm);
> }
> 
> the root causes are
> - no control flow in Open MPI for eager messages (as explained by George)
> and
> - some processes are much slower than others.
> 
> so even if Open MPI provides a fix or workaround, the end user will be left 
> with some important load imbalance, which is far from being optimal from 
> his/her performance point of view.
> 
> 
> Cheers,
> 
> Gilles
> 
> On Sunday, August 21, 2016, r...@open-mpi.org  
> mailto:r...@open-mpi.org>> wrote:
> I don’t disagree with anything you said - however, this problem has been 
> reported in our library for more than a decade (goes way back into the old 
> Trac days), and has yet to be resolved. Meantime, we have a user that is 
> “down” and needs a solution. Whether it is a “cheap shot” or not is 
> irrelevant to them.
> 
> I’ll leave it to you deeper MPI wonks to solve the problem correctly :-) When 
> you have done so, I will happily remove the coll/sync component and tell the 
> user “all has been resolved”.
> 
> 
>> RROn Aug 20, 2016, at 11:44 AM, George Bosilca > > wrote:
>> 
>> Ralph,
>> 
>> Bringing back the coll/sync is a cheap shot at hiding a real issue behind a 
>> smoke curtain. As Nathan described in his email, Open MPI lacks of control 
>> flow on eager messages is the real culprit here, and the loop around any 
>> one-to-many collective (bcast and scatter*) was only helping to exacerbate 
>> the issue. However, doing a loop around a small MPI_Send will also end on a 
>> memory exhaustion issue, one that would not be easily circumvented by adding 
>> synchronizations deep inside the library.
>> 
>>   George.
>> 
>> 
>> On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org 
>>  > > wrote:
>> I can not provide the user report as it is a proprietary problem. However, 
>> it consists of a large loop of calls to MPI_Bcast that crashes due to 
>> unexpected messages. We have been looking at instituting flow control, but 
>> that has way too widespread an impact. The coll/sync component would be a 
>> simple solution.
>> 
>> I honestly don’t believe the issue I was resolving was due to a bug - it was 
>> a simple problem of one proc running slow and creating an overload of 
>> unexpected messages that eventually consumed too much memory. Rather, I 
>> think you solved a different problem - by the time you arrived at LANL, the 
>> app I was working with had already modified their code to no longer create 
>> the problem (essentially refactoring the algorithm to avoid the massive loop 
>> over allreduce).
>> 
>> I have no issue supporting it as it takes near-zero effort to maintain, and 
>> this is a fairly common problem with legacy codes that don’t want to 
>> refactor their algorithms.
>> 
>> 
>> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm > > > wrote:
>> >
>> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org 
>> >>  wrote:
>> >>
>> >> Hi folks
>> >>
>> >> I had a question arise regarding a problem being seen by an OMPI user - 
>> >> has to do with the old bugaboo I originally dealt with back in my LANL 
>> >> days. The problem is with an app that repeatedly hammers on a collective, 
>> >> and gets overwhelmed by unexpected messages when one of the procs falls 
>> >> behind.
>> >
>> > I did some investigation on roadrunner several years ago and determined 
>> > that the user code issue coll/sync was attempting to fix was due to a bug 
>> > in ob1/cksum (really can’t remember). coll/sync was simply masking a 
>> > live-lock problem. I committed a workaround for the bug in r26575 
>> > (https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d14ec4d2a2e5ef0cea5e35
>> >  
>> > )
>> >  and tested it with the user code. After this change the user code ran 
>> > fine without coll/sync. Since lanl no longer had any users of coll/sync we 
>> > stopped supporting it.
>> >
>> >> I solved this back then by introducing the “sync” component in 
>> >> ompi/mca/coll, which injected a barrier operation every N collectives. 
>> >> You could even “tune” it by doing the injection for only specific 
>> >> collectives.
>> >>
>> >> However, I can no longer find that component in the code base - I find it 
>> >> in the 1.6 series, but someone removed it during the 1.7 series.
>> >>
>> >> Can someone tell me why this was done??

Re: [OMPI devel] Coll/sync component missing???

[Gilles Gouaillardet]

I got your point, I was just pointing to an other option "in the meantime".

fwiw, the bits of code I posted earlier can be put in a shared library that
is LD_PRELOAD'ed,
so the application is kept unmodified.

Cheers,

Gilles

On Sunday, August 21, 2016, r...@open-mpi.org  wrote:

> As I said earlier, modifying these legacy apps is not a desirable
> solution. The coll/sync component was developed specifically to alleviate
> these problems in an acceptable manner, albeit not optimal. Performance, in
> this case, is secondary to just getting the app to run.
>
>
> On Aug 20, 2016, at 7:38 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com
> > wrote:
>
> Ralph,
>
> in the meantime, and if not done already, your user can simply redefine
> MPI_Bcast in the app.
>
>
> int MPI_Bcast(void *buffer, int count, MPI_Datatype type, int root,
> MPI_Comm comm) {
> PMPI_Barrier(comm);
> return PMPI_Bcast(buffer, count, datatype, root, comm);
> }
>
> the root causes are
> - no control flow in Open MPI for eager messages (as explained by George)
> and
> - some processes are much slower than others.
>
> so even if Open MPI provides a fix or workaround, the end user will be
> left with some important load imbalance, which is far from being optimal
> from his/her performance point of view.
>
>
> Cheers,
>
> Gilles
>
> On Sunday, August 21, 2016, r...@open-mpi.org
>   > wrote:
>
>> I don’t disagree with anything you said - however, this problem has been
>> reported in our library for more than a decade (goes way back into the old
>> Trac days), and has yet to be resolved. Meantime, we have a user that is
>> “down” and needs a solution. Whether it is a “cheap shot” or not is
>> irrelevant to them.
>>
>> I’ll leave it to you deeper MPI wonks to solve the problem correctly :-)
>> When you have done so, I will happily remove the coll/sync component and
>> tell the user “all has been resolved”.
>>
>>
>> RROn Aug 20, 2016, at 11:44 AM, George Bosilca 
>> wrote:
>>
>> Ralph,
>>
>> Bringing back the coll/sync is a cheap shot at hiding a real issue behind
>> a smoke curtain. As Nathan described in his email, Open MPI lacks of
>> control flow on eager messages is the real culprit here, and the loop
>> around any one-to-many collective (bcast and scatter*) was only helping to
>> exacerbate the issue. However, doing a loop around a small MPI_Send will
>> also end on a memory exhaustion issue, one that would not be easily
>> circumvented by adding synchronizations deep inside the library.
>>
>>   George.
>>
>>
>> On Sat, Aug 20, 2016 at 12:30 AM, r...@open-mpi.org 
>> wrote:
>>
>>> I can not provide the user report as it is a proprietary problem.
>>> However, it consists of a large loop of calls to MPI_Bcast that crashes due
>>> to unexpected messages. We have been looking at instituting flow control,
>>> but that has way too widespread an impact. The coll/sync component would be
>>> a simple solution.
>>>
>>> I honestly don’t believe the issue I was resolving was due to a bug - it
>>> was a simple problem of one proc running slow and creating an overload of
>>> unexpected messages that eventually consumed too much memory. Rather, I
>>> think you solved a different problem - by the time you arrived at LANL, the
>>> app I was working with had already modified their code to no longer create
>>> the problem (essentially refactoring the algorithm to avoid the massive
>>> loop over allreduce).
>>>
>>> I have no issue supporting it as it takes near-zero effort to maintain,
>>> and this is a fairly common problem with legacy codes that don’t want to
>>> refactor their algorithms.
>>>
>>>
>>> > On Aug 19, 2016, at 8:48 PM, Nathan Hjelm  wrote:
>>> >
>>> >> On Aug 19, 2016, at 4:24 PM, r...@open-mpi.org wrote:
>>> >>
>>> >> Hi folks
>>> >>
>>> >> I had a question arise regarding a problem being seen by an OMPI user
>>> - has to do with the old bugaboo I originally dealt with back in my LANL
>>> days. The problem is with an app that repeatedly hammers on a collective,
>>> and gets overwhelmed by unexpected messages when one of the procs falls
>>> behind.
>>> >
>>> > I did some investigation on roadrunner several years ago and
>>> determined that the user code issue coll/sync was attempting to fix was due
>>> to a bug in ob1/cksum (really can’t remember). coll/sync was simply masking
>>> a live-lock problem. I committed a workaround for the bug in r26575 (
>>> https://github.com/open-mpi/ompi/commit/59e529cf1dfe986e40d
>>> 14ec4d2a2e5ef0cea5e35) and tested it with the user code. After this
>>> change the user code ran fine without coll/sync. Since lanl no longer had
>>> any users of coll/sync we stopped supporting it.
>>> >
>>> >> I solved this back then by introducing the “sync” component in
>>> ompi/mca/coll, which injected a barrier operation every N collectives. You
>>> could even “tune” it by doing the injection for only specific collectives.
>>> >>
>>> >> However, I can no longer find that component in the code base - I
>>> find i