Re: [OMPI devel] [OMPI bugs] [Open MPI] #2322: Something is wrong with rdmacm cpc

[Jeff Squyres (jsquyres)]

I did test the patch 2 version and that does seem to be working for me. 
However, that obviously doesn't mean that it's safe. 

Should we put some atomics in there, to be absolutely sure?  Or put a lock 
around the dlsym code to ensure that only 1 thread calls dlsym?

-jms
Sent from my PDA.  No type good.

- Original Message -
From: bugs-boun...@open-mpi.org 
Cc: b...@osl.iu.edu 
Sent: Wed Mar 03 01:45:38 2010
Subject: Re: [OMPI bugs] [Open MPI] #2322: Something is wrong with rdmacm cpc

#2322: Something is wrong with rdmacm cpc
-+--
Reporter:  jsquyres  |   Owner:
Type:  defect|  Status:  new   
Priority:  blocker   |   Milestone:  Open MPI 1.4.2
 Version:  trunk |Keywords:
-+--

Comment(by cyeoh):

 I might be misunderstanding whats going on here, but I don't think the
 volatile will help with respect to the second two of the three causes of
 the segv as AFAIK volatile doesn't make any guarantees when it comes
 atomicity. It will force a reload each time but if the read is not atomic
 anyway then there still could be problems

 However I do suspect the latter two are not causing the problems. The
 static pointer should be aligned and I think in that case the assignments
 should be atomic (will be for powerpc, I think so for other
 architectures).

 I think its much more likely that dlsym was getting called when its not
 safe to (the first theory).

 The consolidation of the code looks good - sorry I should have done that
 in the first place.

 If its working now is it possible that its because its now consolidated
 into opal_mem_free_ptmalloc2_munmap that the munmap_real is getting called
 consistently early at a time when its safe to call dlsym (ie not through
 the munmap_real path which appears to be sometimes unsafe) ?

-- 
Ticket URL: 
Open MPI 

___
bugs mailing list
b...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/bugs

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[Jeff Squyres]

I'm not sure I agree with change #1.  I understand in principle why the change 
was made, but I'm uncomfortable with:

1. The individual entries now behave like pseudo-regexp's rather that strict 
matching.  We used strict matching before this for a reason.  If we want to 
allow regexp-like behavior, then I think we should enable that with special 
characters -- that's the customary/usual way to do it.

2. All other _in|exclude behavior in ompi is strict matching, not prefix 
matching.  I'm uncomfortable with the disparity.

Additionally, if loopback is now handled properly via change #2, shouldn't the 
default value for the btl_tcp_if_exclude parameter now be empty?

Actually -- thinking about this a little more, does opal_net_islocalhost() 
guarantee to work on peer interfaces?  



On Mar 3, 2010, at 10:51 AM,  wrote:

> Author: igb
> Date: 2010-03-03 10:51:15 EST (Wed, 03 Mar 2010)
> New Revision: 22762
> URL: https://svn.open-mpi.org/trac/ompi/changeset/22762
> 
> Log:
> Fixed two problems:
> 
> 1. The code that looks at btl_tcp_if_exclude before doing a
>modex_send uses strcmp rather than strncmp. That means that
>"lo0" gets sent even though "lo" is excluded.
> 
> 2. The code that determines whether a particular local TCP
>interface can connect to a particular remote interface doesn't
>check for loopback interfaces. With this fix, users can now
>enable "lo" and be assured that it will only be used for intra-
>node communication.
> 
> 
> Text files modified:
>trunk/ompi/mca/btl/tcp/btl_tcp_component.c | 2 +-  
>
>trunk/ompi/mca/btl/tcp/btl_tcp_proc.c  |25 
> +++--  
>2 files changed, 24 insertions(+), 3 deletions(-)
> 
> Modified: trunk/ompi/mca/btl/tcp/btl_tcp_component.c
> ==
> --- trunk/ompi/mca/btl/tcp/btl_tcp_component.c  (original)
> +++ trunk/ompi/mca/btl/tcp/btl_tcp_component.c  2010-03-03 10:51:15 EST (Wed, 
> 03 Mar 2010)
> @@ -610,7 +610,7 @@
>  /* check to see if this interface exists in the exclude list */
>  argv = exclude;
>  while(argv && *argv) {
> -if(strcmp(*argv,if_name) == 0)
> +if(strncmp(*argv,if_name,strlen(*argv)) == 0)
>  break;
>  argv++;
>  }
> 
> Modified: trunk/ompi/mca/btl/tcp/btl_tcp_proc.c
> ==
> --- trunk/ompi/mca/btl/tcp/btl_tcp_proc.c   (original)
> +++ trunk/ompi/mca/btl/tcp/btl_tcp_proc.c   2010-03-03 10:51:15 EST (Wed, 
> 03 Mar 2010)
> @@ -504,8 +504,18 @@
>  if(NULL != local_interfaces[i]->ipv4_address &&
> NULL != peer_interfaces[j]->ipv4_address) {
> 
> +/*  check for loopback */
> +if ((opal_net_islocalhost((struct sockaddr 
> *)local_interfaces[i]->ipv4_address)
> + && !opal_net_islocalhost((struct sockaddr 
> *)peer_interfaces[j]->ipv4_address))
> +|| (opal_net_islocalhost((struct sockaddr 
> *)peer_interfaces[j]->ipv4_address)
> +&& !opal_net_islocalhost((struct sockaddr 
> *)local_interfaces[i]->ipv4_address))
> +|| (opal_net_islocalhost((struct sockaddr 
> *)local_interfaces[i]->ipv4_address)
> +&& 
> !opal_ifislocal(btl_proc->proc_ompi->proc_hostname))) {
> +
> +/* No connection is possible on these interfaces */
> +   
>  /*  check for RFC1918 */
> -if(opal_net_addr_isipv4public((struct sockaddr*) 
> local_interfaces[i]->ipv4_address)
> +} else if(opal_net_addr_isipv4public((struct sockaddr*) 
> local_interfaces[i]->ipv4_address)
> && opal_net_addr_isipv4public((struct sockaddr*)
>   
> peer_interfaces[j]->ipv4_address)) {
>  if(opal_net_samenetwork((struct sockaddr*) 
> local_interfaces[i]->ipv4_address,
> @@ -534,7 +544,18 @@
>   */
>  if(NULL != local_interfaces[i]->ipv6_address &&
> NULL != peer_interfaces[j]->ipv6_address) {
> -if(opal_net_samenetwork((struct sockaddr*) 
> local_interfaces[i]->ipv6_address,
> +
> +/*  check for loopback */
> +if ((opal_net_islocalhost((struct sockaddr 
> *)local_interfaces[i]->ipv6_address)
> + && !opal_net_islocalhost((struct sockaddr 
> *)peer_interfaces[j]->ipv6_address))
> +|| (opal_net_islocalhost((struct sockaddr 
> *)peer_interfaces[j]->ipv6_address)
> +&& !opal_net_islocalhost((struct sockaddr 
> *)local_interfaces[i]->ipv6_address))
> +|| (opal_net_islocalhost((struct sockaddr 
> *)local_interfaces[i]->ipv6_address)
> +  

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[Iain Bason]

On Mar 3, 2010, at 1:24 PM, Jeff Squyres wrote:

> I'm not sure I agree with change #1.  I understand in principle why the 
> change was made, but I'm uncomfortable with:
> 
> 1. The individual entries now behave like pseudo-regexp's rather that strict 
> matching.  We used strict matching before this for a reason.  If we want to 
> allow regexp-like behavior, then I think we should enable that with special 
> characters -- that's the customary/usual way to do it.

The history of this particular piece of code is that it used to use strncmp.  
George Bosilca changed it last summer, incidental to a larger change (r21652).  
The commit comment was not particularly illuminating on this issue, in my 
opinion:

http://www.open-mpi.org/hg/hgwebdir.cgi/ompi-svn-mirror/rev/bde31d3db7ba

> 2. All other _in|exclude behavior in ompi is strict matching, not prefix 
> matching.  I'm uncomfortable with the disparity.

That turns out not to be the case.  Look in 
btl_tcp_proc.c/mca_btl_tcp_retrieve_local_interfaces.

> Additionally, if loopback is now handled properly via change #2, shouldn't 
> the default value for the btl_tcp_if_exclude parameter now be empty?

That's a good question.  Enabling the "lo" interface results in intra-node 
messages being striped across that interface in addition to the others on a 
system.  I don't know what impact that would have, if any.

> Actually -- thinking about this a little more, does opal_net_islocalhost() 
> guarantee to work on peer interfaces?  

It looks to see whether the IP address is (v4) 127.0.0.1, or (v6) ::1.  I 
believe that these values are dictated by the relevant RFCs (but I haven't 
looked to make sure).

Iain

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[Jeff Squyres]

On Mar 3, 2010, at 2:06 PM, Iain Bason wrote:

> > 1. The individual entries now behave like pseudo-regexp's rather that 
> > strict matching.  We used strict matching before this for a reason.  If we 
> > want to allow regexp-like behavior, then I think we should enable that with 
> > special characters -- that's the customary/usual way to do it.
> 
> The history of this particular piece of code is that it used to use strncmp.  
> George Bosilca changed it last summer, incidental to a larger change 
> (r21652).  The commit comment was not particularly illuminating on this 
> issue, in my opinion:
> 
> http://www.open-mpi.org/hg/hgwebdir.cgi/ompi-svn-mirror/rev/bde31d3db7ba

You're right -- it's not illuminating... :-\

> > 2. All other _in|exclude behavior in ompi is strict matching, not 
> > prefix matching.  I'm uncomfortable with the disparity.
> 
> That turns out not to be the case.  Look in 
> btl_tcp_proc.c/mca_btl_tcp_retrieve_local_interfaces.

Mmmm... good point.  I was thinking specifically of the if_in|exclude behavior 
in the openib BTL.  That uses strcmp, not strncmp.  Here's a complete list:

ompi_info --param all all --parsable | grep include | grep :value:
mca:opal:base:param:opal_event_include:value:pollmca:btl:ofud:param:btl_ofud_if_include:value:
mca:btl:openib:param:btl_openib_if_include:value:
mca:btl:openib:param:btl_openib_ipaddr_include:value:mca:btl:openib:param:btl_openib_cpc_include:value:
mca:btl:sctp:param:btl_sctp_if_include:value:
mca:btl:tcp:param:btl_tcp_if_include:value:
mca:btl:base:param:btl_base_include:value:
mca:oob:tcp:param:oob_tcp_if_include:value:

Do we know what these others do?  I only checked openib_if_*clude -- it's 
strcmp.

> > Additionally, if loopback is now handled properly via change #2, shouldn't 
> > the default value for the btl_tcp_if_exclude parameter now be empty?
> 
> That's a good question.  Enabling the "lo" interface results in intra-node 
> messages being striped across that interface in addition to the others on a 
> system.  I don't know what impact that would have, if any.

sm and self should still be prioritized above it, right?  If so, we should be 
ok.

However, I think you're right that the addition of striping across lo* in 
addition to the other interfaces might have an unknown effect.

Here's a random question -- if a user does not use the sm btl, would sending 
messages through lo for on-node communication be potentially better than 
sending it through a real device, given that that real device may be far away 
(in the NUMA sense of "far")?  I.e., are OS's typically smart enough to know 
that loopback traffic may be able to stay local to the NUMA node, vs. sending 
it out to a device and back?  Or are OS's smart enough to know that if the both 
ends of a TCP socket are on the same node -- regardless of what IP interface 
they use -- and if both processes are on the same NUMA locality, that the data 
can stay local and not have to make a round trip to the device?

(I admit that this is a fairly corner case -- doing on-node communication but 
*not* using the sm btl...)

> > Actually -- thinking about this a little more, does opal_net_islocalhost() 
> > guarantee to work on peer interfaces? 
> 
> It looks to see whether the IP address is (v4) 127.0.0.1, or (v6) ::1.  I 
> believe that these values are dictated by the relevant RFCs (but I haven't 
> looked to make sure).

Good enough -- thanks!  (I was thinking that it might be checking interfaces, 
not IP addrs -- so 127.x checking should be fine here)

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[Iain Bason]

On Mar 3, 2010, at 3:04 PM, Jeff Squyres wrote:

> 
> Mmmm... good point.  I was thinking specifically of the if_in|exclude 
> behavior in the openib BTL.  That uses strcmp, not strncmp.  Here's a 
> complete list:
> 
> ompi_info --param all all --parsable | grep include | grep :value:
> mca:opal:base:param:opal_event_include:value:pollmca:btl:ofud:param:btl_ofud_if_include:value:
> mca:btl:openib:param:btl_openib_if_include:value:
> mca:btl:openib:param:btl_openib_ipaddr_include:value:mca:btl:openib:param:btl_openib_cpc_include:value:
> mca:btl:sctp:param:btl_sctp_if_include:value:
> mca:btl:tcp:param:btl_tcp_if_include:value:
> mca:btl:base:param:btl_base_include:value:
> mca:oob:tcp:param:oob_tcp_if_include:value:
> 
> Do we know what these others do?  I only checked openib_if_*clude -- it's 
> strcmp.

I haven't looked at those, but it's easy to grep for strncmp...

It looks as though sctp is the only other BTL that uses strncmp.

Of course, if we decide to change the default so that it no longer includes 
"lo" then maybe using strncmp doesn't matter.  The problem has been that the 
name of the interface is different on different platforms.

(I should note that the default also excludes "sppp".  I don't know anything 
about that interface.)

>>> Additionally, if loopback is now handled properly via change #2, shouldn't 
>>> the default value for the btl_tcp_if_exclude parameter now be empty?
>> 
>> That's a good question.  Enabling the "lo" interface results in intra-node 
>> messages being striped across that interface in addition to the others on a 
>> system.  I don't know what impact that would have, if any.
> 
> sm and self should still be prioritized above it, right?  If so, we should be 
> ok.

Yes, that's true.  It would only affect those who restrict intra-node 
communication to TCP.

> However, I think you're right that the addition of striping across lo* in 
> addition to the other interfaces might have an unknown effect.
> 
> Here's a random question -- if a user does not use the sm btl, would sending 
> messages through lo for on-node communication be potentially better than 
> sending it through a real device, given that that real device may be far away 
> (in the NUMA sense of "far")?  I.e., are OS's typically smart enough to know 
> that loopback traffic may be able to stay local to the NUMA node, vs. sending 
> it out to a device and back?  Or are OS's smart enough to know that if the 
> both ends of a TCP socket are on the same node -- regardless of what IP 
> interface they use -- and if both processes are on the same NUMA locality, 
> that the data can stay local and not have to make a round trip to the device?
> 
> (I admit that this is a fairly corner case -- doing on-node communication but 
> *not* using the sm btl...)

Good question.  For the loopback interface there is no physical device, so 
there should be no NUMA effect.  For an interface with a physical device there 
may be some reason that a packet would actually have to go out to the device.  
If there is no such reason, I would expect Unix to be smart enough not to do 
it, given how much intra-node TCP traffic one commonly sees on Unix.  I 
couldn't hazard a guess about Windows.

Iain

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[George Bosilca]

On Mar 3, 2010, at 15:04 , Jeff Squyres wrote:

> On Mar 3, 2010, at 2:06 PM, Iain Bason wrote:
> 
>>> 1. The individual entries now behave like pseudo-regexp's rather that 
>>> strict matching.  We used strict matching before this for a reason.  If we 
>>> want to allow regexp-like behavior, then I think we should enable that with 
>>> special characters -- that's the customary/usual way to do it.
>> 
>> The history of this particular piece of code is that it used to use strncmp. 
>>  George Bosilca changed it last summer, incidental to a larger change 
>> (r21652).  The commit comment was not particularly illuminating on this 
>> issue, in my opinion:
>> 
>> http://www.open-mpi.org/hg/hgwebdir.cgi/ompi-svn-mirror/rev/bde31d3db7ba
> 
> You're right -- it's not illuminating... :-\
> 
>>> 2. All other _in|exclude behavior in ompi is strict matching, not 
>>> prefix matching.  I'm uncomfortable with the disparity.
>> 
>> That turns out not to be the case.  Look in 
>> btl_tcp_proc.c/mca_btl_tcp_retrieve_local_interfaces.

I guess this is the result different developers with different ideas working on 
a non consistent way. This is without talking about the fact that we do the 
same checking in several places, and we duplicate the code in a way that 
doesn't enforce any consistency. Anyway, now that this problem is highlighted, 
we should clearly fix it.

> Mmmm... good point.  I was thinking specifically of the if_in|exclude 
> behavior in the openib BTL.  That uses strcmp, not strncmp.  Here's a 
> complete list:
> 
> ompi_info --param all all --parsable | grep include | grep :value:
> mca:opal:base:param:opal_event_include:value:pollmca:btl:ofud:param:btl_ofud_if_include:value:
> mca:btl:openib:param:btl_openib_if_include:value:
> mca:btl:openib:param:btl_openib_ipaddr_include:value:mca:btl:openib:param:btl_openib_cpc_include:value:
> mca:btl:sctp:param:btl_sctp_if_include:value:
> mca:btl:tcp:param:btl_tcp_if_include:value:
> mca:btl:base:param:btl_base_include:value:
> mca:oob:tcp:param:oob_tcp_if_include:value:
> 
> Do we know what these others do?  I only checked openib_if_*clude -- it's 
> strcmp.
> 
>>> Additionally, if loopback is now handled properly via change #2, shouldn't 
>>> the default value for the btl_tcp_if_exclude parameter now be empty?
>> 
>> That's a good question.  Enabling the "lo" interface results in intra-node 
>> messages being striped across that interface in addition to the others on a 
>> system.  I don't know what impact that would have, if any.
> 
> sm and self should still be prioritized above it, right?  If so, we should be 
> ok.
> 
> However, I think you're right that the addition of striping across lo* in 
> addition to the other interfaces might have an unknown effect.

This is not supposed to happen. The sm BTL has a high exclusivity, which will 
prevent the TCP BTL to be used for the same peer. But again, this was the case 
a while ago, there is nothing to guarantee that the code is still doing what it 
was supposed to.

  george.

> Here's a random question -- if a user does not use the sm btl, would sending 
> messages through lo for on-node communication be potentially better than 
> sending it through a real device, given that that real device may be far away 
> (in the NUMA sense of "far")?  I.e., are OS's typically smart enough to know 
> that loopback traffic may be able to stay local to the NUMA node, vs. sending 
> it out to a device and back?  Or are OS's smart enough to know that if the 
> both ends of a TCP socket are on the same node -- regardless of what IP 
> interface they use -- and if both processes are on the same NUMA locality, 
> that the data can stay local and not have to make a round trip to the device?
> 
> (I admit that this is a fairly corner case -- doing on-node communication but 
> *not* using the sm btl...)
> 
>>> Actually -- thinking about this a little more, does opal_net_islocalhost() 
>>> guarantee to work on peer interfaces? 
>> 
>> It looks to see whether the IP address is (v4) 127.0.0.1, or (v6) ::1.  I 
>> believe that these values are dictated by the relevant RFCs (but I haven't 
>> looked to make sure).
> 
> Good enough -- thanks!  (I was thinking that it might be checking interfaces, 
> not IP addrs -- so 127.x checking should be fine here)
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[Jeff Squyres]

On Mar 3, 2010, at 3:26 PM, George Bosilca wrote:

> I guess this is the result different developers with different ideas working 
> on a non consistent way. This is without talking about the fact that we do 
> the same checking in several places, and we duplicate the code in a way that 
> doesn't enforce any consistency. Anyway, now that this problem is 
> highlighted, we should clearly fix it.

When I made that ompi-info list in the last mail, I was thinking the same thing 
- should we make an opal routine to examine in/exclude kind of behavior?  If 
so, then all those places could just call that instead of replicating the code.

Consolidating the code might also enable doing some kind of simple 
globbing-like or regexp-like behavior (e.g., "lo*" with the globbing meaning, 
or "lo.*" with the regexp meaning -- globbing is probably a little easier to 
implement).  

Such a thing might kill 2 birds with one stone...

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[Terry Dontje]

Iain Bason wrote:

On Mar 3, 2010, at 3:04 PM, Jeff Squyres wrote:

  

Mmmm... good point.  I was thinking specifically of the if_in|exclude behavior 
in the openib BTL.  That uses strcmp, not strncmp.  Here's a complete list:

ompi_info --param all all --parsable | grep include | grep :value:
mca:opal:base:param:opal_event_include:value:pollmca:btl:ofud:param:btl_ofud_if_include:value:
mca:btl:openib:param:btl_openib_if_include:value:
mca:btl:openib:param:btl_openib_ipaddr_include:value:mca:btl:openib:param:btl_openib_cpc_include:value:
mca:btl:sctp:param:btl_sctp_if_include:value:
mca:btl:tcp:param:btl_tcp_if_include:value:
mca:btl:base:param:btl_base_include:value:
mca:oob:tcp:param:oob_tcp_if_include:value:

Do we know what these others do?  I only checked openib_if_*clude -- it's 
strcmp.



I haven't looked at those, but it's easy to grep for strncmp...

It looks as though sctp is the only other BTL that uses strncmp.

Of course, if we decide to change the default so that it no longer includes 
"lo" then maybe using strncmp doesn't matter.  The problem has been that the 
name of the interface is different on different platforms.

(I should note that the default also excludes "sppp".  I don't know anything 
about that interface.)
  
I may be wrong for the usage here but the old Sun Starcats had a tcp 
interface named sppp to its diagnostic processor that we needed to skip.
Not sure if this is the same reason done here, I couldn't find where 
sppp was referenced so I could find the history of the line in opengrok.


--td
  

Additionally, if loopback is now handled properly via change #2, shouldn't the 
default value for the btl_tcp_if_exclude parameter now be empty?


That's a good question.  Enabling the "lo" interface results in intra-node 
messages being striped across that interface in addition to the others on a system.  I 
don't know what impact that would have, if any.
  

sm and self should still be prioritized above it, right?  If so, we should be 
ok.



Yes, that's true.  It would only affect those who restrict intra-node 
communication to TCP.

  

However, I think you're right that the addition of striping across lo* in 
addition to the other interfaces might have an unknown effect.

Here's a random question -- if a user does not use the sm btl, would sending messages 
through lo for on-node communication be potentially better than sending it through a real 
device, given that that real device may be far away (in the NUMA sense of 
"far")?  I.e., are OS's typically smart enough to know that loopback traffic 
may be able to stay local to the NUMA node, vs. sending it out to a device and back?  Or 
are OS's smart enough to know that if the both ends of a TCP socket are on the same node 
-- regardless of what IP interface they use -- and if both processes are on the same NUMA 
locality, that the data can stay local and not have to make a round trip to the device?

(I admit that this is a fairly corner case -- doing on-node communication but 
*not* using the sm btl...)



Good question.  For the loopback interface there is no physical device, so 
there should be no NUMA effect.  For an interface with a physical device there 
may be some reason that a packet would actually have to go out to the device.  
If there is no such reason, I would expect Unix to be smart enough not to do 
it, given how much intra-node TCP traffic one commonly sees on Unix.  I 
couldn't hazard a guess about Windows.

Iain

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
  

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r22762

[Terry Dontje]

Terry Dontje wrote:

Iain Bason wrote:

On Mar 3, 2010, at 3:04 PM, Jeff Squyres wrote:

 
Mmmm... good point.  I was thinking specifically of the 
if_in|exclude behavior in the openib BTL.  That uses strcmp, not 
strncmp.  Here's a complete list:


ompi_info --param all all --parsable | grep include | grep :value:
mca:opal:base:param:opal_event_include:value:pollmca:btl:ofud:param:btl_ofud_if_include:value: 


mca:btl:openib:param:btl_openib_if_include:value:
mca:btl:openib:param:btl_openib_ipaddr_include:value:mca:btl:openib:param:btl_openib_cpc_include:value: 


mca:btl:sctp:param:btl_sctp_if_include:value:
mca:btl:tcp:param:btl_tcp_if_include:value:
mca:btl:base:param:btl_base_include:value:
mca:oob:tcp:param:oob_tcp_if_include:value:

Do we know what these others do?  I only checked openib_if_*clude -- 
it's strcmp.



I haven't looked at those, but it's easy to grep for strncmp...

It looks as though sctp is the only other BTL that uses strncmp.

Of course, if we decide to change the default so that it no longer 
includes "lo" then maybe using strncmp doesn't matter.  The problem 
has been that the name of the interface is different on different 
platforms.


(I should note that the default also excludes "sppp".  I don't know 
anything about that interface.)
  
I may be wrong for the usage here but the old Sun Starcats had a tcp 
interface named sppp to its diagnostic processor that we needed to skip.
Not sure if this is the same reason done here, I couldn't find where 
sppp was referenced so I could find the history of the line in opengrok.


--td
Close, r19988 added sppp to the excluded interfaces for the Sun M9000 
server.  I believe for the same reason I gave above.


--td
 
Additionally, if loopback is now handled properly via change #2, 
shouldn't the default value for the btl_tcp_if_exclude parameter 
now be empty?

That's a good question.  Enabling the "lo" interface results in 
intra-node messages being striped across that interface in addition 
to the others on a system.  I don't know what impact that would 
have, if any.
  
sm and self should still be prioritized above it, right?  If so, we 
should be ok.



Yes, that's true.  It would only affect those who restrict intra-node 
communication to TCP.


 
However, I think you're right that the addition of striping across 
lo* in addition to the other interfaces might have an unknown effect.


Here's a random question -- if a user does not use the sm btl, would 
sending messages through lo for on-node communication be potentially 
better than sending it through a real device, given that that real 
device may be far away (in the NUMA sense of "far")?  I.e., are OS's 
typically smart enough to know that loopback traffic may be able to 
stay local to the NUMA node, vs. sending it out to a device and 
back?  Or are OS's smart enough to know that if the both ends of a 
TCP socket are on the same node -- regardless of what IP interface 
they use -- and if both processes are on the same NUMA locality, 
that the data can stay local and not have to make a round trip to 
the device?


(I admit that this is a fairly corner case -- doing on-node 
communication but *not* using the sm btl...)



Good question.  For the loopback interface there is no physical 
device, so there should be no NUMA effect.  For an interface with a 
physical device there may be some reason that a packet would actually 
have to go out to the device.  If there is no such reason, I would 
expect Unix to be smart enough not to do it, given how much 
intra-node TCP traffic one commonly sees on Unix.  I couldn't hazard 
a guess about Windows.


Iain

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
  


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel