[OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Jeff Squyres]

This is an open question to OMPI developers...

It looks like RHEL (and maybe others?) adds the "virbr0" IP interface when Xen 
is activated.  This IP interface is only used to communicate with the local Xen 
instance(s); it is not used to communicate over the real network.  

In a case that I saw, the interface is created, set to "up", and is given an IP 
address in the 192.168.1.x range.  This was done by default -- all the user had 
done was either say "yes, I want Xen enabled", or he didn't say he wanted it 
*disabled* (I'm not sure which).

This causes a problem if you have Xen enabled on multiple machines in an OMPI 
job.  OMPI will see the 192.168.1.x address and see that it's "up", so it'll 
add it to the eligible subnets that can be used.  When OMPI sees that its peer 
processes also have 192.168.1.x, it'll try to use that network for OOB/BTL 
traffic -- which will fail, because these are local-only interfaces.

Should we add "virbr0" to the default value for [btl|oob]_tcp_if_exclude?  

Or is there another way to detect that an interface is local-only and should 
not be used for OOB/BTL communication?

See this post on the user's list:

http://www.open-mpi.org/community/lists/users/2012/02/18432.php

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Ralph Castain]

On Feb 10, 2012, at 9:50 AM, Jeff Squyres wrote:

> This is an open question to OMPI developers...
> 
> It looks like RHEL (and maybe others?) adds the "virbr0" IP interface when 
> Xen is activated.  This IP interface is only used to communicate with the 
> local Xen instance(s); it is not used to communicate over the real network.  
> 
> In a case that I saw, the interface is created, set to "up", and is given an 
> IP address in the 192.168.1.x range.  This was done by default -- all the 
> user had done was either say "yes, I want Xen enabled", or he didn't say he 
> wanted it *disabled* (I'm not sure which).
> 
> This causes a problem if you have Xen enabled on multiple machines in an OMPI 
> job.  OMPI will see the 192.168.1.x address and see that it's "up", so it'll 
> add it to the eligible subnets that can be used.  When OMPI sees that its 
> peer processes also have 192.168.1.x, it'll try to use that network for 
> OOB/BTL traffic -- which will fail, because these are local-only interfaces.
> 
> Should we add "virbr0" to the default value for [btl|oob]_tcp_if_exclude?  

How generic is that name? I've looked and can't find a way to detect a 
local-only interface, though you might be able to do it via ARP. Looking for a 
name, though, is pretty hit/miss.

> 
> Or is there another way to detect that an interface is local-only and should 
> not be used for OOB/BTL communication?
> 
> See this post on the user's list:
> 
>http://www.open-mpi.org/community/lists/users/2012/02/18432.php
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[TERRY DONTJE]

On 2/10/2012 11:50 AM, Jeff Squyres wrote:

This is an open question to OMPI developers...

It looks like RHEL (and maybe others?) adds the "virbr0" IP interface when Xen 
is activated.  This IP interface is only used to communicate with the local Xen 
instance(s); it is not used to communicate over the real network.

In a case that I saw, the interface is created, set to "up", and is given an IP address 
in the 192.168.1.x range.  This was done by default -- all the user had done was either say 
"yes, I want Xen enabled", or he didn't say he wanted it *disabled* (I'm not sure which).
I've done the latter and hit the same problem.  There were instructions 
somewhere on the web that I found that told one how to disable vibr0.


This causes a problem if you have Xen enabled on multiple machines in an OMPI job.  OMPI 
will see the 192.168.1.x address and see that it's "up", so it'll add it to the 
eligible subnets that can be used.  When OMPI sees that its peer processes also have 
192.168.1.x, it'll try to use that network for OOB/BTL traffic -- which will fail, 
because these are local-only interfaces.

Should we add "virbr0" to the default value for [btl|oob]_tcp_if_exclude?
What happens to that value if you then set btl_tcp_if_exclude to some 
value on the mpirun command line?  So this brings me to something that 
has annoyed me for a bit.  It seems to me that maybe it would be nice to 
have a conf file that you can dump interface names to exclude but would 
not be interpreted as a btl_tcp_if_exclude options.  For example there 
were some interfaces on certain Sun machine (a long time ago) that went 
to the diagnostic processor and caused a similar issue as the virbr0 
issue.  So we started delivering a conf file that set btl_tcp_if_exclude 
but then this precluded anyone from being able to set 
btl_tcp_if_include.  If we had a file one could specify the set of 
interfaces to use or exclude but allow the user to operate on the result 
of that set it seems that would be nice.


--td


Or is there another way to detect that an interface is local-only and should 
not be used for OOB/BTL communication?

See this post on the user's list:

 http://www.open-mpi.org/community/lists/users/2012/02/18432.php



--
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle *- Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Jeff Squyres]

On Feb 10, 2012, at 1:00 PM, TERRY DONTJE wrote:

>> Should we add "virbr0" to the default value for [btl|oob]_tcp_if_exclude?  
>> 
> What happens to that value if you then set btl_tcp_if_exclude to some value 
> on the mpirun command line?  

It works just fine.  I.e., if you

mpirun --mca btl_tcp_if_exclude lo,virbr0 ...

That works like a champ.

But per Ralph's question, I don't know how generic that name is.  It *seems* to 
be specific to a virtualization interface (I assume "virbr" = "virtual 
bridge"), but I can't say that for sure.

> So this brings me to something that has annoyed me for a bit.  It seems to me 
> that maybe it would be nice to have a conf file that you can dump interface 
> names to exclude but would not be interpreted as a btl_tcp_if_exclude 
> options.  For example there were some interfaces on certain Sun machine (a 
> long time ago) that went to the diagnostic processor and caused a similar 
> issue as the virbr0 issue.  So we started delivering a conf file that set 
> btl_tcp_if_exclude but then this precluded anyone from being able to set 
> btl_tcp_if_include.  If we had a file one could specify the set of interfaces 
> to use or exclude but allow the user to operate on the result of that set it 
> seems that would be nice.

I'm not sure what you're saying.  CLI always overrides config files...?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Paul H. Hargrove]

On 2/10/2012 10:38 AM, Jeff Squyres wrote:

On Feb 10, 2012, at 1:00 PM, TERRY DONTJE wrote:


>>  Should we add "virbr0" to the default value for [btl|oob]_tcp_if_exclude?
>>  

>  What happens to that value if you then set btl_tcp_if_exclude to some value 
on the mpirun command line?

It works just fine.  I.e., if you

 mpirun --mca btl_tcp_if_exclude lo,virbr0 ...

That works like a champ.


Since the situation described is one where the user didn't know they 
could/should disable xen, it is reasonable to think they ALSO don't know 
they need to exclude virbr0.  So, I read the question as meaning the 
following:

 What happens when a user who doesn't know anything about virbr0 does
  mpirun --mca btl_tcp_if_exclude lo,eth8
And my guess is "nothing good happens".

-Paul

--
Paul H. Hargrove  phhargr...@lbl.gov
Future Technologies Group
HPC Research Department   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Jeff Squyres]

On Feb 10, 2012, at 1:44 PM, Paul H. Hargrove wrote:

> Since the situation described is one where the user didn't know they 
> could/should disable xen, it is reasonable to think they ALSO don't know they 
> need to exclude virbr0.  

That's what I'm thinking.

> So, I read the question as meaning the following:
> What happens when a user who doesn't know anything about virbr0 does
>  mpirun --mca btl_tcp_if_exclude lo,eth8

I'm not sure I understand your question -- the above will exclude loopback and 
eth8.  

(where did eth8 come from?)

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Paul H. Hargrove]

On 2/10/2012 12:03 PM, Jeff Squyres wrote:

On Feb 10, 2012, at 1:44 PM, Paul H. Hargrove wrote:


Since the situation described is one where the user didn't know they 
could/should disable xen, it is reasonable to think they ALSO don't know they 
need to exclude virbr0.

That's what I'm thinking.


So, I read the question as meaning the following:
 What happens when a user who doesn't know anything about virbr0 does
  mpirun --mca btl_tcp_if_exclude lo,eth8

I'm not sure I understand your question -- the above will exclude loopback and 
eth8.

(where did eth8 come from?)



Sorry, if I wasn't clear.
I'll try again:

+ User knows nothing about xen, and thus nothing about virbr0
+ User has a local-only interface (eth8 in my made up example)
+ User reads FAQ entry "220. How do I tell Open MPI which TCP networks 
to use?"
+ User follows instructions given in said FAQ, yielding my example 
command line.


-Paul

--
Paul H. Hargrove  phhargr...@lbl.gov
Future Technologies Group
HPC Research Department   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Jeff Squyres]

On Feb 10, 2012, at 3:14 PM, Paul H. Hargrove wrote:

> + User knows nothing about xen, and thus nothing about virbr0
> + User has a local-only interface (eth8 in my made up example)
> + User reads FAQ entry "220. How do I tell Open MPI which TCP networks to 
> use?"
> + User follows instructions given in said FAQ, yielding my example command 
> line.

Do you mean that eth8 is the only non-loopback interface on their laptop, and 
it's disconnected?  (e.g., sitting on a train with no wifi and no wired 
ethernet)

Then OMPI would have disqualified that interface, anyway (because it wasn't up).

I think I'm missing the zen of your question... :-\

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Paul H. Hargrove]

On 2/10/2012 12:21 PM, Jeff Squyres wrote:

On Feb 10, 2012, at 3:14 PM, Paul H. Hargrove wrote:


+ User knows nothing about xen, and thus nothing about virbr0
+ User has a local-only interface (eth8 in my made up example)
+ User reads FAQ entry "220. How do I tell Open MPI which TCP networks to use?"
+ User follows instructions given in said FAQ, yielding my example command line.

Do you mean that eth8 is the only non-loopback interface on their laptop, and 
it's disconnected?  (e.g., sitting on a train with no wifi and no wired 
ethernet)

Then OMPI would have disqualified that interface, anyway (because it wasn't up).

I think I'm missing the zen of your question... :-\



The point of the question isn't related to WHY eth8 is useless - just 
assume it is.
Assume it is UP, but useless for whatever reasons motivated writing FAQ 
#220.

It could be Terry's example of a port connected to the service processor.

The concern is what happens in this situation when the user, following 
the advice in the FAQ, passes an explicit setting for 
btl_tcp_if_exclude, which DOES NOT include virbr0?
They don't know it was there before, or that it needs to be there (the 
FAQ states that lo MUST be included).

So, by following the FAQ they don't resolve their problem.
OMPI ceases any attempts use of eth8 (or whatever), but loss of the 
implicit virbr0 from the exclude list results in their system attempting 
to use virbr0 (and thus continue to fail).  Right?


Maybe the FAQ just needs an update to address my concern.

-Paul

--
Paul H. Hargrove  phhargr...@lbl.gov
Future Technologies Group
HPC Research Department   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

Re: [OMPI devel] RFC: Java MPI bindings

[Jeff Squyres]

This topic has gotten a *lot* of off-list discussion.  :-)

Just to tie up the thread for the web archives: this topic will be discussed on 
the Tuesday teleconf next week.  Terry: please add this to the Tuesday agenda.



On Feb 8, 2012, at 9:53 PM, Ralph Castain wrote:

> I agree - we do have better things to do than argue about things that have no 
> impact on anyone not choosing to use them.
> 
> If you had read the RFC, you would have seen that the Hadoop businesses are 
> unwilling to trust their future to a 3rd party bolt-on that already shows 
> bit-rot. Hence, the desire to hire/fund people to put it in the mainstream.
> 
> As I said, this doesn't impact anyone. So let's just agree to disagree and 
> move on.
> 
> 
> On Feb 8, 2012, at 7:34 PM, George Bosilca wrote:
> 
>> 3 anonymous accesses and 12 developer accesses from 2010-11-25 till today, 
>> that's what the mpiJava project hosted on sourceforge got. It tends to say 
>> something about the need for such a binding now, the size of the community 
>> requiring such a binding and about the support Open MPI should throw to it 
>> ahead.
>> 
>> Make it grow, flourish, build its own community and then let's reevaluate 
>> your request. Until then, we all have more interesting things to do.
>> 
>>   george.
>> 
>> http://sourceforge.net/projects/mpijava/stats/scm?repo=HgRepository&dates=2010-07-10+to+2012-02-09
>> 
>> 
>> On Feb 7, 2012, at 16:16 , Ralph Castain wrote:
>> 
>>> No problems, Paul. I appreciate your input.
>>> 
>>> If everything in the trunk was required to in the standard, then much of 
>>> the trunk would have to be removed (e.g., all the fault tolerance code). As 
>>> Jeff indicated, the trunk is an area in which we bring new functionality 
>>> for broader exposure. I very much doubt there will be much instability for 
>>> anyone not using the Java bindings (though I do expect some degree of 
>>> debugging in that arena), but my ears are open if/when someone finds 
>>> something.
>>> 
>>> You are welcome to try the branch, if it would help resolve concerns:
>>> 
>>> https://bitbucket.org/rhc/ompi-jv2
>>> 
>>> 
>>> On Feb 7, 2012, at 2:06 PM, Paul H. Hargrove wrote:
>>> 
 Ralph,
 
 I think you and I may be confusing each other with the meaning of 
 "standard":
 
 You asked me
> So I'm not sure what you are asking that hasn't already been done…
 
 My reply to that question is that when I wrote
> a) a standard to which the bindings can claim to conform
 
 I meant "a) JAVA bindings standardized by the MPI Forum."
 In other words, I feel that new language binding should kept out of the 
 trunk until there is a standard from the MPI Forum.
 I don't think that is a "chicken-and-egg" problem, because the branch 
 would be available to the Hadoop community to show the Forum that 
 existence of the necessary community. IN FACT, download stats for that 
 branch could be shown as evidence of that interest.
 
 So, now I hope it is at least clear where we disagree.
 I am, as I said at the start, NOT an OMPI developer and so I won't argue 
 this point any further.
 
 -Paul
 
 On 2/7/2012 12:48 PM, Ralph Castain wrote:
> On Feb 7, 2012, at 1:43 PM, Paul H. Hargrove wrote:
> 
>> Forgive me if I misunderstand, but I am under the impression that the 
>> MPI Forum has not begun any standardization of MPI bindings for JAVA. 
>> Have I missed something?
> No, they haven't - but that doesn't mean that the bindings cannot conform 
> to the standard. Remember, the standard doesn't dictate that you can't 
> have Java bindings - it just doesn't currently require that you do. Big 
> difference in those two statements.
> 
>> -Paul
>> 
>> On 2/7/2012 12:39 PM, Ralph Castain wrote:
>>> We already have a stable, standard interface for non-C language 
>>> bindings, Paul - the C++ bindings, for example, are built on top of 
>>> them.
>>> 
>>> The binding codes are all orthogonal to the base code. All they do is 
>>> massage data access and then loop back to the C bindings. This is the 
>>> normal way we handle all non-C bindings, so nothing different there.
>>> 
>>> The work has been done on a branch, and an RFC issued. The bindings 
>>> conform to the MPI standard, and the implementation uses an existing 
>>> external, third-party binding that has been tested.
>>> 
>>> So I'm not sure what you are asking that hasn't already been done…
>>> 
>>> On Feb 7, 2012, at 1:33 PM, Paul H. Hargrove wrote:
>>> 
 As an HPC software developer and user of OMPI, I'd like to add my 
 $0.02 here even though I am not an OMPI developer.
 
 Nothing in George's response seems to me to preclude the interested 
 institutions (listed as FROM in the RFC) from forking a branch to 
 pursue this work until there can be standardiz

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Jeff Squyres]

On Feb 10, 2012, at 3:32 PM, Paul H. Hargrove wrote:

> The point of the question isn't related to WHY eth8 is useless - just assume 
> it is.
> Assume it is UP, but useless for whatever reasons motivated writing FAQ #220.
> It could be Terry's example of a port connected to the service processor.
> 
> The concern is what happens in this situation when the user, following the 
> advice in the FAQ, passes an explicit setting for btl_tcp_if_exclude, which 
> DOES NOT include virbr0?
> They don't know it was there before, or that it needs to be there (the FAQ 
> states that lo MUST be included).
> So, by following the FAQ they don't resolve their problem.
> OMPI ceases any attempts use of eth8 (or whatever), but loss of the implicit 
> virbr0 from the exclude list results in their system attempting to use virbr0 
> (and thus continue to fail).  Right?
> 
> Maybe the FAQ just needs an update to address my concern.

Got it.  Sure, I can update the faq to be a bit more loose in the definition of 
what must be excluded.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Jeff Squyres (jsquyres)]

Check out #220 now; I updated it. 

Sent from my phone. No type good. 

On Feb 10, 2012, at 4:46 PM, "Jeff Squyres"  wrote:

> On Feb 10, 2012, at 3:32 PM, Paul H. Hargrove wrote:
> 
>> The point of the question isn't related to WHY eth8 is useless - just assume 
>> it is.
>> Assume it is UP, but useless for whatever reasons motivated writing FAQ #220.
>> It could be Terry's example of a port connected to the service processor.
>> 
>> The concern is what happens in this situation when the user, following the 
>> advice in the FAQ, passes an explicit setting for btl_tcp_if_exclude, which 
>> DOES NOT include virbr0?
>> They don't know it was there before, or that it needs to be there (the FAQ 
>> states that lo MUST be included).
>> So, by following the FAQ they don't resolve their problem.
>> OMPI ceases any attempts use of eth8 (or whatever), but loss of the implicit 
>> virbr0 from the exclude list results in their system attempting to use 
>> virbr0 (and thus continue to fail).  Right?
>> 
>> Maybe the FAQ just needs an update to address my concern.
> 
> Got it.  Sure, I can update the faq to be a bit more loose in the definition 
> of what must be excluded.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Paul H. Hargrove]

Much better - at least to the extent that users actually read FAQs :-)
-Paul

On 2/10/2012 5:01 PM, Jeff Squyres (jsquyres) wrote:

Check out #220 now; I updated it.


--
Paul H. Hargrove  phhargr...@lbl.gov
Future Technologies Group
HPC Research Department   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Jeff Squyres]

FWIW: google analytics indicates that the FAQ and the mailing list archives are 
among the most heavily used sections of the web site.  :-)

On Feb 10, 2012, at 8:09 PM, Paul H. Hargrove wrote:

> Much better - at least to the extent that users actually read FAQs :-)
> -Paul
> 
> On 2/10/2012 5:01 PM, Jeff Squyres (jsquyres) wrote:
>> Check out #220 now; I updated it.
> 
> -- 
> Paul H. Hargrove  phhargr...@lbl.gov
> Future Technologies Group
> HPC Research Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] RFC: Add "virbr0" to [btl|oob]_tcp_if_exclude?

[Paul H. Hargrove]

That's probably a reflection of the status of the "Open MPI User 
Documentation" sub-project :-)


On 2/10/2012 5:12 PM, Jeff Squyres wrote:

FWIW: google analytics indicates that the FAQ and the mailing list archives are 
among the most heavily used sections of the web site.  :-)


--
Paul H. Hargrove  phhargr...@lbl.gov
Future Technologies Group
HPC Research Department   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

Re: [OMPI devel] 1.4.5rc5 has been released

[Jeff Squyres]

On Feb 7, 2012, at 4:25 PM, Paul H. Hargrove wrote:

> + I can't build w/ the PGI compilers on MacOS Lion.
> This was previously reported in 
> http://www.open-mpi.org/community/lists/devel/2012/01/10258.php

I'm officially pushing this to v1.6.  See 
https://svn.open-mpi.org/trac/ompi/ticket/3011.

> + Building w/ Solaris Studio 12.2 or 12.3 on Linux x86-64, with "-m32" 
> required setting LD_LIBRARY_PATH.
> This is could either be Oracle's bug in the compiler, or a libtool problem.
> My report was: http://www.open-mpi.org/community/lists/devel/2012/01/10272.php

This has been resolved.  1.5.5rc2 coming soon.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] 1.4.5rc5 has been released

[Jeff Squyres]

On Feb 10, 2012, at 8:57 PM, Jeff Squyres wrote:

> 1.5.5rc2 coming soon.

I should qualify that statement: many things have been resolved, but there's a 
few more things to go.  A new rc will come when they have been resolved:

  https://svn.open-mpi.org/trac/ompi/report/15

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

[OMPI devel] v1.4.5rc6 released

[Jeff Squyres]

Usual location:

http://www.open-mpi.org/software/ompi/v1.4/

Changes since rc5:

- document LD_LIBRARY_PATH for -m32 with Ubuntu/Sun compilers
- refuse to configure with gccffs
- LANL TLCC2 platform files

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/