Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Ralph Castain]

On Jul 3, 2013, at 1:00 PM, Riccardo Murri  wrote:

> Hi Jeff, Ralph,
> 
> first of all: thanks for your work on this!
> 
> On 3 July 2013 21:09, Jeff Squyres (jsquyres)  wrote:
>> 1. The root cause of the issue is that you are assigning a
>> non-existent IP address to a name.  I.e.,  maps to 127.0.1.1,
>> but that IP address does not exist anywhere.  Hence, OMPI will never
>> conclude that that  is "local".  If you had assigned  to
>> the 127.0.0.1 address, things should have worked fine.
> 
> Ok, I see.  Would that have worked also if I had added the 127.0.1.1
> address to the "lo" interface (in addition to 127.0.0.1)?

Probably, but I can't say for sure.

> 
> 
>> Just curious: why are you doing this?
> 
> It's commonplace in Ubuntu/Debian installations; see, e.g.,
> http://serverfault.com/questions/363095/what-does-127-0-1-1-represent-in-etc-hosts
> 
> In our case, it was rolled out as a fix for some cron job running on
> Apache servers (apparently Debian's Apache looks up 127.0.1.1 and uses
> that as the ServerName, unless a server name is not explicitly
> configured), which was later extended to all hosts because "what harm
> can it do?".
> 
> (Needless to say, we have rolled back the change.)

Weird - never heard of that before!

> 
> 
>> 2. That being said, OMPI is not currently looking at all the
>> responses from gethostbyname() -- we're only looking at the first
>> one.  In the spirit of how clients are supposed to behave when
>> multiple IP addresses are returned from a single name lookup, OMPI
>> should examine all of those addresses and see if it finds one that
>> it "likes", and then use that.  So we should extend OMPI to examine
>> all the IP addresses from gethostbyname().
> 
> Just for curiosity: would it have worked, had I compiled OMPI with
> IPv6 support?  (As far as I understand IPv6, an application is
> required to examine all the addresses returned for a host name, and
> not just pick the first one.)

Actually, yes - for some reason, the code path when IPv6 support is enabled had 
already been extended to look at all addresses. Not sure why, but that change 
was never carried over to the IPv6-disabled code path. I've done so now, so 
this won't be a problem in the future.

> 
> 
>> Ralph is going to work on this, but it'll likely take him a little
>> time to get it done.  We'll get it into the trunk and probably ask
>> you to verify that it works for you.  And if so, we'll back-port to
>> the v1.6 and v1.7 series.
> 
> I'm glad to help and verify, but I guess we do not need the backport
> or an urgent fix.  The easy workaround for us was to remove the
> 127.0.1.1 line from the compute nodes (we keep it only on Apache
> servers where it originated).

Glad you found an easy solution!
Ralph

> 
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

Hi Jeff, Ralph,

first of all: thanks for your work on this!

On 3 July 2013 21:09, Jeff Squyres (jsquyres)  wrote:
> 1. The root cause of the issue is that you are assigning a
> non-existent IP address to a name.  I.e.,  maps to 127.0.1.1,
> but that IP address does not exist anywhere.  Hence, OMPI will never
> conclude that that  is "local".  If you had assigned  to
> the 127.0.0.1 address, things should have worked fine.

Ok, I see.  Would that have worked also if I had added the 127.0.1.1
address to the "lo" interface (in addition to 127.0.0.1)?


> Just curious: why are you doing this?

It's commonplace in Ubuntu/Debian installations; see, e.g.,
http://serverfault.com/questions/363095/what-does-127-0-1-1-represent-in-etc-hosts

In our case, it was rolled out as a fix for some cron job running on
Apache servers (apparently Debian's Apache looks up 127.0.1.1 and uses
that as the ServerName, unless a server name is not explicitly
configured), which was later extended to all hosts because "what harm
can it do?".

(Needless to say, we have rolled back the change.)


> 2. That being said, OMPI is not currently looking at all the
> responses from gethostbyname() -- we're only looking at the first
> one.  In the spirit of how clients are supposed to behave when
> multiple IP addresses are returned from a single name lookup, OMPI
> should examine all of those addresses and see if it finds one that
> it "likes", and then use that.  So we should extend OMPI to examine
> all the IP addresses from gethostbyname().

Just for curiosity: would it have worked, had I compiled OMPI with
IPv6 support?  (As far as I understand IPv6, an application is
required to examine all the addresses returned for a host name, and
not just pick the first one.)


> Ralph is going to work on this, but it'll likely take him a little
> time to get it done.  We'll get it into the trunk and probably ask
> you to verify that it works for you.  And if so, we'll back-port to
> the v1.6 and v1.7 series.

I'm glad to help and verify, but I guess we do not need the backport
or an urgent fix.  The easy workaround for us was to remove the
127.0.1.1 line from the compute nodes (we keep it only on Apache
servers where it originated).

Thanks,
Riccardo

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Jeff Squyres (jsquyres)]

Ralph and I talked some more about this.

Here's what we think:

1. The root cause of the issue is that you are assigning a non-existent IP 
address to a name.  I.e.,  maps to 127.0.1.1, but that IP address does not 
exist anywhere.  Hence, OMPI will never conclude that that  is "local".  
If you had assigned  to the 127.0.0.1 address, things should have worked 
fine.

Just curious: why are you doing this?

2. That being said, OMPI is not currently looking at all the responses from 
gethostbyname() -- we're only looking at the first one.  In the spirit of how 
clients are supposed to behave when multiple IP addresses are returned from a 
single name lookup, OMPI should examine all of those addresses and see if it 
finds one that it "likes", and then use that.  So we should extend OMPI to 
examine all the IP addresses from gethostbyname().  This should also fix your 
issue.

Ralph is going to work on this, but it'll likely take him a little time to get 
it done.  We'll get it into the trunk and probably ask you to verify that it 
works for you.  And if so, we'll back-port to the v1.6 and v1.7 series.  

One final caveat, however: at this point, it does not look likely that 1.6.6 
will ever happen.  If this all works out, the fix will be committed to the v1.6 
tree, and you can grab a nightly tarball snapshot (which are identical to our 
release tarballs except for their version numbers), or you can patch your 1.6.5 
installation.  But if 1.6.6 is ever released, the fix will be included.


On Jul 2, 2013, at 9:53 AM, Riccardo Murri  wrote:

> Hi,
> 
> sorry for the delay in replying -- pretty busy week :-(
> 
> 
> On 28 June 2013 21:54, Jeff Squyres (jsquyres)  wrote:
>> Here's what we think we know (I'm using the name "foo" instead of
>> your actual hostname because it's easier to type):
>> 
>> 1. When you run "hostname", you get foo.local back
> 
> Yes.
> 
> 
>> 2. In your /etc/hosts file, foo.local is listed on two lines:
>>   127.0.1.1
>>   10.1.255.201
>> 
> 
> Yes:
> 
>[rmurri@nh64-5-9 ~]$ fgrep nh64-5-9 /etc/hosts
>127.0.1.1   nh64-5-9.local nh64-5-9
>10.1.255.194nh64-5-9.local nh64-5-9
> 
> 
>> 3. When you login to the "foo" server and execute mpirun with a hostfile
>> that contains "foo", Open MPI incorrectly thinks that the local machine is
>> not foo, and therefore tries to ssh to it (and things go downhill from
>> there).
>> 
> 
> Yes.
> 
> 
>> 4. When you login to the "foo" server and execute mpirun with a hostfile
>> that contains "foo.local" (you said "FQDN", but never said exactly what you
>> meant by that -- I'm assuming "foo.local", not "foo.yourdomain.com"), then
>> Open MPI behaves properly.
>> 
> 
> Yes.
> 
> FQDN = foo.local.  (This is a compute node in a cluster that does not
> have any public IP address not DNS entry -- it only has an interface
> to the cluster-private network.  I presume this is not relevant to
> OpenMPI as long as all names are correctly resolved via `/etc/hosts`.)
> 
> 
>> Is that all correct?
> 
> Yes, all correct.
> 
> 
>> We have some followup questions for you:
>> 
>> 1. What happens when you try to resolve "foo"? (e.g., via the "dig" program
>> -- "dig foo")
> 
> Here's what happens with `dig`:
> 
>[rmurri@nh64-5-9 ~]$ dig nh64-5-9
> 
>; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> nh64-5-9
>;; global options:  printcmd
>;; Got answer:
>;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4373
>;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
> 
>;; QUESTION SECTION:
>;nh64-5-9. IN  A
> 
>;; AUTHORITY SECTION:
>.  3600IN  SOA a.root-servers.net. 
> nstld.verisign-grs.com.
> 2013070200 1800 900 604800 86400
> 
>;; Query time: 17 msec
>;; SERVER: 10.1.1.1#53(10.1.1.1)
>;; WHEN: Tue Jul  2 15:47:57 2013
>;; MSG SIZE  rcvd: 101
> 
> However, `getent hosts` has a different reply:
> 
>[rmurri@nh64-5-9 ~]$ getent hosts nh64-5-9
>127.0.1.1   nh64-5-9.local nh64-5-9
> 
> 
>> 2. What happens when you try to resolve "foo.local"? (e.g., "dig foo.local")
> 
> Here's what happens with `dig`:
> 
>[rmurri@nh64-5-9 ~]$ dig nh64-5-9.local
> 
>; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> nh64-5-9.local
>;; global options:  printcmd
>;; Got answer:
>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62092
>;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
> 
>;; QUESTION SECTION:
>;nh64-5-9.local.   IN  A
> 
>;; ANSWER SECTION:
>nh64-5-9.local.259200  IN  A   10.1.255.194
> 
>;; AUTHORITY SECTION:
>local. 259200  IN  NS  ns.local.
> 
>;; ADDITIONAL SECTION:
>ns.local.  259200  IN  A   127.0.0.1
> 
>;; Query time: 0 msec
>;; SERVER: 10.1.1.1#53(10.1.1.1)
>;; WHEN: Tue Jul  2 15:48:50 2013
>;; MSG SIZE  rcvd: 81

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

Hi,

sorry for the delay in replying -- pretty busy week :-(


On 28 June 2013 21:54, Jeff Squyres (jsquyres)  wrote:
> Here's what we think we know (I'm using the name "foo" instead of
> your actual hostname because it's easier to type):
>
> 1. When you run "hostname", you get foo.local back

Yes.


> 2. In your /etc/hosts file, foo.local is listed on two lines:
>127.0.1.1
>10.1.255.201
>

Yes:

[rmurri@nh64-5-9 ~]$ fgrep nh64-5-9 /etc/hosts
127.0.1.1   nh64-5-9.local nh64-5-9
10.1.255.194nh64-5-9.local nh64-5-9


> 3. When you login to the "foo" server and execute mpirun with a hostfile
> that contains "foo", Open MPI incorrectly thinks that the local machine is
> not foo, and therefore tries to ssh to it (and things go downhill from
> there).
>

Yes.


> 4. When you login to the "foo" server and execute mpirun with a hostfile
> that contains "foo.local" (you said "FQDN", but never said exactly what you
> meant by that -- I'm assuming "foo.local", not "foo.yourdomain.com"), then
> Open MPI behaves properly.
>

Yes.

FQDN = foo.local.  (This is a compute node in a cluster that does not
have any public IP address not DNS entry -- it only has an interface
to the cluster-private network.  I presume this is not relevant to
OpenMPI as long as all names are correctly resolved via `/etc/hosts`.)


> Is that all correct?

Yes, all correct.


> We have some followup questions for you:
>
> 1. What happens when you try to resolve "foo"? (e.g., via the "dig" program
> -- "dig foo")

Here's what happens with `dig`:

[rmurri@nh64-5-9 ~]$ dig nh64-5-9

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> nh64-5-9
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4373
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;nh64-5-9.  IN  A

;; AUTHORITY SECTION:
.   3600IN  SOA a.root-servers.net. 
nstld.verisign-grs.com.
2013070200 1800 900 604800 86400

;; Query time: 17 msec
;; SERVER: 10.1.1.1#53(10.1.1.1)
;; WHEN: Tue Jul  2 15:47:57 2013
;; MSG SIZE  rcvd: 101

However, `getent hosts` has a different reply:

[rmurri@nh64-5-9 ~]$ getent hosts nh64-5-9
127.0.1.1   nh64-5-9.local nh64-5-9


> 2. What happens when you try to resolve "foo.local"? (e.g., "dig foo.local")

Here's what happens with `dig`:

[rmurri@nh64-5-9 ~]$ dig nh64-5-9.local

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> nh64-5-9.local
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62092
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;nh64-5-9.local.IN  A

;; ANSWER SECTION:
nh64-5-9.local. 259200  IN  A   10.1.255.194

;; AUTHORITY SECTION:
local.  259200  IN  NS  ns.local.

;; ADDITIONAL SECTION:
ns.local.   259200  IN  A   127.0.0.1

;; Query time: 0 msec
;; SERVER: 10.1.1.1#53(10.1.1.1)
;; WHEN: Tue Jul  2 15:48:50 2013
;; MSG SIZE  rcvd: 81

Same query resolved via `getent hosts`:

[rmurri@nh64-5-9 ~]$ getent hosts nh64-5-9
127.0.1.1   nh64-5-9.local nh64-5-9


> 3. What happens when you try to resolve "foo.yourdomain.com"? (e.g., "dig
> foo.yourdomain.com")

This yields an empty response from both `dig` and `getent hosts` as the node
is only attached to a private network and not registered in DNS:

[rmurri@nh64-5-9 ~]$ getent hosts nh64-5-9.uzh.ch
[rmurri@nh64-5-9 ~]$ dig nh64-5-9.uzh.ch

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> nh64-5-9.uzh.ch
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 61801
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;nh64-5-9.uzh.ch.   IN  A

;; AUTHORITY SECTION:
uzh.ch. 8921IN  SOA ns1.uzh.ch. 
hostmaster.uzh.ch. 384627811
3600 1800 360 10800

;; Query time: 0 msec
;; SERVER: 10.1.1.1#53(10.1.1.1)
;; WHEN: Tue Jul  2 15:50:54 2013
;; MSG SIZE  rcvd: 84


> 4. Please apply the attached patch to your Open MPI 1.6.5 build (please note
> that it adds diagnostic output; do *not* put this patch into production)
> and:
>4a. Run with one of your "bad" cases and send us the output
>4b. Run with one of your "good" cases and send us the output

Please find the outputs attached.  The exact `mpiexec` invocation and
the machines file are at the beginning of each file.

Note that I allocated 8 slots (on 4 nodes), but only use 2 slots (on 1 node).

Thanks,
Riccardo


exam01.out.BAD
Description: Binary data


exam01.out.GOOD
Description: Binary data

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Jeff Squyres (jsquyres)]

Ralph and I talked about this issue this afternoon.  We're still struggling to 
understand the details of your configuration, in part because this thread was 
hijacked twice with issues unrelated to this 127.0.1.1 issue.  Here's what we 
think we know (I'm using the name "foo" instead of your actual hostname because 
it's easier to type):

1. When you run "hostname", you get foo.local back

2. In your /etc/hosts file, foo.local is listed on two lines:
   127.0.1.1
   10.1.255.201

3. When you login to the "foo" server and execute mpirun with a hostfile that 
contains "foo", Open MPI incorrectly thinks that the local machine is not foo, 
and therefore tries to ssh to it (and things go downhill from there).

4. When you login to the "foo" server and execute mpirun with a hostfile that 
contains "foo.local" (you said "FQDN", but never said exactly what you meant by 
that -- I'm assuming "foo.local", not "foo.yourdomain.com"), then Open MPI 
behaves properly.

Is that all correct?

We have some followup questions for you:

1. What happens when you try to resolve "foo"? (e.g., via the "dig" program -- 
"dig foo")
2. What happens when you try to resolve "foo.local"? (e.g., "dig foo.local")
3. What happens when you try to resolve "foo.yourdomain.com"? (e.g., "dig 
foo.yourdomain.com")
4. Please apply the attached patch to your Open MPI 1.6.5 build (please note 
that it adds diagnostic output; do *not* put this patch into production) and:
   4a. Run with one of your "bad" cases and send us the output
   4b. Run with one of your "good" cases and send us the output




On Jun 26, 2013, at 7:38 AM, Ralph Castain  wrote:

> The root cause of the problem is that you are assigning your host name to the 
> loopback device. This is rather unusual, but not forbidden. Normally, people 
> would name that interface something like "localhost" since it cannot be used 
> to communicate off-node.
>
> Doing it the way you have could cause problems for you as programs that do a 
> lookup to communicate will get the loopback address when they might have 
> expected something else. Still, we should handle this case.
>
> I'll see what we can do
>
>
>
> On Wed, Jun 26, 2013 at 2:26 AM, Riccardo Murri  wrote:
> Hello,
>
> On 26 June 2013 03:11, Ralph Castain  wrote:
> > I've been reviewing the code, and I think I'm getting a handle on
> > the issue.
> >
> > Just to be clear - your hostname resolves to the 127 address? And you are on
> > a Linux (not one of the BSD flavors out there)?
>
> Yes (but resolves to 127.0.1.1 -- not the usual 127.0.0.1), and yes
> (Rocks 5.3 ~= CentOS 5.3).
>
>
> > If the answer to both is "yes", then the problem is that we ignore loopback
> > devices if anything else is present. When we check to see if the hostname we
> > were given is the local node, we resolve the name to the address and then
> > check our list of interfaces. The loopback device is ignored and therefore
> > not on the list. So if you resolve to the 127 address, we will decide this
> > is a different node than the one we are on.
> >
> > I can modify that logic, but want to ensure this accurately captures the
> > problem. I'll also have to discuss the change with the other developers to
> > ensure we don't shoot ourselves in the foot if we make it.
>
> Ok, thanks -- I'll keep an eye on your replies.
>
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/



islocal-debug.patch
Description: islocal-debug.patch

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Ralph Castain]

The root cause of the problem is that you are assigning your host name to
the loopback device. This is rather unusual, but not forbidden. Normally,
people would name that interface something like "localhost" since it cannot
be used to communicate off-node.

Doing it the way you have could cause problems for you as programs that do
a lookup to communicate will get the loopback address when they might have
expected something else. Still, we should handle this case.

I'll see what we can do



On Wed, Jun 26, 2013 at 2:26 AM, Riccardo Murri wrote:

> Hello,
>
> On 26 June 2013 03:11, Ralph Castain  wrote:
> > I've been reviewing the code, and I think I'm getting a handle on
> > the issue.
> >
> > Just to be clear - your hostname resolves to the 127 address? And you
> are on
> > a Linux (not one of the BSD flavors out there)?
>
> Yes (but resolves to 127.0.1.1 -- not the usual 127.0.0.1), and yes
> (Rocks 5.3 ~= CentOS 5.3).
>
>
> > If the answer to both is "yes", then the problem is that we ignore
> loopback
> > devices if anything else is present. When we check to see if the
> hostname we
> > were given is the local node, we resolve the name to the address and then
> > check our list of interfaces. The loopback device is ignored and
> therefore
> > not on the list. So if you resolve to the 127 address, we will decide
> this
> > is a different node than the one we are on.
> >
> > I can modify that logic, but want to ensure this accurately captures the
> > problem. I'll also have to discuss the change with the other developers
> to
> > ensure we don't shoot ourselves in the foot if we make it.
>
> Ok, thanks -- I'll keep an eye on your replies.
>
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

Hello,

On 26 June 2013 03:11, Ralph Castain  wrote:
> I've been reviewing the code, and I think I'm getting a handle on
> the issue.
>
> Just to be clear - your hostname resolves to the 127 address? And you are on
> a Linux (not one of the BSD flavors out there)?

Yes (but resolves to 127.0.1.1 -- not the usual 127.0.0.1), and yes
(Rocks 5.3 ~= CentOS 5.3).


> If the answer to both is "yes", then the problem is that we ignore loopback
> devices if anything else is present. When we check to see if the hostname we
> were given is the local node, we resolve the name to the address and then
> check our list of interfaces. The loopback device is ignored and therefore
> not on the list. So if you resolve to the 127 address, we will decide this
> is a different node than the one we are on.
>
> I can modify that logic, but want to ensure this accurately captures the
> problem. I'll also have to discuss the change with the other developers to
> ensure we don't shoot ourselves in the foot if we make it.

Ok, thanks -- I'll keep an eye on your replies.

Thanks,
Riccardo

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Ralph Castain]

I'll ignore the rest of this thread as it kinda diverged from your original
question. I've been reviewing the code, and I think I'm getting a handle on
the issue.

Just to be clear - your hostname resolves to the 127 address? And you are
on a Linux (not one of the BSD flavors out there)?

If the answer to both is "yes", then the problem is that we ignore loopback
devices if anything else is present. When we check to see if the hostname
we were given is the local node, we resolve the name to the address and
then check our list of interfaces. The loopback device is ignored and
therefore not on the list. So if you resolve to the 127 address, we will
decide this is a different node than the one we are on.

I can modify that logic, but want to ensure this accurately captures the
problem. I'll also have to discuss the change with the other developers to
ensure we don't shoot ourselves in the foot if we make it.



On Thu, Jun 20, 2013 at 2:56 AM, Riccardo Murri wrote:

> On 20 June 2013 06:33, Ralph Castain  wrote:
> > Been trying to decipher this problem, and think maybe I'm beginning to
> > understand it. Just to clarify:
> >
> > * when you execute "hostname", you get the .local response?
>
> Yes:
>
> [rmurri@nh64-2-11 ~]$ hostname
> nh64-2-11.local
>
> [rmurri@nh64-2-11 ~]$ uname -n
> nh64-2-11.local
>
> [rmurri@nh64-2-11 ~]$ hostname -s
> nh64-2-11
>
> [rmurri@nh64-2-11 ~]$ hostname -f
> nh64-2-11.local
>
>
> > * you somewhere have it setup so that 10.x.x.x resolves to , with
> no
> > ".local" extension?
>
> No. Host name resolution is correct, but the hostname resolves to the
> 127.0.1.1 address:
>
> [rmurri@nh64-2-11 ~]$ getent hosts `hostname`
> 127.0.1.1nh64-2-11.local nh64-2-11
>
> Note that `/etc/hosts` also lists a 10.x.x.x address, which is the one
> actually assigned to the ethernet interface:
>
> [rmurri@nh64-2-11 ~]$ fgrep `hostname -s` /etc/hosts
> 127.0.1.1   nh64-2-11.local nh64-2-11
> 10.1.255.201nh64-2-11.local nh64-2-11
> 192.168.255.206 nh64-2-11-myri0
>
> If we remove the `127.0.1.1` line from `/etc/hosts`, then everything
> works again.  Also, everything works if we use only FQDNs in the
> hostfile.
>
> So it seems that the 127.0.1.1 address is treated specially.
>
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

On 20 June 2013 11:29, Riccardo Murri  wrote:
> However, I cannot reproduce the issue now

Just to be clear: the "issue" in that mail refers to the OpenMPI SGE
ras plugin not working with our version of SGE.

The issue with 127.0.1.1 addresses is reproducible at will.

Thanks,
Riccardo

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Jeff Squyres (jsquyres)]

Er... are you having problems with host IP addresses 127.0.1.1, or did you 
reply to the wrong thread?  

I thought you were asking about problems with multiple mpf90's in your PATH, 
etc. -- not 127.0.1.1 IP address issues.  IIRC, there were a bunch of 
suggestions over on that thread about how to fix your problem.  If those were 
not helpful to you, it might be easier to find a local Linux/OS X/shell guru 
and get them to help you setup your PATH / LD_LIBRARY_PATH correctly, and give 
you a quick tutorial on shell basics.


On Jun 20, 2013, at 10:04 AM, Lorenzo Donà  wrote:

> Dear all that help me thanks to everyone.
> I compiled open MPI with all yours advices posted but the error is always the 
> same I'm also able to run the examples found with the package.
> but really I don't know what can I do to solve the problem.
> I trust in you to help me.
> Dearly Lorenzo.
>  
> Il giorno 20/giu/2013, alle ore 06.33, Ralph Castain ha scritto:
> 
>> Been trying to decipher this problem, and think maybe I'm beginning to 
>> understand it. Just to clarify:
>> 
>> * when you execute "hostname", you get the .local response?
>> 
>> * you somewhere have it setup so that 10.x.x.x resolves to , with no 
>> ".local" extension?
>> 
>> Correct?
>> 
>> 
>> 
>> On Wed, Jun 19, 2013 at 1:17 PM, Riccardo Murri  
>> wrote:
>> On 19 June 2013 20:42, Ralph Castain  wrote:
>> > I'm assuming that the offending host has some other address besides
>> > just 127.0.1.1 as otherwise it couldn't connect to anything.
>> 
>> Yes, it has an IP on some 10.x.x.x network.
>> 
>> 
>> > I'm heading out the door for a couple of weeks, but can try to look at it 
>> > when I return.
>> 
>> We have a workaround (just create the hostfile using FQDNs -- actually
>> FQDNs or UQDNS depending on what `uname -n` returns), so it's
>> definitely not urgent for us.  But if you think it's a bug worth
>> fixing, I can provide details and/or test code.
>> 
>> Thanks,
>> Riccardo
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Lorenzo Donà]

Dear all that help me thanks to everyone.
I compiled open MPI with all yours advices posted but the error is always the 
same I'm also able to run the examples found with the package.
but really I don't know what can I do to solve the problem.
I trust in you to help me.
Dearly Lorenzo.
 
Il giorno 20/giu/2013, alle ore 06.33, Ralph Castain ha scritto:

> Been trying to decipher this problem, and think maybe I'm beginning to 
> understand it. Just to clarify:
> 
> * when you execute "hostname", you get the .local response?
> 
> * you somewhere have it setup so that 10.x.x.x resolves to , with no 
> ".local" extension?
> 
> Correct?
> 
> 
> 
> On Wed, Jun 19, 2013 at 1:17 PM, Riccardo Murri  wrote:
> On 19 June 2013 20:42, Ralph Castain  wrote:
> > I'm assuming that the offending host has some other address besides
> > just 127.0.1.1 as otherwise it couldn't connect to anything.
> 
> Yes, it has an IP on some 10.x.x.x network.
> 
> 
> > I'm heading out the door for a couple of weeks, but can try to look at it 
> > when I return.
> 
> We have a workaround (just create the hostfile using FQDNs -- actually
> FQDNs or UQDNS depending on what `uname -n` returns), so it's
> definitely not urgent for us.  But if you think it's a bug worth
> fixing, I can provide details and/or test code.
> 
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

On 19 June 2013 23:52, Reuti  wrote:
> Am 19.06.2013 um 22:14 schrieb Riccardo Murri:
>
>> On 19 June 2013 20:42, Reuti  wrote:
>>> Am 19.06.2013 um 19:43 schrieb Riccardo Murri :
>>>
 On 19 June 2013 16:01, Ralph Castain  wrote:
> How is OMPI picking up this hostfile? It isn't being specified on the cmd 
> line - are you running under some resource manager?

 Via the environment variable `OMPI_MCA_orte_default_hostfile`.

 We're running under SGE, but disable the OMPI/SGE integration (rather
>
> BTW: Which version of SGE?

SGE6.2u4 running under Rocks 5.3:

$ qstat -h
GE 6.2u4

$ cat /etc/rocks-release
Rocks release 5.3 (Rolled Tacos)


>> It's enabled but (IIRC) the problem is that OpenMPI detects the
>> presence of SGE from some environment variable
>
> Correct.
>
>
>> , which, in our version
>> of SGE, simply isn't there.
>
> Do you use a custom "starter_method" in the queue definition?

No custom starter_method.


> Does a submitted script with:
>
> #!/bin/sh
> env
>
> list at least some of the SGE* environment variables - or none at all?

Quite some SGE_* variables are in the environment:

$ cat env.sh
env | sort

$ qsub -pe mpi 2 env.sh
Your job 29590 ("env.sh") has been submitted

$ egrep ^SGE_ env.sh.o29590
SGE_ACCOUNT=sge
SGE_ARCH=lx26-amd64
...

However, I cannot reproduce the issue now -- it's quite possible that
it originated on a older cluster (now decommisioned) and we just kept
the submission
script on newer hardware without checking.

Thanks for the help,
Riccardo

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

On 20 June 2013 06:33, Ralph Castain  wrote:
> Been trying to decipher this problem, and think maybe I'm beginning to
> understand it. Just to clarify:
>
> * when you execute "hostname", you get the .local response?

Yes:

[rmurri@nh64-2-11 ~]$ hostname
nh64-2-11.local

[rmurri@nh64-2-11 ~]$ uname -n
nh64-2-11.local

[rmurri@nh64-2-11 ~]$ hostname -s
nh64-2-11

[rmurri@nh64-2-11 ~]$ hostname -f
nh64-2-11.local


> * you somewhere have it setup so that 10.x.x.x resolves to , with no
> ".local" extension?

No. Host name resolution is correct, but the hostname resolves to the
127.0.1.1 address:

[rmurri@nh64-2-11 ~]$ getent hosts `hostname`
127.0.1.1nh64-2-11.local nh64-2-11

Note that `/etc/hosts` also lists a 10.x.x.x address, which is the one
actually assigned to the ethernet interface:

[rmurri@nh64-2-11 ~]$ fgrep `hostname -s` /etc/hosts
127.0.1.1   nh64-2-11.local nh64-2-11
10.1.255.201nh64-2-11.local nh64-2-11
192.168.255.206 nh64-2-11-myri0

If we remove the `127.0.1.1` line from `/etc/hosts`, then everything
works again.  Also, everything works if we use only FQDNs in the
hostfile.

So it seems that the 127.0.1.1 address is treated specially.

Thanks,
Riccardo

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Ralph Castain]

Been trying to decipher this problem, and think maybe I'm beginning to
understand it. Just to clarify:

* when you execute "hostname", you get the .local response?

* you somewhere have it setup so that 10.x.x.x resolves to , with no
".local" extension?

Correct?



On Wed, Jun 19, 2013 at 1:17 PM, Riccardo Murri wrote:

> On 19 June 2013 20:42, Ralph Castain  wrote:
> > I'm assuming that the offending host has some other address besides
> > just 127.0.1.1 as otherwise it couldn't connect to anything.
>
> Yes, it has an IP on some 10.x.x.x network.
>
>
> > I'm heading out the door for a couple of weeks, but can try to look at
> it when I return.
>
> We have a workaround (just create the hostfile using FQDNs -- actually
> FQDNs or UQDNS depending on what `uname -n` returns), so it's
> definitely not urgent for us.  But if you think it's a bug worth
> fixing, I can provide details and/or test code.
>
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Reuti]

Am 19.06.2013 um 22:14 schrieb Riccardo Murri:

> On 19 June 2013 20:42, Reuti  wrote:
>> Am 19.06.2013 um 19:43 schrieb Riccardo Murri :
>> 
>>> On 19 June 2013 16:01, Ralph Castain  wrote:
 How is OMPI picking up this hostfile? It isn't being specified on the cmd 
 line - are you running under some resource manager?
>>> 
>>> Via the environment variable `OMPI_MCA_orte_default_hostfile`.
>>> 
>>> We're running under SGE, but disable the OMPI/SGE integration (rather

BTW: Which version of SGE?


>> 
>> It's disabled by default, you would have to activate it during `configure` 
>> of Open MPI.
> 
> It's enabled but (IIRC) the problem is that OpenMPI detects the
> presence of SGE from some environment variable

Correct.


> , which, in our version
> of SGE, simply isn't there.

Do you use a custom "starter_method" in the queue definition? Does a submitted 
script with:

#!/bin/sh
env

list at least some of the SGE* environment variables - or none at all?

-- Reuti


> I can dig up the details if you're interested.
> 
> Regards,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

On 19 June 2013 20:42, Reuti  wrote:
> Am 19.06.2013 um 19:43 schrieb Riccardo Murri :
>
>> On 19 June 2013 16:01, Ralph Castain  wrote:
>>> How is OMPI picking up this hostfile? It isn't being specified on the cmd 
>>> line - are you running under some resource manager?
>>
>> Via the environment variable `OMPI_MCA_orte_default_hostfile`.
>>
>> We're running under SGE, but disable the OMPI/SGE integration (rather
>
> It's disabled by default, you would have to activate it during `configure` of 
> Open MPI.

It's enabled but (IIRC) the problem is that OpenMPI detects the
presence of SGE from some environment variable, which, in our version
of SGE, simply isn't there.

I can dig up the details if you're interested.

Regards,
Riccardo

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Reuti]

Am 19.06.2013 um 19:43 schrieb Riccardo Murri :

> On 19 June 2013 16:01, Ralph Castain  wrote:
>> How is OMPI picking up this hostfile? It isn't being specified on the cmd 
>> line - are you running under some resource manager?
> 
> Via the environment variable `OMPI_MCA_orte_default_hostfile`.
> 
> We're running under SGE, but disable the OMPI/SGE integration (rather

It's disabled by default, you would have to activate it during `configure` of 
Open MPI.


> old version of SGE, does not coordinate well with OpenMPI);

In what sense? What do you observe in case you use it? The `qrsh` startup is 
working fine for a long time now.

-- Reuti


> here's the
> relevant snippet from our startup script:
> 
># the OMPI/SGE integration does not seem to work with
># our SGE version; so use the `mpi` PE and direct OMPI
># to look for a "plain old" machine file
>unset PE_HOSTFILE
>if [ -r "${TMPDIR}/machines" ]; then
>OMPI_MCA_orte_default_hostfile="${TMPDIR}/machines"
>export OMPI_MCA_orte_default_hostfile
>fi
>GMSCOMMAND="$openmpi_root/bin/mpiexec -n $NCPUS --nooversubscribe
> $gamess $INPUT -scr $(pwd)"
> 
> The `$TMPDIR/machines` hostfile is created from SGE's $PE_HOSTFILE by
> extracting the host names, and repeating each one for the given number
> of slots (unmodified code that comes with SGE):
> 
>PeHostfile2MachineFile()
>{
>   cat $1 | while read line; do
>  # echo $line
>  host=`echo $line|cut -f1 -d" "|cut -f1 -d"."`
>  nslots=`echo $line|cut -f2 -d" "`
>  i=1
>  while [ $i -le $nslots ]; do
> echo $host
> i=`expr $i + 1`
>  done
>   done
>}
> 
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Ralph Castain]

Hmmm..certainly sounds like a bug. It should pickup that the node is local. It 
checks the hostname (as returned by gethostname), but it also checks to see if 
host resolves to a local address. I'm assuming that the offending host has some 
other address besides just 127.0.1.1 as otherwise it couldn't connect to 
anything.

I'm heading out the door for a couple of weeks, but can try to look at it when 
I return.

On Jun 19, 2013, at 10:43 AM, Riccardo Murri  wrote:

> On 19 June 2013 16:01, Ralph Castain  wrote:
>> How is OMPI picking up this hostfile? It isn't being specified on the cmd 
>> line - are you running under some resource manager?
> 
> Via the environment variable `OMPI_MCA_orte_default_hostfile`.
> 
> We're running under SGE, but disable the OMPI/SGE integration (rather
> old version of SGE, does not coordinate well with OpenMPI); here's the
> relevant snippet from our startup script:
> 
># the OMPI/SGE integration does not seem to work with
># our SGE version; so use the `mpi` PE and direct OMPI
># to look for a "plain old" machine file
>unset PE_HOSTFILE
>if [ -r "${TMPDIR}/machines" ]; then
>OMPI_MCA_orte_default_hostfile="${TMPDIR}/machines"
>export OMPI_MCA_orte_default_hostfile
>fi
>GMSCOMMAND="$openmpi_root/bin/mpiexec -n $NCPUS --nooversubscribe
> $gamess $INPUT -scr $(pwd)"
> 
> The `$TMPDIR/machines` hostfile is created from SGE's $PE_HOSTFILE by
> extracting the host names, and repeating each one for the given number
> of slots (unmodified code that comes with SGE):
> 
>PeHostfile2MachineFile()
>{
>   cat $1 | while read line; do
>  # echo $line
>  host=`echo $line|cut -f1 -d" "|cut -f1 -d"."`
>  nslots=`echo $line|cut -f2 -d" "`
>  i=1
>  while [ $i -le $nslots ]; do
> echo $host
> i=`expr $i + 1`
>  done
>   done
>}
> 
> Thanks,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

On 19 June 2013 16:01, Ralph Castain  wrote:
> How is OMPI picking up this hostfile? It isn't being specified on the cmd 
> line - are you running under some resource manager?

Via the environment variable `OMPI_MCA_orte_default_hostfile`.

We're running under SGE, but disable the OMPI/SGE integration (rather
old version of SGE, does not coordinate well with OpenMPI); here's the
relevant snippet from our startup script:

# the OMPI/SGE integration does not seem to work with
# our SGE version; so use the `mpi` PE and direct OMPI
# to look for a "plain old" machine file
unset PE_HOSTFILE
if [ -r "${TMPDIR}/machines" ]; then
OMPI_MCA_orte_default_hostfile="${TMPDIR}/machines"
export OMPI_MCA_orte_default_hostfile
fi
GMSCOMMAND="$openmpi_root/bin/mpiexec -n $NCPUS --nooversubscribe
$gamess $INPUT -scr $(pwd)"

The `$TMPDIR/machines` hostfile is created from SGE's $PE_HOSTFILE by
extracting the host names, and repeating each one for the given number
of slots (unmodified code that comes with SGE):

PeHostfile2MachineFile()
{
   cat $1 | while read line; do
  # echo $line
  host=`echo $line|cut -f1 -d" "|cut -f1 -d"."`
  nslots=`echo $line|cut -f2 -d" "`
  i=1
  while [ $i -le $nslots ]; do
 echo $host
 i=`expr $i + 1`
  done
   done
}

Thanks,
Riccardo

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Ralph Castain]

How is OMPI picking up this hostfile? It isn't being specified on the cmd line 
- are you running under some resource manager?

I haven't seen that confusion elsewhere, so I'm trying to understand which code 
path is involved - hence the questions.

On Jun 19, 2013, at 6:26 AM, Riccardo Murri  wrote:

> Hi,
> 
> (colleague of OP here)
> 
> On 19 June 2013 15:09, Ralph Castain  wrote:
>> I don't see a hostfile on your command line - so I assume you are using a
>> default hostfile? What is in it?
> 
> The hostfile comes from the batch system; it just contains the
> unqualified host names:
> 
>$ cat $TMPDIR/machines
>nh64-1-17
>nh64-1-17
> 
> No problem if we modify the setup script to create the hostfile using
> FQDNs instead.  (`uname -n` returns the FQDN, not the unqualified host name.)
> 
> Thanks,
> Riccardo
> 
> --
> Riccardo Murri
> http://www.gc3.uzh.ch/people/rm
> 
> Grid Computing Competence Centre
> University of Zurich
> Winterthurerstrasse 190, CH-8057 Zürich (Switzerland)
> Tel: +41 44 635 4222
> Fax: +41 44 635 6888
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Riccardo Murri]

Hi,

(colleague of OP here)

On 19 June 2013 15:09, Ralph Castain  wrote:
> I don't see a hostfile on your command line - so I assume you are using a
> default hostfile? What is in it?

The hostfile comes from the batch system; it just contains the
unqualified host names:

$ cat $TMPDIR/machines
nh64-1-17
nh64-1-17

No problem if we modify the setup script to create the hostfile using
FQDNs instead.  (`uname -n` returns the FQDN, not the unqualified host name.)

Thanks,
Riccardo

--
Riccardo Murri
http://www.gc3.uzh.ch/people/rm

Grid Computing Competence Centre
University of Zurich
Winterthurerstrasse 190, CH-8057 Zürich (Switzerland)
Tel: +41 44 635 4222
Fax: +41 44 635 6888

Re: [OMPI users] openmpi 1.6.3 fails to identify local host if its IP is 127.0.1.1

[Ralph Castain]

I don't see a hostfile on your command line - so I assume you are using a 
default hostfile? What is in it?

On Jun 19, 2013, at 1:49 AM, Sergio Maffioletti  
wrote:

> Hello,
> 
> we have been hit observing a strange behavior with OpenMPI 1.6.3
> 
> strace -f /share/apps/openmpi/1.6.3/bin/mpiexec -n 2
> --nooversubscribe --display-allocation --display-map --tag-output
> /share/apps/gamess/2011R1/gamess.2011R1.x
> /state/partition1/rmurri/29515/exam01.F05 -scr
> /state/partition1/rmurri/29515
> 
> ==   ALLOCATED NODES   ==
> 
>  Data for node: nh64-1-17.local Num slots: 0Max slots: 0
>  Data for node: nh64-1-17   Num slots: 2Max slots: 0
> 
> =
> 
>     JOB MAP   
> 
>  Data for node: nh64-1-17   Num procs: 2
> Process OMPI jobid: [37108,1] Process rank: 0
> Process OMPI jobid: [37108,1] Process rank: 1
> 
>  =
> 
> As you can see, the host file lists the *unqualified* local host name;
> OpenMPI fails to recognize that as the same host where it is running,
> and uses `ssh` to spawn a remote `orted`, as use of `strace -f` shows:
> 
> Process 16552 attached
> [pid 16552] execve("//usr/bin/ssh", ["/usr/bin/ssh", "-x",
> "nh64-1-17", "OPAL_PREFIX=/share/apps/openmpi/1.6.3 ; export
> OPAL_PREFIX; PATH=/share/apps/openmpi/1.6.3/bin:$PATH ; export PATH ;
> LD_LIBRARY_PATH=/share/apps/openmpi/1.6.3/lib:$LD_LIBRARY_PATH ;
> export LD_LIBRARY_PATH ;
> DYLD_LIBRARY_PATH=/share/apps/openmpi/1.6.3/lib:$", "--daemonize",
> "-mca", "ess", "env", "-mca", "orte_ess_jobid", "2431909888", "-mca",
> "orte_ess_vpid", "1", "-mca", "orte_ess_num_procs", "2", "--hnp-uri",
> "\"2431909888.0;tcp://10.1.255.237:33154\"", "-mca", "plm", "rsh"],
> ["OLI235=/state/partition1/rmurri/29515/exam01.F235", ...
> 
> If the machine file lists the FQDNs instead, `mpiexec` spawns the jobs
> directly via fork()/exec().
> 
> This seems related to the fact that each compute node advertises
> 127.0.1.1 as the IP address associated to its hostname:
> 
> $ ssh nh64-1-17 getent hosts nh64-1-17
> 127.0.1.1nh64-1-17.local nh64-1-17
> 
> Indeed, if I change /etc/hosts so that a compute node associates a
> "real" IP with its hostname, `mpiexec` works as expected.
> 
> Is this a known feature/bug/easter egg?
> 
> For the record: using OpenMPI 1.6.3 on Rocks 5.2.
> 
> Thanks,
> on behalf of the GC3 Team
> Sergio :)
> 
> GC3: Grid Computing Competence Center
> http://www.gc3.uzh.ch/
> University of Zurich
> Winterthurerstrasse 190
> CH-8057 Zurich Switzerland
> Tel: +41 44 635 4222
> Fax: +41 44 635 6888
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users