Hi Josh
It was my mistake. The status of error generating node is pasted below
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0018:8b90:97fe:94fe
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 4: PortConfigurationTraining
rate: 10 Gb/sec (4X)
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:0018:8b90:97fe:94ff
base lid: 0x29
sm lid: 0x15
As you see one port is down. I have all sysadmin rights as I am managing
the cluster, but my level of knowledge is not expert. Can you explain a bit
about ports. Does each infiniband card in a system has 2 physical ports?
What to look for if one port status is down.?
Ahsan
On Tue, Jul 22, 2014 at 6:14 PM, Joshua Ladd <jladd.m...@gmail.com> wrote:
> Sayed,
>
> You might try this link (or have your sysadmin do it if you do not have
> admin privileges.) To me it looks like your second port is in the "INIT"
> state but has not been added by the subnet manager.
>
>
> https://software.intel.com/en-us/articles/troubleshooting-infiniband-connection-issues-using-ofed-tools
>
> You might also try try running only over port 1 with the mca parameter:
>
> -mca btl_openib_if_include mlx4_0:1
>
> Hope this helps.
>
> Josh
>
>
> On Tue, Jul 22, 2014 at 12:10 AM, Syed Ahsan Ali <ahsansha...@gmail.com>
> wrote:
>
>> And where I can find run/job/submission ?
>>
>> On Mon, Jul 21, 2014 at 6:57 PM, Shamis, Pavel <sham...@ornl.gov> wrote:
>>
>>>
>>> You have to check the ports states on *all* nodes in the
>>> run/job/submission. Checking on a single node is not enough.
>>> My guess is the 01-00 tries to connect 01-01 and the ports are down on
>>> 01-01.
>>>
>>> You may disable support for infiniband by adding --mca btl ^openib.
>>>
>>> Best,
>>> Pavel (Pasha) Shamis
>>> ---
>>> Computer Science Research Group
>>> Computer Science and Math Division
>>> Oak Ridge National Laboratory
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Jul 21, 2014, at 3:17 AM, Syed Ahsan Ali <ahsansha...@gmail.com
>>> <mailto:ahsansha...@gmail.com>> wrote:
>>>
>>> Dear All
>>>
>>> I need your help to solve this cluster related issue causing mpirun
>>> malfunction. I get following warning for some of the nodes and then the
>>> route failure message comes causing failure to mpirun.
>>>
>>>
>>> WARNING: There is at least one OpenFabrics device found but there are no
>>> active ports detected (or Open MPI was unable to use them). This
>>> is most certainly not what you wanted. Check your cables, subnet
>>> manager configuration, etc. The openib BTL will be ignored for this
>>> job.
>>> Local host: compute-01-01.private.dns.zone
>>>
>>> --------------------------------------------------------------------------
>>> SETUP OF THE LM
>>> INITIALIZATIONS
>>> INPUT OF THE NAMELISTS
>>> [pmd.pakmet.com:30198<http://pmd.pakmet.com:30198/>] 7 more processes
>>> have sent help message help-mpi-btl-openib.txt / no active ports found
>>> [pmd.pakmet.com:30198<http://pmd.pakmet.com:30198/>] Set MCA parameter
>>> "orte_base_help_aggregate" to 0 to see all help / error messages
>>> [compute-01-00.private.dns.zone][[40500,1],5][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
>>> connect() to 192.168.108.14 failed: No route to host (113)
>>> [compute-01-00.private.dns.zone][[40500,1],4][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
>>> connect() to 192.168.108.14 failed: No route to host (113)
>>> My questions are.
>>> I don't include flags for running openmpi over infiniband then why it
>>> still gives warning. If the infiniband ports are not active then it should
>>> start the job over gigabit ethernet of cluster. Why it is unable to find
>>> the route while the node can be pinged and ssh from other nodes and master
>>> node as well.
>>> The ibstatus of the above node (for which I was getting error) shows
>>> that both ports are up. What is causing error then?
>>>
>>> [root@compute-01-00 ~]# ibstatus
>>> Infiniband device 'mlx4_0' port 1 status:
>>> default gid: fe80:0000:0000:0000:0024:e890:97ff:1c61
>>> base lid: 0x5
>>> sm lid: 0x1
>>> state: 4: ACTIVE
>>> phys state: 5: LinkUp
>>> rate: 20 Gb/sec (4X DDR)
>>> Infiniband device 'mlx4_0' port 2 status:
>>> default gid: fe80:0000:0000:0000:0024:e890:97ff:1c62
>>> base lid: 0x0
>>> sm lid: 0x0
>>> state: 2: INIT
>>> phys state: 5: LinkUp
>>> rate: 20 Gb/sec (4X DDR)
>>>
>>>
>>> Thank you in advance for your guidance and support.
>>>
>>> Regards
>>>
>>> --
>>> Ahsan
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org<mailto:us...@open-mpi.org>
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2014/07/24833.php
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2014/07/24835.php
>>>
>>
>>
>>
>> --
>> Syed Ahsan Ali Bokhari
>> Electronic Engineer (EE)
>>
>> Research & Development Division
>> Pakistan Meteorological Department H-8/4, Islamabad.
>> Phone # off +92518358714
>> Cell # +923155145014
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/07/24842.php
>>
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/07/24844.php
>
--
Syed Ahsan Ali Bokhari
Electronic Engineer (EE)
Research & Development Division
Pakistan Meteorological Department H-8/4, Islamabad.
Phone # off +92518358714
Cell # +923155145014
