HI, Thanks for replying In this artectuire, when we are doing ibv_rc_pingpong between two nodes connected with same switch we are getting result. But when we use two nodes with 2 switches we are getting error.
Success:- [root@oss1 ~]# ibv_rc_pingpong local address: LID 0x001e, QPN 0x2c004a, PSN 0x554863, GID :: remote address: LID 0x0022, QPN 0x20004a, PSN 0x7c9dc2, GID :: 8192000 bytes in 0.01 seconds = 6992.74 Mbit/sec 1000 iters in 0.01 seconds = 9.37 usec/iter [root@oss1 ~]# [root@mds1 ~]# ibv_rc_pingpong 173.16.1.52 local address: LID 0x0022, QPN 0x20004a, PSN 0x7c9dc2, GID :: remote address: LID 0x001e, QPN 0x2c004a, PSN 0x554863, GID :: 8192000 bytes in 0.01 seconds = 7084.97 Mbit/sec 1000 iters in 0.01 seconds = 9.25 usec/iter [root@mds1 ~]# Error [root@nalanda mvapich2-1.9]# ibv_rc_pingpong local address: LID 0x0001, QPN 0x56004e, PSN 0x704d51 remote address: LID 0x0022, QPN 0x1c004a, PSN 0x07a0b2 [root@mds1 ~]# ibv_rc_pingpong 173.16.1.1 local address: LID 0x0022, QPN 0x1c004a, PSN 0x07a0b2, GID :: client read: Success Couldn't read remote address [root@mds1 ~]# And how we test our ftree topology is working fine. Please go through the attachment. Thank You Atul Yadav On Sat, Apr 12, 2014 at 12:14 AM, Hal Rosenstock <h...@dev.mellanox.co.il>wrote: > On 4/11/2014 2:21 PM, Atul Yadav wrote: > > Dear Team, > > > > We are trying to build Fat tree topology. > > The details are given below: > > Unmanaged switches 36 port quantity 5 > > As per the some blog we need to modify the opensm.conf file > > But we are unable to identify some parameter like:- > > root_guid_file ??????? > > Fat tree routing will try to autodetect the roots but this may not work > and it is better to specify the root GUIDs. In your case, they are the > GUIDs for switches A and B. > > The root GUID file is then provided to OpenSM either via the conf file or > command line parameters. The command line parameter is [-a | > --root_guid_file <path to file>] > > OpenSM man page says: > > -a, --root_guid_file <file name> > Set the root nodes for the Up/Down or Fat-Tree routing > algorithm > to the guids provided in the given file (one to a line). > > It also says: > > If the root guid file is not provided (?-a? or > ?--root_guid_file? > options), the topology has to be pure fat-tree that complies with > the > following rules: > - Tree rank should be between two and eight (inclusively) > - Switches of the same rank should have the same number > of UP-going port groups*, unless they are root switches, > in which case the shouldn?t have UP-going ports at all. > - Switches of the same rank should have the same number > of DOWN-going port groups, unless they are leaf switches. > - Switches of the same rank should have the same number > of ports in each UP-going port group. > - Switches of the same rank should have the same number > of ports in each DOWN-going port group. > - All the CAs have to be at the same tree level (rank). > > If the root guid file is provided, the topology doesn?t have to be > pure > fat-tree, and it should only comply with the following rules: > - Tree rank should be between two and eight (inclusively) > - All the Compute Nodes** have to be at the same tree level > (rank). > Note that non-compute node CAs are allowed here to be at > different > tree ranks. > > * ports that are connected to the same remote switch are > referenced as > port group. > > ** list of compute nodes (CNs) can be specified by -u > or > --cn_guid_file OpenSM options. > > -- Hal > > > > > Need your input for this ? > > > > > > > > > > Thank You > > Atul Yadav > > > > > > > > > > _______________________________________________ > > ewg mailing list > > ewg@lists.openfabrics.org > > http://lists.openfabrics.org/mailman/listinfo/ewg > >
[root@nalanda mvapich2-1.9]# ibnetdiscover # # Topology file: generated on Sat Apr 12 16:22:11 2014 # # Initiated from node 0002c9030042e420 port 0002c9030042e421 vendid=0x2c9 devid=0xc738 sysimgguid=0x2c903008fd500 switchguid=0x2c903008fd500(2c903008fd500) Switch 36 "S-0002c903008fd500" # "SwitchX - Mellanox Technologies" base port 0 lid 4 lmc 0 [1] "S-0002c90300901480"[10] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [2] "S-0002c90300901480"[11] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [3] "S-0002c90300901480"[12] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [4] "S-0002c90300901480"[13] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [5] "S-0002c90300901480"[14] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [6] "S-0002c90300901480"[15] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [7] "S-0002c90300901480"[16] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [8] "S-0002c90300901480"[17] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [9] "S-0002c90300901480"[18] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [10] "S-0002c90300902c00"[10] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [11] "S-0002c90300902c00"[11] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [12] "S-0002c90300902c00"[12] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [13] "S-0002c90300902c00"[13] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [14] "S-0002c90300902c00"[14] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [15] "S-0002c90300902c00"[15] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [16] "S-0002c90300902c00"[16] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [17] "S-0002c90300902c00"[17] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [18] "S-0002c90300902c00"[18] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [19] "H-0002c9030042e440"[1](2c9030042e441) # "compute-0-11 HCA-1" lid 19 4xQDR [20] "H-0002c9030042e300"[1](2c9030042e301) # "compute-0-10 HCA-1" lid 6 4xQDR [21] "H-0002c9030042e390"[1](2c9030042e391) # "compute-0-7 HCA-1" lid 32 4xQDR [22] "H-0002c9030042e450"[1](2c9030042e451) # "compute-0-6 HCA-1" lid 22 4xQDR [23] "H-0002c9030042e380"[1](2c9030042e381) # "compute-0-13 HCA-1" lid 28 4xQDR [24] "H-0002c9030042e360"[1](2c9030042e361) # "compute-0-14 HCA-1" lid 24 4xQDR [25] "H-0002c9030042e320"[1](2c9030042e321) # "compute-0-9 HCA-1" lid 14 4xQDR [26] "H-0002c9030042e3e0"[1](2c9030042e3e1) # "compute-0-8 HCA-1" lid 38 4xQDR [27] "H-0002c9030042e330"[1](2c9030042e331) # "compute-0-2 HCA-1" lid 17 4xQDR [28] "H-24be05ffff856df0"[1](24be05ffff856df1) # "compute-0-29 HCA-1" lid 39 4xQDR [29] "H-0002c90300423a10"[1](2c90300423a11) # "compute-0-32 HCA-1" lid 9 4xQDR [30] "H-24be05ffff85bc90"[1](24be05ffff85bc91) # "compute-0-23 HCA-1" lid 31 4xQDR [31] "H-24be05ffff85bce0"[1](24be05ffff85bce1) # "compute-0-24 HCA-1" lid 36 4xQDR vendid=0x2c9 devid=0xc738 sysimgguid=0x2c90300903600 switchguid=0x2c90300903600(2c90300903600) Switch 36 "S-0002c90300903600" # "SwitchX - Mellanox Technologies" base port 0 lid 3 lmc 0 [1] "S-0002c90300902c00"[1] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [2] "S-0002c90300902c00"[2] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [3] "S-0002c90300902c00"[3] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [4] "S-0002c90300902c00"[4] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [5] "S-0002c90300902c00"[5] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [6] "S-0002c90300902c00"[6] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [7] "S-0002c90300902c00"[7] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [8] "S-0002c90300902c00"[8] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [9] "S-0002c90300902c00"[9] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [10] "S-0002c90300901480"[19] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [11] "S-0002c90300901480"[20] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [12] "S-0002c90300901480"[21] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [13] "S-0002c90300901480"[22] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [14] "S-0002c90300901480"[23] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [15] "S-0002c90300901480"[24] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [16] "S-0002c90300901480"[25] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [17] "S-0002c90300901480"[26] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [18] "S-0002c90300901480"[27] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [19] "H-0002c90300524982"[1](2c90300524983) # "oss2 mlx4_0" lid 29 4xQDR [20] "H-0002c90300524992"[1](2c90300524993) # "mds2 mlx4_0" lid 33 4xQDR [21] "H-0002c9030052499a"[1](2c9030052499b) # "mds1 mlx4_0" lid 34 4xQDR [22] "H-0002c9030052498e"[1](2c9030052498f) # "oss1 mlx4_0" lid 30 4xQDR [23] "H-24be05ffff85cc20"[1](24be05ffff85cc21) # "compute-0-28 HCA-1" lid 13 4xQDR [24] "H-24be05ffff85bcf0"[1](24be05ffff85bcf1) # "compute-0-26 HCA-1" lid 40 4xQDR [25] "H-24be05ffff85cc30"[1](24be05ffff85cc31) # "compute-0-30 HCA-1" lid 16 4xQDR [26] "H-0002c9030041f800"[1](2c9030041f801) # "compute-0-31 HCA-1" lid 8 4xQDR [27] "H-24be05ffff856d00"[1](24be05ffff856d01) # "compute-0-27 HCA-1" lid 5 4xQDR [28] "H-24be05ffff85cc60"[1](24be05ffff85cc61) # "compute-0-25 HCA-1" lid 23 4xQDR [29] "H-0002c9030041f7f0"[1](2c9030041f7f1) # "compute-0-34 HCA-1" lid 44 4xQDR [30] "H-0002c9030041f840"[1](2c9030041f841) # "compute-0-33 HCA-1" lid 20 4xQDR vendid=0x2c9 devid=0xc738 sysimgguid=0x2c90300902c00 switchguid=0x2c90300902c00(2c90300902c00) Switch 36 "S-0002c90300902c00" # "SwitchX - Mellanox Technologies" base port 0 lid 2 lmc 0 [1] "S-0002c90300903600"[1] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [2] "S-0002c90300903600"[2] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [3] "S-0002c90300903600"[3] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [4] "S-0002c90300903600"[4] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [5] "S-0002c90300903600"[5] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [6] "S-0002c90300903600"[6] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [7] "S-0002c90300903600"[7] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [8] "S-0002c90300903600"[8] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [9] "S-0002c90300903600"[9] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [10] "S-0002c903008fd500"[10] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [11] "S-0002c903008fd500"[11] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [12] "S-0002c903008fd500"[12] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [13] "S-0002c903008fd500"[13] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [14] "S-0002c903008fd500"[14] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [15] "S-0002c903008fd500"[15] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [16] "S-0002c903008fd500"[16] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [17] "S-0002c903008fd500"[17] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [18] "S-0002c903008fd500"[18] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [19] "S-0002c9030075c870"[10] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [20] "S-0002c9030075c870"[11] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [21] "S-0002c9030075c870"[12] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [23] "S-0002c9030075c870"[14] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [24] "S-0002c9030075c870"[15] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [25] "S-0002c9030075c870"[16] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [26] "S-0002c9030075c870"[17] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [27] "S-0002c9030075c870"[18] # "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0xc738 sysimgguid=0x2c90300901480 switchguid=0x2c90300901480(2c90300901480) Switch 36 "S-0002c90300901480" # "SwitchX - Mellanox Technologies" base port 0 lid 27 lmc 0 [1] "S-0002c9030075c870"[1] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [2] "S-0002c9030075c870"[2] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [3] "S-0002c9030075c870"[3] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [4] "S-0002c9030075c870"[4] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [5] "S-0002c9030075c870"[5] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [6] "S-0002c9030075c870"[6] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [7] "S-0002c9030075c870"[7] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [8] "S-0002c9030075c870"[8] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [9] "S-0002c9030075c870"[9] # "SwitchX - Mellanox Technologies" lid 25 4xQDR [10] "S-0002c903008fd500"[1] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [11] "S-0002c903008fd500"[2] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [12] "S-0002c903008fd500"[3] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [13] "S-0002c903008fd500"[4] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [14] "S-0002c903008fd500"[5] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [15] "S-0002c903008fd500"[6] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [16] "S-0002c903008fd500"[7] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [17] "S-0002c903008fd500"[8] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [18] "S-0002c903008fd500"[9] # "SwitchX - Mellanox Technologies" lid 4 4xQDR [19] "S-0002c90300903600"[10] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [20] "S-0002c90300903600"[11] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [21] "S-0002c90300903600"[12] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [22] "S-0002c90300903600"[13] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [23] "S-0002c90300903600"[14] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [24] "S-0002c90300903600"[15] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [25] "S-0002c90300903600"[16] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [26] "S-0002c90300903600"[17] # "SwitchX - Mellanox Technologies" lid 3 4xQDR [27] "S-0002c90300903600"[18] # "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0xc738 sysimgguid=0x2c9030075c870 switchguid=0x2c9030075c870(2c9030075c870) Switch 36 "S-0002c9030075c870" # "SwitchX - Mellanox Technologies" base port 0 lid 25 lmc 0 [1] "S-0002c90300901480"[1] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [2] "S-0002c90300901480"[2] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [3] "S-0002c90300901480"[3] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [4] "S-0002c90300901480"[4] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [5] "S-0002c90300901480"[5] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [6] "S-0002c90300901480"[6] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [7] "S-0002c90300901480"[7] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [8] "S-0002c90300901480"[8] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [9] "S-0002c90300901480"[9] # "SwitchX - Mellanox Technologies" lid 27 4xQDR [10] "S-0002c90300902c00"[19] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [11] "S-0002c90300902c00"[20] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [12] "S-0002c90300902c00"[21] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [14] "S-0002c90300902c00"[23] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [15] "S-0002c90300902c00"[24] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [16] "S-0002c90300902c00"[25] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [17] "S-0002c90300902c00"[26] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [18] "S-0002c90300902c00"[27] # "SwitchX - Mellanox Technologies" lid 2 4xQDR [19] "H-0002c9030042e250"[1](2c9030042e251) # "compute-0-20 HCA-1" lid 21 4xQDR [20] "H-0002c9030042e310"[1](2c9030042e311) # "compute-0-17 HCA-1" lid 10 4xQDR [21] "H-0002c9030042e4f0"[1](2c9030042e4f1) # "compute-0-3 HCA-1" lid 43 4xQDR [22] "H-0002c9030042e2e0"[1](2c9030042e2e1) # "compute-0-19 HCA-1" lid 37 4xQDR [23] "H-0002c9030042e520"[1](2c9030042e521) # "compute-0-18 HCA-1" lid 15 4xQDR [24] "H-0002c9030042e3f0"[1](2c9030042e3f1) # "compute-0-22 HCA-1" lid 42 4xQDR [25] "H-0002c9030042e400"[1](2c9030042e401) # "compute-0-1 HCA-1" lid 7 4xQDR [26] "H-0002c9030042e430"[1](2c9030042e431) # "compute-0-15 HCA-1" lid 18 4xQDR [27] "H-0002c9030042e420"[1](2c9030042e421) # "nalanda HCA-1" lid 1 4xQDR [28] "H-0002c9030042e510"[1](2c9030042e511) # "compute-0-21 HCA-1" lid 12 4xQDR [29] "H-0002c9030042e470"[1](2c9030042e471) # "compute-0-16 HCA-1" lid 26 4xQDR [30] "H-0002c9030042e2f0"[1](2c9030042e2f1) # "compute-0-12 HCA-1" lid 41 4xQDR [31] "H-0002c9030042e410"[1](2c9030042e411) # "compute-0-5 HCA-1" lid 11 4xQDR [32] "H-0002c9030042e2d0"[1](2c9030042e2d1) # "compute-0-4 HCA-1" lid 35 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030041f843 caguid=0x2c9030041f840 Ca 2 "H-0002c9030041f840" # "compute-0-33 HCA-1" [1](2c9030041f841) "S-0002c90300903600"[30] # lid 20 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030041f7f3 caguid=0x2c9030041f7f0 Ca 2 "H-0002c9030041f7f0" # "compute-0-34 HCA-1" [1](2c9030041f7f1) "S-0002c90300903600"[29] # lid 44 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff85cc63 caguid=0x24be05ffff85cc60 Ca 2 "H-24be05ffff85cc60" # "compute-0-25 HCA-1" [1](24be05ffff85cc61) "S-0002c90300903600"[28] # lid 23 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff856d03 caguid=0x24be05ffff856d00 Ca 2 "H-24be05ffff856d00" # "compute-0-27 HCA-1" [1](24be05ffff856d01) "S-0002c90300903600"[27] # lid 5 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030041f803 caguid=0x2c9030041f800 Ca 2 "H-0002c9030041f800" # "compute-0-31 HCA-1" [1](2c9030041f801) "S-0002c90300903600"[26] # lid 8 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff85cc33 caguid=0x24be05ffff85cc30 Ca 2 "H-24be05ffff85cc30" # "compute-0-30 HCA-1" [1](24be05ffff85cc31) "S-0002c90300903600"[25] # lid 16 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff85bcf3 caguid=0x24be05ffff85bcf0 Ca 2 "H-24be05ffff85bcf0" # "compute-0-26 HCA-1" [1](24be05ffff85bcf1) "S-0002c90300903600"[24] # lid 40 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff85cc23 caguid=0x24be05ffff85cc20 Ca 2 "H-24be05ffff85cc20" # "compute-0-28 HCA-1" [1](24be05ffff85cc21) "S-0002c90300903600"[23] # lid 13 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x673c sysimgguid=0x2c90300524991 caguid=0x2c9030052498e Ca 2 "H-0002c9030052498e" # "oss1 mlx4_0" [1](2c9030052498f) "S-0002c90300903600"[22] # lid 30 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x673c sysimgguid=0x2c9030052499d caguid=0x2c9030052499a Ca 2 "H-0002c9030052499a" # "mds1 mlx4_0" [1](2c9030052499b) "S-0002c90300903600"[21] # lid 34 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x673c sysimgguid=0x2c90300524995 caguid=0x2c90300524992 Ca 2 "H-0002c90300524992" # "mds2 mlx4_0" [1](2c90300524993) "S-0002c90300903600"[20] # lid 33 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x673c sysimgguid=0x2c90300524985 caguid=0x2c90300524982 Ca 2 "H-0002c90300524982" # "oss2 mlx4_0" [1](2c90300524983) "S-0002c90300903600"[19] # lid 29 lmc 0 "SwitchX - Mellanox Technologies" lid 3 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff85bce3 caguid=0x24be05ffff85bce0 Ca 2 "H-24be05ffff85bce0" # "compute-0-24 HCA-1" [1](24be05ffff85bce1) "S-0002c903008fd500"[31] # lid 36 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff85bc93 caguid=0x24be05ffff85bc90 Ca 2 "H-24be05ffff85bc90" # "compute-0-23 HCA-1" [1](24be05ffff85bc91) "S-0002c903008fd500"[30] # lid 31 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c90300423a13 caguid=0x2c90300423a10 Ca 2 "H-0002c90300423a10" # "compute-0-32 HCA-1" [1](2c90300423a11) "S-0002c903008fd500"[29] # lid 9 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x24be05ffff856df3 caguid=0x24be05ffff856df0 Ca 2 "H-24be05ffff856df0" # "compute-0-29 HCA-1" [1](24be05ffff856df1) "S-0002c903008fd500"[28] # lid 39 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e333 caguid=0x2c9030042e330 Ca 2 "H-0002c9030042e330" # "compute-0-2 HCA-1" [1](2c9030042e331) "S-0002c903008fd500"[27] # lid 17 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e3e3 caguid=0x2c9030042e3e0 Ca 2 "H-0002c9030042e3e0" # "compute-0-8 HCA-1" [1](2c9030042e3e1) "S-0002c903008fd500"[26] # lid 38 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e323 caguid=0x2c9030042e320 Ca 2 "H-0002c9030042e320" # "compute-0-9 HCA-1" [1](2c9030042e321) "S-0002c903008fd500"[25] # lid 14 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e363 caguid=0x2c9030042e360 Ca 2 "H-0002c9030042e360" # "compute-0-14 HCA-1" [1](2c9030042e361) "S-0002c903008fd500"[24] # lid 24 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e383 caguid=0x2c9030042e380 Ca 2 "H-0002c9030042e380" # "compute-0-13 HCA-1" [1](2c9030042e381) "S-0002c903008fd500"[23] # lid 28 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e453 caguid=0x2c9030042e450 Ca 2 "H-0002c9030042e450" # "compute-0-6 HCA-1" [1](2c9030042e451) "S-0002c903008fd500"[22] # lid 22 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e393 caguid=0x2c9030042e390 Ca 2 "H-0002c9030042e390" # "compute-0-7 HCA-1" [1](2c9030042e391) "S-0002c903008fd500"[21] # lid 32 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e303 caguid=0x2c9030042e300 Ca 2 "H-0002c9030042e300" # "compute-0-10 HCA-1" [1](2c9030042e301) "S-0002c903008fd500"[20] # lid 6 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e443 caguid=0x2c9030042e440 Ca 2 "H-0002c9030042e440" # "compute-0-11 HCA-1" [1](2c9030042e441) "S-0002c903008fd500"[19] # lid 19 lmc 0 "SwitchX - Mellanox Technologies" lid 4 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e2d3 caguid=0x2c9030042e2d0 Ca 2 "H-0002c9030042e2d0" # "compute-0-4 HCA-1" [1](2c9030042e2d1) "S-0002c9030075c870"[32] # lid 35 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e413 caguid=0x2c9030042e410 Ca 2 "H-0002c9030042e410" # "compute-0-5 HCA-1" [1](2c9030042e411) "S-0002c9030075c870"[31] # lid 11 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e2f3 caguid=0x2c9030042e2f0 Ca 2 "H-0002c9030042e2f0" # "compute-0-12 HCA-1" [1](2c9030042e2f1) "S-0002c9030075c870"[30] # lid 41 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e473 caguid=0x2c9030042e470 Ca 2 "H-0002c9030042e470" # "compute-0-16 HCA-1" [1](2c9030042e471) "S-0002c9030075c870"[29] # lid 26 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e513 caguid=0x2c9030042e510 Ca 2 "H-0002c9030042e510" # "compute-0-21 HCA-1" [1](2c9030042e511) "S-0002c9030075c870"[28] # lid 12 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e433 caguid=0x2c9030042e430 Ca 2 "H-0002c9030042e430" # "compute-0-15 HCA-1" [1](2c9030042e431) "S-0002c9030075c870"[26] # lid 18 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e403 caguid=0x2c9030042e400 Ca 2 "H-0002c9030042e400" # "compute-0-1 HCA-1" [1](2c9030042e401) "S-0002c9030075c870"[25] # lid 7 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e3f3 caguid=0x2c9030042e3f0 Ca 2 "H-0002c9030042e3f0" # "compute-0-22 HCA-1" [1](2c9030042e3f1) "S-0002c9030075c870"[24] # lid 42 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e523 caguid=0x2c9030042e520 Ca 2 "H-0002c9030042e520" # "compute-0-18 HCA-1" [1](2c9030042e521) "S-0002c9030075c870"[23] # lid 15 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e2e3 caguid=0x2c9030042e2e0 Ca 2 "H-0002c9030042e2e0" # "compute-0-19 HCA-1" [1](2c9030042e2e1) "S-0002c9030075c870"[22] # lid 37 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e4f3 caguid=0x2c9030042e4f0 Ca 2 "H-0002c9030042e4f0" # "compute-0-3 HCA-1" [1](2c9030042e4f1) "S-0002c9030075c870"[21] # lid 43 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e313 caguid=0x2c9030042e310 Ca 2 "H-0002c9030042e310" # "compute-0-17 HCA-1" [1](2c9030042e311) "S-0002c9030075c870"[20] # lid 10 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e253 caguid=0x2c9030042e250 Ca 2 "H-0002c9030042e250" # "compute-0-20 HCA-1" [1](2c9030042e251) "S-0002c9030075c870"[19] # lid 21 lmc 0 "SwitchX - Mellanox Technologies" lid 25 4xQDR vendid=0x2c9 devid=0x1003 sysimgguid=0x2c9030042e423 caguid=0x2c9030042e420 Ca 2 "H-0002c9030042e420" # "nalanda HCA-1" [1](2c9030042e421) "S-0002c9030075c870"[27] [root@nalanda mvapich2-1.9]# ibhosts Ca : 0x0002c9030041f840 ports 2 "compute-0-33 HCA-1" Ca : 0x0002c9030041f7f0 ports 2 "compute-0-34 HCA-1" Ca : 0x24be05ffff85cc60 ports 2 "compute-0-25 HCA-1" Ca : 0x24be05ffff856d00 ports 2 "compute-0-27 HCA-1" Ca : 0x0002c9030041f800 ports 2 "compute-0-31 HCA-1" Ca : 0x24be05ffff85cc30 ports 2 "compute-0-30 HCA-1" Ca : 0x24be05ffff85bcf0 ports 2 "compute-0-26 HCA-1" Ca : 0x24be05ffff85cc20 ports 2 "compute-0-28 HCA-1" Ca : 0x0002c9030052498e ports 2 "oss1 mlx4_0" Ca : 0x0002c9030052499a ports 2 "mds1 mlx4_0" Ca : 0x0002c90300524992 ports 2 "mds2 mlx4_0" Ca : 0x0002c90300524982 ports 2 "oss2 mlx4_0" Ca : 0x24be05ffff85bce0 ports 2 "compute-0-24 HCA-1" Ca : 0x24be05ffff85bc90 ports 2 "compute-0-23 HCA-1" Ca : 0x0002c90300423a10 ports 2 "compute-0-32 HCA-1" Ca : 0x24be05ffff856df0 ports 2 "compute-0-29 HCA-1" Ca : 0x0002c9030042e330 ports 2 "compute-0-2 HCA-1" Ca : 0x0002c9030042e3e0 ports 2 "compute-0-8 HCA-1" Ca : 0x0002c9030042e320 ports 2 "compute-0-9 HCA-1" Ca : 0x0002c9030042e360 ports 2 "compute-0-14 HCA-1" Ca : 0x0002c9030042e380 ports 2 "compute-0-13 HCA-1" Ca : 0x0002c9030042e450 ports 2 "compute-0-6 HCA-1" Ca : 0x0002c9030042e390 ports 2 "compute-0-7 HCA-1" Ca : 0x0002c9030042e300 ports 2 "compute-0-10 HCA-1" Ca : 0x0002c9030042e440 ports 2 "compute-0-11 HCA-1" Ca : 0x0002c9030042e2d0 ports 2 "compute-0-4 HCA-1" Ca : 0x0002c9030042e410 ports 2 "compute-0-5 HCA-1" Ca : 0x0002c9030042e2f0 ports 2 "compute-0-12 HCA-1" Ca : 0x0002c9030042e470 ports 2 "compute-0-16 HCA-1" Ca : 0x0002c9030042e510 ports 2 "compute-0-21 HCA-1" Ca : 0x0002c9030042e430 ports 2 "compute-0-15 HCA-1" Ca : 0x0002c9030042e400 ports 2 "compute-0-1 HCA-1" Ca : 0x0002c9030042e3f0 ports 2 "compute-0-22 HCA-1" Ca : 0x0002c9030042e520 ports 2 "compute-0-18 HCA-1" Ca : 0x0002c9030042e2e0 ports 2 "compute-0-19 HCA-1" Ca : 0x0002c9030042e4f0 ports 2 "compute-0-3 HCA-1" Ca : 0x0002c9030042e310 ports 2 "compute-0-17 HCA-1" Ca : 0x0002c9030042e250 ports 2 "compute-0-20 HCA-1" Ca : 0x0002c9030042e420 ports 2 "nalanda HCA-1" [root@nalanda mvapich2-1.9]# [root@nalanda mvapich2-1.9]# ibswitches Switch : 0x0002c903008fd500 ports 36 "SwitchX - Mellanox Technologies" base port 0 lid 4 lmc 0 Switch : 0x0002c90300903600 ports 36 "SwitchX - Mellanox Technologies" base port 0 lid 3 lmc 0 Switch : 0x0002c90300902c00 ports 36 "SwitchX - Mellanox Technologies" base port 0 lid 2 lmc 0 Switch : 0x0002c90300901480 ports 36 "SwitchX - Mellanox Technologies" base port 0 lid 27 lmc 0 Switch : 0x0002c9030075c870 ports 36 "SwitchX - Mellanox Technologies" base port 0 lid 25 lmc 0 [root@nalanda mvapich2-1.9]#
_______________________________________________ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/mailman/listinfo/ewg
