Hello Hal Thank you for your information.
I read this documents. PP 45. http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf Decide on the PKey to be used in the subnet. Valid values are 0-255. The actual PKey used is a 16-bit number with the most significant bit set. For example, a value of 0 will give a PKey with the value 0x8000. Is this wrong?? 2011/10/27 Hal Rosenstock <hal.rosenst...@gmail.com>: > > > On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <hiroys...@gmail.com> wrote: >> >> Hello Hal and Richard. >> >> I found what is the problem. >> >> Q: Can I create subinterface with 0 (0x8000)??? > > > That uses an invalid pkey so this is not a valid configuration. > > -- Hal > >> >> My Test result. >> >> partitions.conf >> >> Default=0x7fff, : ALL=full ; >> Net0=0x0000, ipoib : ALL=full ; >> Net1=0x0001, ipoib : ALL=full ; >> Net2=0x0002, ipoib : ALL=full ; >> >> 1) Server1 ib.8000 -> Server1 ib.8000 >> Ping NG >> >> 2) Server1 ib.8001 -> Server1 ib.8001 >> >> The differece is subinterfae number. >> >> Yesterday, I always tested subinterface 0x8000. >> >> >> >> And about previous post, >> >> The partitions.conf which I posted few hous ago, >> was semi colon missing. >> >> > * partitions.conf is the following >> > Default=0x7fff, ipoib : ALL=full >> > Net0=0x0001, ipoib : ALL=full >> >> Should be >> >> Default=0x7fff, ipoib : ALL=full ; >> Net0=0x0001, ipoib : ALL=full ; >> >> Thank you again. >> >> >> 2011/10/27 Hiroyuki Sato <hiroys...@gmail.com>: >> > Hello Hal >> > >> > Thank you for your information. >> > >> > I simplified my test environment >> > The environment is the folloing. >> > >> > I'm not sure what is wrong Test2 (Partition test) >> > And Could you please tell me how to check the problem?? >> > (tool, logfile, etc.. ) >> > >> > * Environment >> > >> > OS: Scientific Linux6.1 >> > OFED: 1.5.3. >> > >> > * Diagram >> > >> > +----------+ib0 ib0+----------+ >> > | Server1 |------------------| Server2 | >> > +----------+ +----------+ >> > >> > >> > >> > >> > 1) Test1 (Simple IPoIB no partition Ping test) >> > >> > Server1 >> > >> > a) /sbin/ifconfig ib0 inet 192.168.1.1/24 >> > b) /sbn/service opensmd start >> > 3) /sbin/service opensmd start >> > * no /etc/opensm/partitions.conf >> > >> > Server2 >> > /sbin/ifconfig ib0 inet 192.168.1.1/24 >> > >> > Test from server1(192.168.1.1) to server2(192.168.1.2) >> > ping 192.168.1.2 >> > The result was OK. >> > >> > 2) Test2 (Add partition by hand and Ping test) >> > >> > reboot Server 1 and Server 2 >> > >> > Server 1 >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24 >> > 3) /sbin/service opensmd start >> > >> > * partitions.conf is the following >> > Default=0x7fff, ipoib : ALL=full >> > Net0=0x0001, ipoib : ALL=full >> > >> > Server2 >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24 >> > >> > Test from server1(ib0.8001/192.168.0.1) to >> > server2(ib0.8001/192.168.0.2) >> > >> > ping 192.168.0.2 >> > no response. >> > >> > 3) Test2 Log >> > >> > I enabled debug_log parameter in ib_ipoib modules. >> > >> > cd /sys/module/ib_ipoib/parameters >> > # cat debug_level >> > 1 >> > >> > Could you please tell me what is wrong?? >> > >> > ib0: enabling connected mode will cause multicast packet drops >> > ib0: mtu > 2044 will cause multicast packet drops. >> > ib0: mtu > 2044 will cause multicast packet drops. >> > ib1: enabling connected mode will cause multicast packet drops >> > ib1: mtu > 2044 will cause multicast packet drops. >> > ib1: mtu > 2044 will cause multicast packet drops. >> > ib0: Event 17 on device mthca0 port 1 >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. >> > ib0: Event 11 on device mthca0 port 1 >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. >> > ib0: Event 9 on device mthca0 port 1 >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. >> > ib0.8001: max_srq_sge=27 >> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16 >> > ib0.8001: bringing up interface >> > ib0.8001: IPOIB_FLAG_OPER_UP not set >> > ib0.8001: IPOIB_FLAG_OPER_UP not set >> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready >> > ib0.8001: IPOIB_FLAG_OPER_UP not set >> > >> > regards. >> > >> > 2011/10/27 Hal Rosenstock <hal.rosenst...@gmail.com>: >> >> >> >> >> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroys...@gmail.com> >> >> wrote: >> >>> >> >>> Dear members. >> >>> >> >>> I have some question about Infiniband Partitions. >> >>> >> >>> I would like to build Linux box with IPoIB Router. >> >>> >> >>> * Questions >> >>> >> >>> My Question is the following. >> >>> >> >>> a) Can I create the following IPoIB network ?? >> >>> >> >>> b) If so, how to setup /etc/opensm/partitions.conf?? >> >>> >> >>> * Problem. >> >>> >> >>> Ping IB Client to Router : I got the followig error. >> >>> >> >>> Network is unreachable. >> >>> >> >>> * Environment >> >>> >> >>> OFED: OFED-1.5.3.2 >> >>> OS: Scientific Linux 6.1 >> >>> >> >>> * Linux box (Router) >> >>> >> >>> 1 ethernet interface (eth0) >> >>> 1 Infiniband interfae (ib0) >> >>> Run subnet manager. >> >>> >> >>> * Ethernet setup >> >>> >> >>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24) >> >>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24) >> >>> >> >>> * Infiniband Setup >> >>> a) add subinterface 0 ib0.8000 (192.168.0.0/24) >> >>> b) add subinterface 1 ib0.8001 (192.168.1.0/24) >> >>> c) Run opensm >> >>> >> >>> >> >>> I crated subinterface with the following command. >> >>> >> >>> echo 0 > /sys/class/net/ib0/create_child >> >>> echo 1 > /sys/class/net/ib0/create_child >> >> >> >> >> >> echo 0x8001 > /sys/class/net/ib0/create_child >> >> echo 0x8002 > /sys/class/net/ib0/create_child >> >> >> >>> >> >>> and assign ip address with ifconfig >> >>> >> >>> * Logical Diagram >> >>> >> >>> +----------- 192.168.120.0/24 --- >> >>> | >> >>> | +---- 192.168.130.0/24 >> >>> eth0.120| | eth0.130 >> >>> +----------+ >> >>> | Router | >> >>> +----------+ >> >>> ib0.8000| |ib0.8001 >> >>> 192.168.0.1 | +---- 192.168.1.0/24 >> >>> | >> >>> +----------- 192.168.0.0/24 --- IB Cllient. (ib0.8000) >> >>> (192.168.0.2) >> >>> >> >>> * /etc/opensm/partitions.conf >> >>> >> >>> Default=0x7fff : ALL=full >> >>> Net0=0x00, ipoib : ALL=full >> >>> Net1=0x01, ipoib : ALL=full >> >> >> >> >> >> Default=0x7fff: ALL=full >> >> Net0=0x0001, ipoib: ALL=full >> >> Net1=0x0002, ipoib: ALL=full >> >> >> >> >> >>> >> >>> Coule you please tell me what is wrong?? >> >> >> >> >> >> The one thing in the above I'm not sure of is whether the default >> >> partition >> >> also needs the ipoib flag above so you might need to add that too if >> >> the >> >> above doesn't work. >> >> >> >> -- Hal >> >> >> >>> >> >>> Sincerely. >> >>> >> >>> -- >> >>> Hiroyuki Sato. >> >>> _______________________________________________ >> >>> ewg mailing list >> >>> ewg@lists.openfabrics.org >> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg >> >> >> >> >> > >> > >> > >> > -- >> > Hiroyuki Sato >> > >> >> >> >> -- >> Hiroyuki Sato > > -- Hiroyuki Sato _______________________________________________ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg