On Thu, Oct 27, 2011 at 7:48 AM, Hiroyuki Sato <hiroys...@gmail.com> wrote:
> Hello Hal > > Thank you for your information. > > I read this documents. PP 45. > > http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf > > Decide on the PKey to be used in the subnet. > Valid values are 0-255. > Valid values for pkey are 1-32767 (1-0x7ffff). > The actual PKey used is a 16-bit number with > the most significant bit set. > For example, a value of 0 will give a > PKey with the value 0x8000. > > Is this wrong?? > Pkey 0/0x8000 is invalid so this is wrong. What it's showing about turning on the full membership bit in the pkey is correct. A better example would be showing a value of 1 being transformed into 0x8001 pkey. -- Hal > > > 2011/10/27 Hal Rosenstock <hal.rosenst...@gmail.com>: > > > > > > On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <hiroys...@gmail.com> > wrote: > >> > >> Hello Hal and Richard. > >> > >> I found what is the problem. > >> > >> Q: Can I create subinterface with 0 (0x8000)??? > > > > > > That uses an invalid pkey so this is not a valid configuration. > > > > -- Hal > > > >> > >> My Test result. > >> > >> partitions.conf > >> > >> Default=0x7fff, : ALL=full ; > >> Net0=0x0000, ipoib : ALL=full ; > >> Net1=0x0001, ipoib : ALL=full ; > >> Net2=0x0002, ipoib : ALL=full ; > >> > >> 1) Server1 ib.8000 -> Server1 ib.8000 > >> Ping NG > >> > >> 2) Server1 ib.8001 -> Server1 ib.8001 > >> > >> The differece is subinterfae number. > >> > >> Yesterday, I always tested subinterface 0x8000. > >> > >> > >> > >> And about previous post, > >> > >> The partitions.conf which I posted few hous ago, > >> was semi colon missing. > >> > >> > * partitions.conf is the following > >> > Default=0x7fff, ipoib : ALL=full > >> > Net0=0x0001, ipoib : ALL=full > >> > >> Should be > >> > >> Default=0x7fff, ipoib : ALL=full ; > >> Net0=0x0001, ipoib : ALL=full ; > >> > >> Thank you again. > >> > >> > >> 2011/10/27 Hiroyuki Sato <hiroys...@gmail.com>: > >> > Hello Hal > >> > > >> > Thank you for your information. > >> > > >> > I simplified my test environment > >> > The environment is the folloing. > >> > > >> > I'm not sure what is wrong Test2 (Partition test) > >> > And Could you please tell me how to check the problem?? > >> > (tool, logfile, etc.. ) > >> > > >> > * Environment > >> > > >> > OS: Scientific Linux6.1 > >> > OFED: 1.5.3. > >> > > >> > * Diagram > >> > > >> > +----------+ib0 ib0+----------+ > >> > | Server1 |------------------| Server2 | > >> > +----------+ +----------+ > >> > > >> > > >> > > >> > > >> > 1) Test1 (Simple IPoIB no partition Ping test) > >> > > >> > Server1 > >> > > >> > a) /sbin/ifconfig ib0 inet 192.168.1.1/24 > >> > b) /sbn/service opensmd start > >> > 3) /sbin/service opensmd start > >> > * no /etc/opensm/partitions.conf > >> > > >> > Server2 > >> > /sbin/ifconfig ib0 inet 192.168.1.1/24 > >> > > >> > Test from server1(192.168.1.1) to server2(192.168.1.2) > >> > ping 192.168.1.2 > >> > The result was OK. > >> > > >> > 2) Test2 (Add partition by hand and Ping test) > >> > > >> > reboot Server 1 and Server 2 > >> > > >> > Server 1 > >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child > >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24 > >> > 3) /sbin/service opensmd start > >> > > >> > * partitions.conf is the following > >> > Default=0x7fff, ipoib : ALL=full > >> > Net0=0x0001, ipoib : ALL=full > >> > > >> > Server2 > >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child > >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24 > >> > > >> > Test from server1(ib0.8001/192.168.0.1) to > >> > server2(ib0.8001/192.168.0.2) > >> > > >> > ping 192.168.0.2 > >> > no response. > >> > > >> > 3) Test2 Log > >> > > >> > I enabled debug_log parameter in ib_ipoib modules. > >> > > >> > cd /sys/module/ib_ipoib/parameters > >> > # cat debug_level > >> > 1 > >> > > >> > Could you please tell me what is wrong?? > >> > > >> > ib0: enabling connected mode will cause multicast packet drops > >> > ib0: mtu > 2044 will cause multicast packet drops. > >> > ib0: mtu > 2044 will cause multicast packet drops. > >> > ib1: enabling connected mode will cause multicast packet drops > >> > ib1: mtu > 2044 will cause multicast packet drops. > >> > ib1: mtu > 2044 will cause multicast packet drops. > >> > ib0: Event 17 on device mthca0 port 1 > >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. > >> > ib0: Event 11 on device mthca0 port 1 > >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. > >> > ib0: Event 9 on device mthca0 port 1 > >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. > >> > ib0.8001: max_srq_sge=27 > >> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16 > >> > ib0.8001: bringing up interface > >> > ib0.8001: IPOIB_FLAG_OPER_UP not set > >> > ib0.8001: IPOIB_FLAG_OPER_UP not set > >> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready > >> > ib0.8001: IPOIB_FLAG_OPER_UP not set > >> > > >> > regards. > >> > > >> > 2011/10/27 Hal Rosenstock <hal.rosenst...@gmail.com>: > >> >> > >> >> > >> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroys...@gmail.com > > > >> >> wrote: > >> >>> > >> >>> Dear members. > >> >>> > >> >>> I have some question about Infiniband Partitions. > >> >>> > >> >>> I would like to build Linux box with IPoIB Router. > >> >>> > >> >>> * Questions > >> >>> > >> >>> My Question is the following. > >> >>> > >> >>> a) Can I create the following IPoIB network ?? > >> >>> > >> >>> b) If so, how to setup /etc/opensm/partitions.conf?? > >> >>> > >> >>> * Problem. > >> >>> > >> >>> Ping IB Client to Router : I got the followig error. > >> >>> > >> >>> Network is unreachable. > >> >>> > >> >>> * Environment > >> >>> > >> >>> OFED: OFED-1.5.3.2 > >> >>> OS: Scientific Linux 6.1 > >> >>> > >> >>> * Linux box (Router) > >> >>> > >> >>> 1 ethernet interface (eth0) > >> >>> 1 Infiniband interfae (ib0) > >> >>> Run subnet manager. > >> >>> > >> >>> * Ethernet setup > >> >>> > >> >>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24) > >> >>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24) > >> >>> > >> >>> * Infiniband Setup > >> >>> a) add subinterface 0 ib0.8000 (192.168.0.0/24) > >> >>> b) add subinterface 1 ib0.8001 (192.168.1.0/24) > >> >>> c) Run opensm > >> >>> > >> >>> > >> >>> I crated subinterface with the following command. > >> >>> > >> >>> echo 0 > /sys/class/net/ib0/create_child > >> >>> echo 1 > /sys/class/net/ib0/create_child > >> >> > >> >> > >> >> echo 0x8001 > /sys/class/net/ib0/create_child > >> >> echo 0x8002 > /sys/class/net/ib0/create_child > >> >> > >> >>> > >> >>> and assign ip address with ifconfig > >> >>> > >> >>> * Logical Diagram > >> >>> > >> >>> +----------- 192.168.120.0/24 --- > >> >>> | > >> >>> | +---- 192.168.130.0/24 > >> >>> eth0.120| | eth0.130 > >> >>> +----------+ > >> >>> | Router | > >> >>> +----------+ > >> >>> ib0.8000| |ib0.8001 > >> >>> 192.168.0.1 | +---- 192.168.1.0/24 > >> >>> | > >> >>> +----------- 192.168.0.0/24 --- IB Cllient. > (ib0.8000) > >> >>> (192.168.0.2) > >> >>> > >> >>> * /etc/opensm/partitions.conf > >> >>> > >> >>> Default=0x7fff : ALL=full > >> >>> Net0=0x00, ipoib : ALL=full > >> >>> Net1=0x01, ipoib : ALL=full > >> >> > >> >> > >> >> Default=0x7fff: ALL=full > >> >> Net0=0x0001, ipoib: ALL=full > >> >> Net1=0x0002, ipoib: ALL=full > >> >> > >> >> > >> >>> > >> >>> Coule you please tell me what is wrong?? > >> >> > >> >> > >> >> The one thing in the above I'm not sure of is whether the default > >> >> partition > >> >> also needs the ipoib flag above so you might need to add that too if > >> >> the > >> >> above doesn't work. > >> >> > >> >> -- Hal > >> >> > >> >>> > >> >>> Sincerely. > >> >>> > >> >>> -- > >> >>> Hiroyuki Sato. > >> >>> _______________________________________________ > >> >>> ewg mailing list > >> >>> ewg@lists.openfabrics.org > >> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > >> >> > >> >> > >> > > >> > > >> > > >> > -- > >> > Hiroyuki Sato > >> > > >> > >> > >> > >> -- > >> Hiroyuki Sato > > > > > > > > -- > Hiroyuki Sato >
_______________________________________________ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
