Hal, On Fri, 2005-03-04 at 12:53, Hal Rosenstock wrote: > Hi again Nitin, > > Finally got a chance to work on this. I have a workaround for you for > now. Real patch later... Let me know if this does the trick for you. It > did for me. > > -- Hal > > Index: osm_sa_mcmember_record.c > =================================================================== > --- osm_sa_mcmember_record.c (revision 1953) > +++ osm_sa_mcmember_record.c (working copy) > @@ -1522,9 +1522,11 @@ > if ((IB_MCR_COMPMASK_PROXY & comp_mask) && > (p_rcvd_rec->proxy_join != p_mgrp->mcmember_rec.proxy_join)) goto Exit; > > +#if 0 > /* if defined MUST match exactly !*/ > if ((IB_MCR_COMPMASK_MTU_SEL & comp_mask) && > ((p_rcvd_rec->mtu >> 6) != (p_mgrp->mcmember_rec.mtu >> 6))) goto Exit; > +#endif > > if ((IB_MCR_COMPMASK_MTU & comp_mask) && > ((p_rcvd_rec->mtu & 0x3F) != (p_mgrp->mcmember_rec.mtu & 0x3F))) goto > Exit; This is cool, I have got Solaris IPoIB happily working with the OpenSM now. It plumbs, pings and snoops on 0xffff pkey. Here is some output:
[EMAIL PROTECTED] ~]# cat /etc/path_to_inst | grep ibd "/[EMAIL PROTECTED],600000/[EMAIL PROTECTED]/pci15b3,[EMAIL PROTECTED]/[EMAIL PROTECTED],ffff,ipib" 0 "ibd" "/[EMAIL PROTECTED],600000/[EMAIL PROTECTED]/pci15b3,[EMAIL PROTECTED]/[EMAIL PROTECTED],ffff,ipib" 1 "ibd" [EMAIL PROTECTED] ~]# ifconfig ibd0 ibd0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 3 inet 192.168.100.111 netmask ffffff00 broadcast 192.168.100.255 ipib 0:0:0:16:fe:80:0:0:0:0:0:0:0:2:c9:1:9:76:51:d1 [EMAIL PROTECTED] ~]# ping 192.168.100.112 192.168.100.112 is alive [EMAIL PROTECTED] ~]# snoop -d ibd1 192.168.100.112 -> * ARP C Who is 192.168.100.111, 192.168.100.111 ? 192.168.100.111 -> 192.168.100.112 ARP R 192.168.100.111, 192.168.100.111 is 0:0:0:16:fe:80:0:0:0:0:0:0:0:2:c9:1:9:76:51:d1 192.168.100.111 -> 192.168.100.112 ICMP Echo request (ID: 641 Sequence number: 0) 192.168.100.112 -> 192.168.100.111 ICMP Echo reply (ID: 641 Sequence number: 0) This is fantastic. Thanks Hal !.. BTW, I have not tested it with multiple GetTable reponse - RMPP packet. On other hand, on my linux node, if I try to use 8001 partition and configure IB interface with IP addr (same time while ib0 is using 0xffff pkey), I get the following error, you may want to investigate that.... [EMAIL PROTECTED] ~]# echo 0x8001 > /sys/class/net/ib0/create_child [EMAIL PROTECTED] ~]# ifconfig ib0.8001 10.10.1.1 [EMAIL PROTECTED]: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 ~]# ib0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 [EMAIL PROTECTED] ~]# ib0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 b0.8001: multicast join failed for ff12:401b:8001:0:0:0:ffff:ffff, status -22 Thanks Nitin > > > -----Forwarded Message----- > > From: Hal Rosenstock <[EMAIL PROTECTED]> > To: Nitin Hande <[EMAIL PROTECTED]> > Cc: openib <openib-general@openib.org>, Tom Duffy <[EMAIL PROTECTED]> > Subject: Re: [openib-general] Solaris IPoIB MTU with OpenSM > Date: 24 Feb 2005 08:42:23 -0500 > > Hi Nitin, > > On Wed, 2005-02-23 at 17:19, Nitin Hande wrote: > > Hal, > > > > [comments below] > > On Wed, 2005-02-23 at 02:19, Hal Rosenstock wrote: > > > On Tue, 2005-02-22 at 22:56, Nitin Hande wrote: > > > > So I tried the latest patches and preliminarily things seem to be > > > > working fine. > > > > > > Yipee. > > [snip..] > > > > > > > > > > > So after this test above, I try to run snoop on the solaris interface > > > > and get the following error message from the layer below IPoIB: > > > > > > > > Feb 22 19:50:25 dongon.SFBay.Sun.COM ibd: [ID 517869 kern.info] NOTICE: > > > > ibd0: HCA GUID 0002c901097651d0 port 1 PKEY ffff Could not get list of > > > > IBA multicast groups > > > > > > > > My preliminary assumption is that OpenSm is not returning the list of > > > > multicast groups that the ibd interface has joined. I will look at the > > > > MAD's tomorrow and try to ascertain that. > > > > > > How does S10 request this ? Remember that if it is a GetTable and > > > doesn't fit in a single MAD, it will be broken now. If that is the case, > > > we will live with this until we have real RMPP. > > Below is an an example of a single GetTable request and response between > > Solaris and OpenSM. OpenSM is not reporting the MCgroups in case of a > > single request/response. I have also provided a MAD output between > > Solaris IPoIB driver and IBSRM single GetTable request response below > > this example. > > > > Here is the MAD trace between solaris and OpenSM: > > Outgoing MAD: > > BaseVersion: 0x1 > > MgmtClass: 0x3 - SubnAdm > > ClassVersion: 0x2 > > R_Method: 0x12 - SubnAdmGetTable() > > Status: 0x0 - NO_ERROR > > ClassSpecific: 0x0 > > TransactionID: 0x97651d1000000ec > > AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID > > > > 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef > > 0: 01 03 02 12 00 00 00 00 09 76 51 d1 00 00 00 ec .........vQ..... > > 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 .8.............. > > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00 ................ > > 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 50: 00 00 00 00 00 00 00 00 00 00 0b 1b 00 00 84 00 ................ > > 60: ff ff 00 00 00 00 00 00 20 00 00 00 00 00 00 00 ........ ....... > > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > Incoming MAD: > > BaseVersion: 0x1 > > MgmtClass: 0x3 - SubnAdm > > ClassVersion: 0x2 > > R_Method: 0x92 - > > Status: 0x0 - NO_ERROR > > ClassSpecific: 0x0 > > TransactionID: 0x97651d1000000ec > > AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID > > > > 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef > > 0: 01 03 02 92 00 00 00 00 09 76 51 d1 00 00 00 ec .........vQ..... > > 10: 00 38 00 00 ff ff ff ff 01 01 77 00 00 00 00 01 .8........w..... > > 20: 00 00 00 14 00 00 00 00 00 00 00 00 00 07 00 00 ................ > > 30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00 ................ > > 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > It is likely failing the component checking in > osm_sa_mcmember_record.c::__osm_sa_mcm_by_comp_mask_cb due to an endian > issue. Either you can debug this code or I will early next week. > > The component mask in the request is 0x80b4 so the only components > checked are QKey (0xb1b), MTU (exactly 2048 (4)), PKey (0xffff), and > scope (2). > > If I don't hear anything by next week, I will work on this then. > > Thanks. > > -- Hal > > > Here is the transaction between IBSRM and Solaris IPoIB driver. > > > > Outgoing MAD: > > BaseVersion: 0x1 > > MgmtClass: 0x3 - SubnAdm > > ClassVersion: 0x2 > > R_Method: 0x12 - SubnAdmGetTable() > > Status: 0x0 - NO_ERROR > > ClassSpecific: 0x0 > > TransactionID: 0x8fecc610000009a > > AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID > > > > 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef > > 0: 01 03 02 12 00 00 00 00 08 fe cc 61 00 00 00 9a ...........a.... > > 10: 00 38 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 .8.............. > > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 30: 00 00 00 00 00 00 80 b4 00 00 00 00 00 00 00 00 ................ > > 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 50: 00 00 00 00 00 00 00 00 81 23 45 68 00 00 84 00 .........#Eh.... > > 60: 80 01 00 00 00 00 00 00 20 00 00 00 00 00 00 00 ........ ....... > > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > Incoming MAD: > > BaseVersion: 0x1 > > MgmtClass: 0x3 - SubnAdm > > ClassVersion: 0x2 > > R_Method: 0x92 - > > Status: 0x0 - NO_ERROR > > ClassSpecific: 0x0 > > TransactionID: 0x8fecc610000009a > > AttributeID: 0x38 - SA_MCMEMBERRECORD_ATTRID > > > > 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef > > 0: 01 03 02 92 00 00 00 00 08 fe cc 61 00 00 00 9a ...........a.... > > 10: 00 38 00 00 00 00 00 00 01 01 73 00 00 00 00 01 .8........s..... > > 20: 00 00 01 40 00 00 00 00 00 00 00 00 00 07 00 00 [EMAIL PROTECTED] > > 30: 00 00 00 00 00 00 00 00 ff 12 40 1b 80 01 00 00 [EMAIL PROTECTED] > > 40: 00 00 00 00 00 00 00 09 00 00 00 00 00 00 00 00 ................ > > 50: 00 00 00 00 00 00 00 00 81 23 45 68 c0 04 84 00 .........#Eh.... > > 60: 80 01 83 8d 00 00 00 00 20 00 00 00 00 00 00 00 ........ ....... > > 70: ff 12 40 1b 80 01 00 00 00 00 00 00 00 00 00 01 [EMAIL PROTECTED] > > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > 90: 81 23 45 68 c0 03 84 00 80 01 83 8d 00 00 00 00 .#Eh............ > > a0: 20 00 00 00 00 00 00 00 ff 12 40 1b 80 01 00 00 [EMAIL PROTECTED] > > b0: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 ................ > > c0: 00 00 00 00 00 00 00 00 81 23 45 68 c0 00 84 00 .........#Eh.... > > d0: 80 01 83 8d 00 00 00 00 20 00 00 00 00 00 00 00 ........ ....... > > e0: ff 12 60 1b 80 01 00 00 00 00 00 01 ff 76 5b 01 ..`..........v[. > > f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > > > Thanks > > Nitin > > > _______________________________________________ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general