[openib-general] different send and receive CQs
Hi, More dumb questions :) In my ULP (lustre networking) I maintain a common pool of send descriptors and per-connection receive descriptors. So it seems reasonable to have a single CQ for all sends and one CQ per-connection for receives. My code worked OK with a single CQ per connection, but when I try separate send and receive CQs, I don't see any send completions. Has anyone else tried separate send and receive CQs? -- Cheers, Eric ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] different send and receive CQs
Eric> My code worked OK with a single CQ per connection, but when Eric> I try separate send and receive CQs, I don't see any send Eric> completions. Eric> Has anyone else tried separate send and receive CQs? Yes, for example I know Shirley Ma is working on patches that change IPoIB to use separate CQs for sends and receives. Are you not seeing any completions when you poll the CQ, or are you not getting completion events? Some things to check would be that you are requesting notification on all the CQs you want events on, and that you are not posting unsignaled send requests. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] different send and receive CQs
Eric, I have no problem with splitting CQ, you can refer my IPoIB splitting CQ patch. Could you share your code here so we can give you some suggestions? Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] different send and receive CQs
Eric Barton wrote: In my ULP (lustre networking) I maintain a common pool of send descriptors and per-connection receive descriptors. So it seems reasonable to have a single CQ for all sends and one CQ per-connection for receives. Please note that since completions for each CQ make the HCA to generate an interrupt for which a SW handler needs some CPU to run on, multiple CQs scale upto the number of CPUs in the system. Beyond that your code will function quite bad. So its bad both for the client side (connecting to multiple OSTs and the server side connecting to $K-order clients). An easy solution to the issue of a single CQ having RX completions from multiple connections (QPs) is to have the structure pointed by the cookie carrying some TAG (pointer) relating it to the relevant connection. some approaches for multiple CQs are: + separate RX and TX CQs + have multiple (#CPUs) threads being waken up by the CQ interrupt handler, where each (takes a lock...,) poll a completion and handles it. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] different send and receive CQs
> Are you not seeing any completions when you poll the CQ, or are you > not getting completion events? Some things to check would be that you > are requesting notification on all the CQs you want events on, Doh! I must have been having a bit of a Homer moment... BTW, Or Gerlitz reckons there is a performance penalty for using multiple CQs. The reason I'm interested in separate CQs is to avoid CQ overflow as I add connections. On other stacks (Voltaire, Cisco, Silverstorm) I size a single CQ large enough for 'n' connections (i.e. cluster size - 1), but that means I have to refuse connections when 'n' have been established. In one stack it also stressed vmalloc() and prevented me from using a single whole-memory mapping. Is there a consensus? Cheers, Eric ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] different send and receive CQs
Eric Barton wrote: BTW, Or Gerlitz reckons there is a performance penalty for using multiple CQs. The reason I'm interested in separate CQs is to avoid CQ overflow as I add connections. On other stacks (Voltaire, Cisco, Silverstorm) I size a single CQ large enough for 'n' connections (i.e. cluster size - 1), but that means I have to refuse connections when 'n' have been established. Talking about CQ wrt adding connections here's my take: the max CQ size (reported by struct ib_device_attr->max_cqe of ib_query_device) is 128K (this is on memfull HCA, you would need to check the memfree HCA). So when the number of RX credits per connection is low it allows for many-K connections to use the same CQ (eg eight credits allow for 120K connections which is much more then the ~48K limit on LMC0 IB clusters size...). If you need more connections (QPs) than a single CQ can carry, create another one and attach it to new QPs. The CQ callback gets the CQ pointer as its first element, so you need not change you polling/arming logic. Also note that a 128K entries CQ consumes about 4MB (Roland can you confirm?) of the HCA attached memory (or host memory for memfree), so per my taste, coding apps for the cq_resize is kind of over doing. > In one stack it also stressed vmalloc() and prevented me from using a > single whole-memory mapping. Is there a chance that you are confusing CQs with QPs? Before implementing FMR scheme for the voltaire NAL, you were creating a giant QP for which the gen1 driver was allocating the host side memory using vmalloc, so it could not allocate more then ~300 QPs. With the mthca driver you should be able to allocate a CQ with the maximum allowed size (and if not it will be fixed...) Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
