Re: [OMPI devel] [ofa-general] uDAPL EVD queue length issue

[Arlin Davis]

Jon Mason wrote:

While working on OMPI udapl btl, I have noticed some "interesting"
behavior.  OFA udapl wants the evd queues to be a power of 2 and
then will subtract 1 for book keeping (ie, so that internal head and
tail pointers never touch except when the ring is empty).  OFA udapl
will report the queue length as this number (and not the original
size requested) when queried.  This becomes interesting when a power
of 2 is passed in and then queried.  For example, a requested queue
of length 256 will report a length of 255 when queried.  


Something is not right. You should ALWAYS get at least what you request. 
On my system with an mthca, a request of 256 gets you 511. It is the 
verbs provider that is rounding up, not uDAPL.


Here is my uDAPL debug output (DAPL_DBG_TYPE=0x) using dtest:

 cq_object_create: (0x519bb0,0x519d00)
dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=511

This is before and after the ibv_create_cq call. uDAPL builds it's EVD 
resources based on what is returned from this call.


I modified dtest to double check the dat_evd_query and I get the same:

8962 dto_rcv_evd created 0x519e80
8962 dto_req_evd QLEN - requested 256 and actual 511

What OFED release and device are you using?

-arlin

Re: [OMPI devel] [ofa-general] uDAPL EVD queue length issue

[Arlin Davis]

I'm running OFED 1.2.5 and using Chelsio.

 From the linux rdma verbs perspective, ibv_create_cq() will create a cq 
that is >= the requested depth.  The fact that mthca always bumps the 
size up to the next power of 2 isn't something udapl can rely on.


It doesn't.

uDAPL passes the users requested qlen directly to the verbs 
ibv_create_cq call (dapl_openib_cq.c) and then uses the returned qlen to 
allocate EVD's (dapl_evd_util.c) and a ring buffer 
(dapl_ring_buffer_util.c) for managing the free and pending events.


The EVD's are allocated based on returned qlen from verbs and the 
ring_buffer is allocated based on qlen, next power of 2 minus 1. Unless 
I am missing something, I don't see how we can get less then what is 
requested.




Here's the crux:  If creating a udapl evd of 256 results in a cq of 256 
and the udapl returns a evd with size 255, then udapl is broken...


Yes, I agree. But I don't see how this is happening.

Here is output from my dtest when requesting 256 and verbs returning the 
same qlen. You can see before and after verbs we get 256, the rbufs are 
511, and the query EVD call returns 256 to the user.


 cq_object_create: (0x519bb0,0x519d00)
dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=256
 setup_async_cb: ia 0x518270 type 2 hdl 0x519bb0 cb 0x2a957c8e70 ctx 
0x519bb0

 >>> rbuf_alloc: size 256 rsize 511
 >>> rbuf_alloc: size 256 rsize 511
dapl_evd_create () returns 0x0

9920 dto_req_evd QLEN - requested 256 and actual 256
9920 Create events done

Can you turn up DAPL debug(DAPL_DBG_TYPE=0x) so I can see what is 
happening?


Thanks,

-arlin