What: Change the descriptor completion callback function to include cbdata and context pointers.
Old callback:
typedef void (*mca_btl_base_completion_fn_t)(
struct mca_btl_base_module_t* module,
struct mca_btl_base_endpoint_t* endpoint,
struct mca_btl_base_descriptor_t* descriptor,
int status);
New callback:
typedef void (*mca_btl_base_completion_fn_t)(
struct mca_btl_base_module_t* module,
struct mca_btl_base_endpoint_t* endpoint,
struct mca_btl_base_descriptor_t* descriptor,
void *cbdata, void *context, int status);
Why: The BTL interface provides support for using a single descriptor
with multiple concurrent RDMA operations. BTLs support this feature if
the following flag is not set:
/** RDMA put/get calls must have a matching prepare_{src,dst} call
on the target with the same base (and possibly bound). */
#define MCA_BTL_FLAGS_RDMA_MATCHED 0x0040
The problem is that in order to pass back the correct callback data and
context to the completion function BTLs need to modify the
descriptor. This could be a disaster in a multi-threaded application if
one thread is calling the completion callback while another thread is
preparing to start a put/get operation. To avoid issues it is better to
provide the callback data as seperate arguments.
The change is straightforward and the commit will update all BTLs and
BTL users to use the new completion callback signature.
When: As this was discussed at the developer's meeting last month I am
setting a short timeout for this RFC. This times out next Wed (July
16).
I would really like feedback on this change. Can it be improved? Should
the segment data be passed back to the function (not something I need
for RMA but might be useful elsewhere)? Would it be better to remove the
simultaneous RDMA feature in favor of a lightweight descriptor clone (I
have this implemented as well and I have no problem with providing
both features)?
This is another is a series of RFCs to improve (I hope) the BTL
interface for one-sided operations. The next RFC will be on the
one-sided BTL interface.
-Nathan Hjelm
HPC-5, LANL
pgptPwvMIBf_u.pgp
Description: PGP signature
