> -----Original Message----- > From: Sagi Grimberg [mailto:sa...@dev.mellanox.co.il] > Sent: Tuesday, June 30, 2015 4:22 AM > To: Steve Wise; dledf...@redhat.com > Cc: r...@mellanox.com; sa...@mellanox.com; linux-rdma@vger.kernel.org; > jguntho...@obsidianresearch.com; infinip...@intel.com; > e...@mellanox.com; ogerl...@mellanox.com; sean.he...@intel.com > Subject: Re: [PATCH V2 3/5] RDMA/core: transport-independent access flags > > On 6/30/2015 12:36 AM, Steve Wise wrote: > > The semantics for MR access flags are not consistent across RDMA > > protocols. So rather than have applications try and glean what they > > need, have them pass in the intended roles and attributes for the MR to > > be allocated and let the RDMA core select the appropriate access flags > > given the roles, attributes, and device capabilities. > > > > We introduce rdma_mr_roles and rdma_mr_attributes that enumerate the > > possible roles and attributes for a MR. These are documented in the > > enums themselves. > > > > New services exported: > > > > rdma_device_access_flags() - given the intended MR roles and attributes > > passed in, return the ib_access_flags bits for the device. > > > > rdma_get_dma_mr() - allocate a dma mr using the applications intended > > MR roles and MR attributes. This uses rdma_device_access_flags(). > > > > rdma_fast_reg_access_flags() - return the ib_access_flags bits needed > > for a fast register WR given the applications intended MR roles and > > MR attributes. This uses rdma_device_access_flags(). > > > > Signed-off-by: Steve Wise <sw...@opengridcomputing.com> > > --- > > drivers/infiniband/core/verbs.c | 30 ++++++++++++ > > include/rdma/ib_verbs.h | 101 > > +++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 131 insertions(+), 0 deletions(-) > > > > diff --git a/drivers/infiniband/core/verbs.c > > b/drivers/infiniband/core/verbs.c > > index bac3fb4..2aa7c92 100644 > > --- a/drivers/infiniband/core/verbs.c > > +++ b/drivers/infiniband/core/verbs.c > > @@ -1144,6 +1144,36 @@ struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int > > mr_access_flags) > > } > > EXPORT_SYMBOL(ib_get_dma_mr); > > > > +int rdma_device_access_flags(struct ib_pd *pd, int roles, int attrs) > > +{ > > + int access_flags = attrs; > > + > > + if (roles & RDMA_MRR_RECV) > > + access_flags |= IB_ACCESS_LOCAL_WRITE; > > + > > + if (roles & RDMA_MRR_WRITE_DEST) > > + access_flags |= IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_WRITE; > > + > > + if (roles & RDMA_MRR_READ_DEST) { > > + access_flags |= IB_ACCESS_LOCAL_WRITE; > > + if (rdma_protocol_iwarp(pd->device, > > + rdma_start_port(pd->device))) > > + access_flags |= IB_ACCESS_REMOTE_WRITE; > > + } > > + > > + if (roles & RDMA_MRR_READ_SOURCE) > > + access_flags |= IB_ACCESS_REMOTE_READ; > > + > > + if (roles & RDMA_MRR_ATOMIC) > > + access_flags |= IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_ATOMIC; > > + > > + if (roles & RDMA_MRR_MW_BIND) > > + access_flags |= IB_ACCESS_MW_BIND; > > + > > + return access_flags; > > +} > > +EXPORT_SYMBOL(rdma_device_access_flags); > > + > > struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd, > > struct ib_phys_buf *phys_buf_array, > > int num_phys_buf, > > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h > > index 986fddb..135592d 100644 > > --- a/include/rdma/ib_verbs.h > > +++ b/include/rdma/ib_verbs.h > > @@ -2494,6 +2494,107 @@ static inline int ib_req_ncomp_notif(struct ib_cq > > *cq, int wc_cnt) > > struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int mr_access_flags); > > > > /** > > + * rdma_mr_roles - possible roles an RDMA MR will be used for > > + * > > + * This allows a transport independent RDMA application to > > + * create MRs that are usable for all the desired roles w/o > > + * having to understand which access rights are needed. > > + */ > > +enum { > > + > > + /* lkey used in a ib_recv_wr sge */ > > + RDMA_MRR_RECV = 1, > > + > > + /* lkey used for a IB_WR_SEND in the ib_send_wr sge */ > > + RDMA_MRR_SEND = (1<<1), > > + > > + /* rkey used for a IB_WR_RDMA_READ in ib_send_wr wr.rdma.rkey */ > > + RDMA_MRR_READ_SOURCE = (1<<2), > > + > > + /* lkey used for a IB_WR_RDMA_READ in the ib_send_wr sge */ > > + RDMA_MRR_READ_DEST = (1<<3), > > + > > + /* lkey used for a IB_WR_RDMA_WRITE in the ib_send_wr sge */ > > + RDMA_MRR_WRITE_SOURCE = (1<<4), > > + > > + /* rkey used for a IB_WR_RDMA_WRITE in ib_send_wr wr.rdma.rkey */ > > + RDMA_MRR_WRITE_DEST = (1<<5), > > + > > + /* > > + * rkey used for a IB_WR_ATOMIC/MASKED_ATOMIC in ib_send_wr > > + * wr.atomic.rkey > > + */ > > + RDMA_MRR_ATOMIC = (1<<6), > > + > > + /* MR used for a IB_WR_MW_BIND in ib_send_wr wr.bind_mw.bind_info.mr */ > > + RDMA_MRR_MW_BIND = (1<<7), > > +}; > > + > > +/** > > + * rdma_mr_attributes - attributes for rdma memory regions > > + */ > > +enum { > > + RDMA_MRA_ZERO_BASED = IB_ZERO_BASED, > > + RDMA_MRA_ACCESS_ON_DEMAND = IB_ACCESS_ON_DEMAND, > > +}; > > + > > +/** > > + * rdma_device_access_flags - Returns the device-specific MR access flags. > > + * @pd: The protection domain associated with the memory region. > > + * @roles: The intended roles of the MR > > + * @attrs: The desired attributes of the MR > > + * > > + * Use the intended roles from @roles along with @attrs and the device > > + * capabilities to generate the needed access rights. > > + * > > + * Return: the ib_access_flags value needed to allocate the MR. > > + */ > > +int rdma_device_access_flags(struct ib_pd *pd, int roles, int attrs); > > + > > +/** > > + * rdma_get_dma_mr - Returns a memory region for system memory that is > > + * usable for DMA. > > + * @pd: The protection domain associated with the memory region. > > + * @roles: The intended roles of the MR > > + * @attrs: The desired attributes of the MR > > + * > > + * Use the intended roles from @roles along with @attrs and the device > > + * capabilities to define the needed access rights, and call > > + * ib_get_dma_mr() to allocate the MR. > > + * > > + * Note that the ib_dma_*() functions defined below must be used > > + * to create/destroy addresses used with the Lkey or Rkey returned > > + * by ib_get_dma_mr(). > > + * > > + * Return: struct ib_mr pointer upon success, or a pointer encoded errno > > upon > > + * failure. > > + */ > > +static inline struct ib_mr *rdma_get_dma_mr(struct ib_pd *pd, int roles, > > + int attrs) > > +{ > > + return ib_get_dma_mr(pd, rdma_device_access_flags(pd, roles, attrs)); > > +} > > Do we really need the rdma_get_dma_mr() wrapper? > > I suggest to start consolidating to ib_create_mr() that receives an > extensible ib_mr_init_attr and additional attributes can be mr_roles > and mr_attrs. > > I have no problem with renaming it to rdma_create_mr() if people really > want to. > > See my comment in: http://marc.info/?l=linux-rdma&m=143539761710553&w=2 > > Thoughts?
I'm not opposed to your suggestion to consolidate. I'd like to hear what others think? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
