On Jul 27, 2015, at 1:25 PM, Jason Gunthorpe <jguntho...@obsidianresearch.com> wrote:
> On Mon, Jul 27, 2015 at 11:57:46AM -0400, Chuck Lever wrote: >> IMO ib_unmap_fmr is a very different animal from LOCAL_INV WR. > > Sure, but how many of these properties does NFS actually care about, > now that it is running the API properly? > >> ib_unmap_fmr is synchronous, provides no ordering guarantees with >> send queue operations, and does not depend on a connected QP to >> be available. You could emulate asynchronicity with a work queue >> but that still does not provide SQ ordering. There are few if any >> failure modes for ib_unmap_fmr. > > I'm having a hard time seeing how SQ ordering is important when the > API is used properly. Once you explicitly order the DMA unmap after > the invalidate completion you no longer need implicit SQ ordering > > Is there a way to combine SQ implicit ordering and the Linux DMA API > together correctly? > >> flight, the LINV flushes. MRs can be left in a state where the >> MR's rkey is not in sync with the HW, in which case a >> synchronous operation may be required to recover the MR. > > The error handling seems like a trivial difference, a > ib_recover_failed_qp_mr(mr); sort of call could resync everything > after a QP blows up.. Out of interest, why does this need to be exposed to ULPs? I don't feel a ULP should have to deal with broken MRs following a transport disconnect. It falls in that category of things every ULP that supports FRWR has to do, and each has plenty of opportunity to get it wrong. >> The complexion of MR invalidation could be vastly different in >> a few years: handled entirely by the target-side, and only >> verified by the initiator. Verification doesn't need to sleep, >> and the slow path (the target failed to invalidate) can be >> deferred. > > The initiator still needs to have the ability to issue the invalidate > if the target doesn't do it, so all the code still exists.. > > Even ignoring those issues, should we be talking about putting FMR > under the new ib_alloc_mr and ib_map_mr interfaces? Would that help > much even if the post and unmap flows are totally different? My opinion is FMR should be separate from the new API. Some have expressed an interest in combining all kernel registration mechanisms under a single API, but they seem too different from each other to do that successfully. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
