[EMAIL PROTECTED] wrote: > Ok, this is a longer answer. > > At 06:08 PM 3/20/2006, Fabian Tillier wrote: >> You pre-alloc the MPT entry, but not the MTT entries. You then >> populate the MTT by doing posted writes to the HCA memory (or host >> memory for memfree HCAs). ... >> I don't know if allocating MTT entries is really expensive. What >> costs is the fact that you need to do command interface transactions >> to write the MTT entries, while FMRs support posted writes. > > I don't know what MPTs and MTTs are (Mellanox > implementation?) nor do I know exactly what the overhead > difference you refer to really is. > It's less about the overhead and more about the resource contention, > in my experience. > >> That is, just like with alloc_fmr, you need to reserve and format an >> MPT for regular memory registrations, which is a command interface >> transaction. For memory registration, one or more commands precede >> this to write to the MTT. Thus, a memory registration is at a >> minimum a 2 command interface transaction operation, potentially >> more depending on the size of the registration. >> >> Deregistration and freeing (not unmapping) an FMR should be >> equivalent, I would think. > > So, in the RPC/RDMA client, I do ib_alloc_fmr() a bunch of > times way up front, when setting up the connection. This > provides the "windows" which are then used to register chunks > (RPC/RDMA segments). > > As each RPC is placed on the wire, I borrow fmr's from the > above list and call ib_map_phys_fmr() to establish the > mapping for each of its segments. > No allocation is performed on this hot path. > > When the server replies, I call ib_unmap_fmr() to tear down > the mappings. > No deallocation is performed, the fmrs are returned to a > per-mount pool, *after unmaping them*. > > I just want the fastest possible map and unmap. I guess that means I > want fast MTT's. >
Which is exactly what an FMR work request wouuld provide, which is the interface specified by both RDMAC and IB 1.2. When an FMR is a work request it *must* be lightweight (no resource allocation, just copying new pre-validated already prepped values into existing resources) and can be fully pipelined. Work requests are the only vendor neutral, transport neutral, solution that is viable here. _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
