On Apr 30, 2009, at 6:22 PM, Jason Gunthorpe wrote:
After reading all the postings, I think my idea to fix the verbs API
to not, essentially, corrupt an existing registration when the virtual
address space changes is the best bet. This slightly changes the
semantics of the verbs MR to refer to virtual address space within the
process, not the underlying object(s) that happen to be mapped there
when the registration is made.
I'm not sure how this helps MPI -- our registration caches will still
become invalid if the MPI app free()'s registered memory...?
MPI maintains a registration cache because registration is so
expensive. Even if the registration cache becomes "safely" invalid
(e.g., you'll never get a scenario where one virtual address could
have previously pointed to a different hardware address within the
span of one process), it doesn't help.
MPIs can choose to continue to hook malloc/free/etc or not, it doesn't
No no no! We don't want to continue hooking this stuff. The hooks
are horrible, horrible, horrible -- there's real-world apps that break
them.
> While MPI is currently the biggest victim, this broken memory
management
> model is also an enormous roadblock for any other application or
ULP to
> write to verbs.
I'm not sure this is true.. The purpose built verbs apps I've worked
on don't have a problem like MPI, and managing the memory registration
was not hard.
Ok, I'll back off slightly: if you want verbs to go mainstream, there
will be many other ULPs / middleware libraries that have memory models
like MPI's (that the upper layer is responsible for allocating/freeing
message buffers). Put differently: the TCP/sockets stack doesn't have
this restriction; it will be extremely difficult to convert legions of
sockets programmers to verbs if you effectively restrict large
messages to only be allocated/freed by the network layer (kinda
defeats the point of RDMA if you have to copy large messages, right?).
--
Jeff Squyres
Cisco Systems
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
