Hi Roland,
Roland Dreier wrote:
Stick in a separate library then?
I don't think we want the complexity in the kernel -- I personally would
argue against merging it upstream; and given that the userspace solution
is actually faster, it becomes pretty hard to justify.
Memory registration has always been expensive, so it's not in the
critical path (not used for small messages and a system call overhead is
nothing for large messages in MPI). Sure, you can have the kernel notify
the user space through mapped flags, but it's a bit ugly IMHO.
There are cases where the basic registration already uses the same
infrastructure as a regcache. For example, on Solaris, MacOSX and Linux
PowerPC, you really want to register segments as large as possible to
limit the IOMMU overhead. You also don't want to register multiple time
the same page with overlapping registrations, because the IOMMU space is
limited. In short, you already have a registration cache in the driver.
However, if the user space is expected to call register/deregister
often, then I agree that the cache better be in user space.
The big picture is that it's not really important where the regcache
lives, as long as it's out of MPI.
Patrick
