* Zhang, Jingwang <jingwang.zh...@emc.com> [2012-11-20 22:25:11]: > Thank you for your detailed analysis. Here are some comments: > 1. You are right about the memcache, it doesn't cache the contents. The > memcache only caches buffers registered with ibv_reg_mr() in order to avoid > registering and deregistering the same buffer over and over again. > 2. Yes, it would work fine as long as the mapping from "virtual address" to > "physical address" stays the same, no matter the buffer is freed(not freed to > kernel, which might change the mapping very likely) and then re-allocated. > 3. For big trunk memory, the address space of this process might grow or > shrink when the buffer is allocated or freed. As a result, the assumption in > 2 is broken and the message is corrupted. And we found this problem when we > run the test/unit-tests/ zoidfs-io-cunit tool, in which the buffer size is 1M > ~ 8M.
It seems that, for the memcache to be beneficial and still be correct, we need to trap the calls to free to make sure we can remove the entry from the cache the moment the user frees the memory. This is what most (all?) IB implementations that do registration caching use. Thanks, Dries
pgp1mmgyin79O.pgp
Description: PGP signature
_______________________________________________ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
