On Mon, Feb 22, 2010 at 03:22:52AM -0800, Vipul Pandya wrote: > Hello Issac,
Hi Vipul, > ...... > I lowered the map_on_demand value to 16 and now it works fine. > > However, I had once concern, whether lowering down this map_on_demand > value would impact the performance of Lustre or not? For iWARP, you probably have no alternative. I remembered that there's a restriction somewhere in the iWARP stack that limits the size of SQs (which was why the rdma_create_qp errors happened), and lowering map_on_demand is the only way to reduce Lustre SQ length. For infiniband, lowering map_on_demand essentially reduces the # of RDMA WQE needed for each Lustre bulk data movement, at the cost of memory registration/deregistration at most per bulk transfer; without map_on_demand the o2iblnd uses a static MR so there's no memory registration cost. There could a point in the # of frags of the bulk buffer, where the cost of handling RDMA WQEs (which usually equals the # of frags) exceeds the cost of MR, and that's what you should set map_on_demand to. However, since both costs are mostly determined by HCA hardware/firmware implementation, there's no one good setting for all interconnects, and you can only find it by testing. The LNet selftest is a useful tool for running such tests: http://manual.lustre.org/manual/LustreManual16_HTML/LustreIOKit.html#50610302_36273 Hope this helps, Isaac _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
