Hi Issac, This was very helpful. Thanks a lot for your response.
Vipul -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: 26 February 2010 02:27 To: Vipul Pandya Cc: [email protected] Subject: Re: [Lustre-discuss] Lustre-1.8.1.1 over o2ib gives Input/Output error while executing lctl ping On Mon, Feb 22, 2010 at 03:22:52AM -0800, Vipul Pandya wrote: > Hello Issac, Hi Vipul, > ...... > I lowered the map_on_demand value to 16 and now it works fine. > > However, I had once concern, whether lowering down this map_on_demand > value would impact the performance of Lustre or not? For iWARP, you probably have no alternative. I remembered that there's a restriction somewhere in the iWARP stack that limits the size of SQs (which was why the rdma_create_qp errors happened), and lowering map_on_demand is the only way to reduce Lustre SQ length. For infiniband, lowering map_on_demand essentially reduces the # of RDMA WQE needed for each Lustre bulk data movement, at the cost of memory registration/deregistration at most per bulk transfer; without map_on_demand the o2iblnd uses a static MR so there's no memory registration cost. There could a point in the # of frags of the bulk buffer, where the cost of handling RDMA WQEs (which usually equals the # of frags) exceeds the cost of MR, and that's what you should set map_on_demand to. However, since both costs are mostly determined by HCA hardware/firmware implementation, there's no one good setting for all interconnects, and you can only find it by testing. The LNet selftest is a useful tool for running such tests: http://manual.lustre.org/manual/LustreManual16_HTML/LustreIOKit.html#506 10302_36273 Hope this helps, Isaac _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
