Hi Eric, On Fri, Dec 15, 2006 at 11:47:40AM -0000, Eric Barton wrote:
> > One interesting result is that smaller chunk sizes than we currently > > recommend can increase read performance by allowing larger request > > sizes. I'm not sure why it is possible to do a 2 or 4MB request > > through the MD layer with 128K chunks but not with 256K chunks. No > > error is shown on the console in the failing case. > > What is "chunk size"? Chunk size is a RAID parameter, applicable to at least RAID 5. It is the size of the individual pieces written to each disk. For example, if Lustre does a 1MB write on a 4+1 array with a 256K chunk size, it will be broken up into 4 256K chunks and a furthur 256K parity chunk will be computed and written (assuming writes are aligned on chunk boundaries.) We currently recommend selecting chunk size and stripe size such that (stripe size) = (chunk size) * (# active disks). One thought I had on the smaller chunk size is that perhaps 128K IOs end up being merged by Alex's raid5-merge-ios.patch so the disks actually see larger IOs and therefore perform well. Alex, is this possible? Do you have any other theories on why smaller chunk sizes than we recommend actually produce better performance? > > So far the best performance for both reads and writes I've observed is > > with a 128K chunk size and 4MB requests. This request size corresponds > > to a stripe size of 4MB, assuming all layers from obdfilter to the MD > > device preserve the 4MB IO intact. > > What is the page size of this machine? There are issues with scatter/gather > descriptors larger than 1 page which you might be running into. It's x86_64, so 4KB. I don't think scatter/gather descriptors come into play at this point since the 4MB write will be broken up into smaller writes by the MD layer. > By default, the maximum request size issued by clients is 1MByte which > is the LNET MTU. It's possible to build lustre with a larger maximum > I/O size that can be exploited in special cases (e.g. no routing, page > size on client and server > 4K and the LND supports it). Supporting these > in the general case requires changes in lustre (e.g. multiple bulk > transfers in a single RPC). What about echo clients talking directly to a local obdfilter? That is what I am attempting to survey. I realize Lustre in general does not support IOs larger than 1MB; part of the point of my current work is to see if 4 MB IOs are something we need to support in more general cases. Cheers, Jody _______________________________________________ Lustre-devel mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-devel
