On Mon, 2011-01-24 at 10:32 -0500, Or Gerlitz wrote: > David Dillow wrote: > >> if we look on the 50% for SAS/1M IOs that you're presenting, can you tell > >> what made the difference, srp went from sg_tablesize of 255 to 256 so the > >> upper layers where able to provide 1M as one IO > > > This win is from sg_tablesize going from 255 to 256 in this case; the HW > > really likes that better than getting two requests -- one for 1020 KB > > and one for 4 KB. > > Its always nice to find the simplest explanation to the greatest > improvement... going to the 2nd largest gains > > > SAS 2M 520 MB/s 861 MB/s > > SAS 4M 529 MB/s 921 MB/s > > SAS 8M 600 MB/s 951 MB/s > > I wonder what made the difference here? it can't be only the 255 --> > 256 sg_tablesize change, for the 2M case the change to use 512 pages > FMRs could let you use one rkey/fmr for the whole IO but not for 4M/8M
Actually, it very much was the sg_tablesize going from 255 to 2048 (512 actually used for 2 MB requests). This was pushed into 2 FMRs, as I was only using 256 entry FMRs for the test; I did not rerun the tests after I added the 512 entry FMR change. So, for above 1 MB, we always used indirect descriptors. The target used has a high overhead per command, and cannot aggregate back-end disk commands when the write cache is disabled, so it thrives on large requests. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
