On Wed, 2009-10-28 at 13:38 -0600, Chris Worley wrote: > There is no scheduler running on either target or initiator on the > drives in question (sorry I worded that incorrectly initially), or so > I've been told (this information is second-hand).
So, noop scheduler, then? Under noop, the block layer will send requests as soon as it can without merging. If it has more requests outstanding than the queue length on the SRP initiator, then it will merge the new request with the queued ones if possible. > I did see iostat > output from the initiator in his case, where there were long waits and > service times that I'm guessing was due to some coalescing/merging. > There was also a hint in the iostat output that a scheduler was > enabled, as there were non-zero values (occasionally) under the > [rw]qm/s columns, which, if I understand iostat correctly, means there > is a scheduler merging results. > > So you're saying there is no hold-off for merging on the initiator > side of the IB/SRP stack? The SRP initiator just hands off requests as quick as they are sent to it by the block layer. You can control how big those requests are by tuning /sys/block/$DEV/queue/max_sectors_kb up to .../max_hw_sectors_kb which gets set by the max_sect parameter when adding the SRP target. You can get some hold-off potentially by using a non-noop scheduler for the block device, see /sys/block/$DEV/queue/scheduler. 'as' or 'deadline' may fit your bill, but they have a habit of breaking up requests into smaller chunks. Also, you want 'options ib_srp srp_sg_tablesize=255' in /etc/modprobe.conf, as by default it only allows 12 scatter/gather entries, which will only guarantee a 48KB request size. Using 255 guarantees you can send a 1020KB request. Of course, if the pages coalesce in the request, you can send much larger requests before running out of S/G entires. max_sectors_kb will limit what gets sent in either case. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
