On Wed, 2009-10-28 at 16:25 -0400, Chris Worley wrote: > On Wed, Oct 28, 2009 at 1:58 PM, David Dillow <dillo...@ornl.gov> wrote: > > Under noop, the block layer will send requests as soon as it can without > > merging. If it has more requests outstanding than the queue length on > > the SRP initiator, then it will merge the new request with the queued > > ones if possible. > > So, noop will merge requests when the queue is full, but not hold-off > to merge?
Correct. > > The SRP initiator just hands off requests as quick as they are sent to > > it by the block layer. You can control how big those requests are by > > tuning /sys/block/$DEV/queue/max_sectors_kb up to .../max_hw_sectors_kb > > which gets set by the max_sect parameter when adding the SRP target. > > So the block layer may also hold-off on small requests, and decreasing > max_sectors_kb will force it to flush to the SRP initiator ASAP (or is > this just used for fragmentation of large requests)? It is just used for breaking up large requests. The deadline, as, and cfq schedulers may have some hold-off -- I've not checked -- but noop does not. You can check the length of the queue by looking at /sys/class/scsi_disk/$TARGET/device/queue_depth. That may well be 63, which is the maximum queue depth for the SRP initiator unless you patch the source. Keep in mind that those 63 requests are shared across all LUNs on that connection, so you may queue up before that, if you are driving many LUNs. > Note that 'm trying to minimize latency for very small requests. Reads or writes? Are you doing direct IO or plain read/write? File system or block device access? Are you using the SCSI devices (/dev/sda etc) or DM multipath (/dev/mpath/*)? The SRP initiator is playing the cards it has been dealt, but you could be getting coalescing from the rest of the system -- for example, I have no idea if the SRP target code will do read ahead and turn a 4KB request into a 64KB one -- I suspect it is possible. You can also turn on SCSI logging to see what is being handed to the initiator to be sure which side of the connection this is occurring. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
