> From: Kenneth L Jeffries [mailto:[EMAIL PROTECTED] > Sent: Tuesday, November 01, 2005 4:45 PM > > > My objections are the following (as I said in my previous mail): > > - I don't like allocating a 1 KB IU for every send IU, since most of > > that memory will probably never be used. > > - I'm not convinced that it's _ever_ a win to have the target do > > another RDMA to fetch the indirect buffer list. You need to > > convince me that it's not better to simply tell the upper layers > > what the limit on s/g list length is to fit in the current IU size. > > I also don't want to allocate 1KB IU's. If IU's were fixed size, I'd want > (probably, depending on performance testing) a fixed size of 350 bytes > (from Fab Tiller's 64KB i/o, 4KB pages, Windows) or possibly even > the mininum DDBD (as Fab Tiller also says). 1KB IU's with thousands > of RC's causes me a lot of wasted space heartburn.
Even 350 bytes is a burden - imagine a target that supports a queue depth of 1000 I/Os from a few dozen initators. Ideally, I'd like to see us use just DDBDs and the 64-byte IU, along with registering the data buffers on a per-I/O basis, either via FMR or regular MRs. > [as an aside, it sure would be nice if we could do an SRP-3 (since SRP-2 > is dead) where multiple direct descriptors would be allowed. The only > way to get multiple descriptors now is with indirect descriptors.] That saves you 20 bytes - not a huge gain. > I am pretty sure that someone doing a video server might want to do, say, > 1MB i/o's. 1MB with 4KB pages means 256 descriptors and an iu of > something over 4096 bytes. I definitely don't want to be told by the srp > initiator that I need to use 4KB iu's. (So we agree there.) For large I/O, doing a registration of the buffer and sending a DDBD with a single descriptor might well provide the best performance. If you look at the traffic on the wire, having the target do multiple page-sized RDMA operations is far less efficient than creating a virtual contiguous (to the target) region that a single RDMA operation can service. - Fab _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general