[EMAIL PROTECTED] wrote on Tue, 27 Jun 2006 09:06 -0400: > At 02:42 AM 6/27/2006, Michael S. Tsirkin wrote: > >Unless you use it, passing the absolute maximum value supported by > >hardware does > >not seem, to me, to make sense - it will just slow you down, and waste > >resources. Is there a protocol out there that actually has a use for 30 sge? > > It's not a protocol thing, it's a memory registration thing. But I agree, > that's a huge number of segments for send and receive. 2-4 is more > typical. I'd be interested to know what wants 30 as well...
This is the OpenIB port of pvfs2: http://www.pvfs.org/pvfs2/download.html See pvfs2/src/io/bmi/bmi_ib/openib.c for the bottom of the transport stack. The max_sge-1 aspect I'm complaining about isn't checked in yet. It's a file system application. The MPI-IO interface provides datatypes and file views that let a client write complex subsets of the in-memory data to a file with a single call. One case that happens is contiguous-in-file but discontiguous-in-memory, where the file system client writes data from multiple addresses to a single region in a file. The application calls MPI_File_write or a variant, and this complex buffer description filters all the way down to the OpenIB transport, which then has to figure out how to get the data to the server. These separate data regions may have been allocated all at once using MPI_Alloc_mem (rarely), or may have been used previously for file system operations so are already pinned in the registration cache. Are you implying there is more memory registration work that has to happen beyond making sure each of the SGE buffers is pinned and has a valid lkey? It would not be a major problem to avoid using more than a couple of SGEs; however, I didn't see any reason to avoid them. Please let me know if you see a problem with this approach. -- Pete _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general