On Wed, 4 Mar 2009, George Bosilca wrote:
I'm churning a lot and not making much progress, but I'll try chewing on
that idea (unless someone points out it's utterly ridiculous). I'll look
into having PML ignore sendi functions altogether and just make the
"send-immediate" path work fast with normal send functions. If that works,
then we can get rid of sendi functions and hopefully have a solution that
makes sense for everyone.
This is utterly ridiculous (I hope you really expect someone to say it). As I
said before, SM is only one of the networks supported by Open MPI.
Independent on how much I would like to have better shared memory
performance, I will not agree with any PML modifications that are SM
oriented. We did that in the past with other BTLs and it turned out to be a
bad idea, so I'm clearly not in favor of doing the same mistake twice.
Regarding the sendi there are at least 3 networks that can take advantage of
it: Portals, MX and Sicortex. Some of them do this right now, some others in
the near future. Moreover, for these particular networks there is no way to
avoid extra overhead without this feature (for very obscure reasons such as
non contiguous pieces of memory only known by the BTL that can decrease the
number of network operations).
How about removing the MCA parameter from my earlier proposal and just
having r2 filter out the sendi calls if there are multiple BTLs with
heterogeneous BTLs (ie, some with sendi and some without) to the same
peer. That way, the early sendi will be bypassed in that case. But for
the cases of BTLs that support sendi in common usage scenarios
(homogeneous nics), we'll get the optimization? Does that offend you
George? :)
Brian
