Gleb Natapov wrote:
On Wed, Dec 12, 2007 at 04:08:31PM +0200, Pavel Shamis (Pasha) wrote:
Gleb Natapov wrote:
On Wed, Dec 12, 2007 at 03:37:26PM +0200, Pavel Shamis (Pasha) wrote:
Gleb Natapov wrote:
On Tue, Dec 11, 2007 at 08:16:07PM -0500, Jeff Squyres wrote:
Isn't there a better way somehow? Perhaps we should have "select"
call *all* the functions and accept back a priority. The one with the
highest priority then wins. This is quite similar to much of the
other selection logic in OMPI.
Sidenote: Keep in mind that there are some changes coming to select
CPCs on a per-endpoint basis (I can't look up the trac ticket right
now...). This makes things a little complicated -- do we need
btl_openib_cpc_include and btl_openib_cpc_exclude MCA params to
include/exclude CPCs (because you might need more than one CPC in a
single job)? That wouldn't be hard to do.
But then what to do about if someone sets to use some XRC QPs and
selects to use OOB or RDMA CM? How do we catch this and print an
error? It doesn't seem right to put the "if num_xrc_qps>0" check in
every CPC. What happens if you try to make an XRC QP when not using
xoob? Where is the error detected and what kind of error message do
we print?
In my opinion "X" notation for QP specification should be removed. I
didn't want this to prevent XRC merging so I haven't raced this point.
It is enough to have two types of QPs "P" - SW credit management "S" -
HW credit management.
How will you decide witch QP type to use ? (SRQ or XRC)
If both sides support XOOB and priority of XOOB is higher then all other
CPC
then create XRC, otherwise use regular RC.
If some body have connectX hca but he want to use SRQ and not XRC ?
This will be the default. (prio of OOB will be bigger than of XOOB), but
if uses will want to use XRC it will increase XOOB priority by
specifying MCA parameter.
I guess anyway we will be need some additional parameter that will allow
enable/disable XRC, correct ? (So why just not leave the X qp type ?)
Because we want to support mixed setups and create XRC between nodes that
support it and RC between all other nodes.
Ok, sounds reasonable for me.
Just need make sure that the parameters name will be user friendly.
Some thing like --mca enable-xrc that will cause to XOOB priority be
highest (and not something like --mca xoob 10 :-))
Pasha
