Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Jeff Squyres (jsquyres)
I am quite definitely of the opinion that a novice user should be able to "./configure --prefix=blah; make -j 32 install && mpicc my_mpi_app.c && mpirun a.out", and OMPI should generally do the Right Thing. I'm not opposed to being able to set configure-time defaults, but I (fairly strongly) be

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Atchley, Scott
On Oct 21, 2015, at 11:09 AM, Jeff Squyres (jsquyres) wrote: > REVISION 2 (based on feedback in last 24 hours). > > Changes: > > - NETWORK instead of NETWORK_TYPE > - Shared memory and process loopback are not affected by this CLI > - Change the OPAL API usage. > > I actually like points 1-8

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Ralph Castain
I’m not sure you are correct in stating that the simplest case “obviously” still needs to work. In fact, there were several proposals on the last telecon to the contrary as it was unclear that the community would be able to agree on defaults. So people suggested either requiring configuring with

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Jeff Squyres (jsquyres)
On Oct 21, 2015, at 12:05 PM, Ralph Castain wrote: > > It seemed like this topic was straying, so I’m glad to hear that specifying > nothing means we still execute. Yes, I'm not trying to change the simple/easiest case. That obviously still needs to work. > My question remains, though: what

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Ralph Castain
It seemed like this topic was straying, so I’m glad to hear that specifying nothing means we still execute. My question remains, though: what is the default? What are the default values of the networks/qualifiers? > On Oct 21, 2015, at 8:56 AM, Jeff Squyres (jsquyres) > wrote: > > On Oct 21

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Jeff Squyres (jsquyres)
On Oct 21, 2015, at 11:32 AM, Ralph Castain wrote: > > With all due respect, I think this still dodges the key question. Are we now > saying that every user will be *required* to provide this info? If not, then > what is the default? > > Let’s face it: the default is what 90+% of the world is

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Ralph Castain
With all due respect, I think this still dodges the key question. Are we now saying that every user will be *required* to provide this info? If not, then what is the default? Let’s face it: the default is what 90+% of the world is going to use. This all seems rather complex to expect the averag

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Jeff Squyres (jsquyres)
REVISION 2 (based on feedback in last 24 hours). Changes: - NETWORK instead of NETWORK_TYPE - Shared memory and process loopback are not affected by this CLI - Change the OPAL API usage. I actually like points 1-8 below quite a bit. If implemented in ALL BTLs/MTLs/etc., it can solve the "how d

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Jeff Squyres (jsquyres)
On Oct 20, 2015, at 6:37 PM, Paul Hargrove wrote: > > > I am suggesting that a user wishes to NOT USE a specific port at all. > In other words, I want to "obstruct" all of the API paths that might reach > that port. > However, they do want to use some other port of the same type - which means

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Jeff Squyres (jsquyres)
On Oct 21, 2015, at 8:27 AM, Atchley, Scott wrote: > >> 2. --enable would work similar to our "include" MCA params: OMPI will *only* >> use the network type(s) listed. > > In this scenario, will the user still need to “enable” off-node network, sm, > and self? Or do you assume sm and self? I

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Atchley, Scott
Hi Gilles, My main concern is that if the user specifies InfiniBand per Jeff’s proposal, will they get or not get sm (of any flavor) and self. Scott On Oct 21, 2015, at 9:00 AM, Gilles Gouaillardet wrote: > Scott and all, > > two btl are optimized (and work only) for intra node communicatio

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Gilles Gouaillardet
Scott and all, two btl are optimized (and work only) for intra node communications : sm and vader by "sm" I am not sure you mean the sm btl, or any/both sm and vader btl. from an user point of view, and to disambiguate this, maybe we should use the term "shm" (which means sm and/or vader btl for

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-21 Thread Atchley, Scott
On Oct 20, 2015, at 4:45 PM, Jeff Squyres (jsquyres) wrote: > On Oct 20, 2015, at 3:42 PM, Jeff Squyres (jsquyres) > wrote: >> >> I'm guessing we'll talk about this at the Feb dev meeting, but we need to >> think about this a bit before hand. Here's a little more fuel for the fire: >> let's

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Paul Hargrove
On Tue, Oct 20, 2015 at 3:00 PM, Jeff Squyres (jsquyres) wrote: > I don't think it gets up to 7 MCA params to guarantee a specific API path > is used to get to a specific network / port, but your overall point is fair. Jeff, it sounds to me you are responding to a different problem than the one

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Jeff Squyres (jsquyres)
On Oct 20, 2015, at 5:26 PM, Paul Hargrove wrote: > > I have multiple ports of the same type, lets say a dual-port Mellanox HCA, > and just want to disable one of them (reserving it for Luster perhaps). > If OMPI is hiding from me the details of the API selection, how do I > enable/disable spec

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Jeff Squyres (jsquyres)
On Oct 20, 2015, at 5:26 PM, Paul Hargrove wrote: > > I think heterogeneous multirail is still pretty uncommon. It might still be > ok to force users (or better yet, their admins -- via the global > mca-params.conf file) to use level 3 to precisely specify which network / > OMPI API to use (e

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Jeff Squyres (jsquyres)
On Oct 20, 2015, at 5:26 PM, Ralph Castain wrote: > > Understood - but can we narrow it down a bit? Specifically, do we need both > BTL and MTL access to the same network? That's what I'm saying: for usnic, I don't know yet (we haven't finished our tag matching implementation yet). I suspect

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Paul Hargrove
On Tue, Oct 20, 2015 at 1:47 PM, Jeff Squyres (jsquyres) wrote: > On Oct 20, 2015, at 4:35 PM, Paul Hargrove wrote: > > > > As an example, I might have two ethernet cards, one of which is a Cisco > VNIC. > > I would want be able to control which BTL or MTL is used on those NICs > independently,

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Ralph Castain
Understood - but can we narrow it down a bit? Specifically, do we need both BTL and MTL access to the same network? This would cut the combinations by 2x right away. Then we could potentially remove the network-specific MTLs. Then we just have to deal with UCX vs libfabric - so only the two deci

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Jeff Squyres (jsquyres)
On Oct 20, 2015, at 4:49 PM, Ralph Castain wrote: > > Your last point about the qualifiers is kinda what I was hinting at in my > note. If you have usnic support via the OFi MTL, why do you also need it as a > BTL? The BTL needs libfabric anyway, yes? So is there some value in having > both me

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Ralph Castain
Your last point about the qualifiers is kinda what I was hinting at in my note. If you have usnic support via the OFi MTL, why do you also need it as a BTL? The BTL needs libfabric anyway, yes? So is there some value in having both methods? Same question for PSM and PSM2, and probably others I

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Jeff Squyres (jsquyres)
On Oct 20, 2015, at 4:35 PM, Paul Hargrove wrote: > > As an example, I might have two ethernet cards, one of which is a Cisco VNIC. > I would want be able to control which BTL or MTL is used on those NICs > independently, including the option to disable use of one or the other. > I do not want t

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Jeff Squyres (jsquyres)
On Oct 20, 2015, at 3:42 PM, Jeff Squyres (jsquyres) wrote: > > I'm guessing we'll talk about this at the Feb dev meeting, but we need to > think about this a bit before hand. Here's a little more fuel for the fire: > let's at least specify the problem space a bit more precisely... I'm replyi

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Ralph Castain
Don’t you also have the question of, for example, PSM via the mtl/psm versus PSM via the mtl/ofi path? So I think you need to split the entries in #2 as: PSM/MTL PSM/MTL/OFI PSM2/MTL PSM2/MTL/OFI etc. Or we could remove the PSM/PSM2 MTL components and just drive those thru the OFI provider int

Re: [OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Paul Hargrove
I looked quickly over the quoted emails and didn't see something I had hoped/expected to. In addition to the "dimensions" of type, api and pml I think users may also be concerned about the "port" dimension (or device if you prefer). So, it might be worth including that in the discussion of the hig

[OMPI devel] Specifying networks/APIs for OMPI (was: topic for agenda)

2015-10-20 Thread Jeff Squyres (jsquyres)
We talked about this on the call last week. I'm guessing we'll talk about this at the Feb dev meeting, but we need to think about this a bit before hand. Here's a little more fuel for the fire: let's at least specify the problem space a bit more precisely... (this item is on the agenda for the